Summer Of Code – Six Months On

Six months ago, the Google Summer of Code finished up, and the ten Mozilla-related projects submitted their final reports. I’ve just spent some time looking into the current status of all ten projects, and thought I would share those results with you.

The ten projects were:

My original plan was to have a table highlighting the similarities and differences in current status, but it’s hardly worth it. None of the ten projects shows any sign whatsoever of work having continued after the SoC deadline. Not one. No development-related mailing list traffic, no releases, nothing. It’s as if the people vanished off the face of the earth on September 2nd.

Of the ten projects, as far as I can tell only one (Firefox in Hindi) installs in the latest stable build of the relevant product, and that’s not because it’s been updated since the release. I suspect it’s merely due to a loose XPI maxVersion setting.

The four localisation projects may have some excuse; unfortunately, it turned out that the evaluators chose four projects which were all duplicates of existing nascent l10n efforts. However, the SoCcers concerned were switched to translating other products – and those translations have not been maintained either. In one case (Hindi), the original group has very recently acquired CVS checkin rights, but for the other three, neither team seems to have made progress towards becoming an official, reliable source of a localisation in that language.

I read recently that Venture Capital firms evaluate the team behind a company much more closely than they evaluate their product idea. As it happens, as far as I know, none of the SoCcers chosen were well-known in the Mozilla community. (Of course, I don’t know how many well-known community members applied.) This may be relevant; we’d probably need data from other projects to know for certain. But perhaps the SoC form should contain a “why should we pick you” section as well as a “why should we pick your project” section.

A couple of the projects (“Bittorrent client for Firefox”, “Graphical theme builder for XUL”) were far too large for a single person working in their spare time over the summer. So it’s unsurprising that the SoCcers either got discouraged, or produced incomplete work. Evaluators have a responsibility to size projects, and assess whether they are suitable for the SoC scheme. Not every idea is.

I believe that the money was paid to all the SoCcers except for one; that person did not produce any code at all. All the people chosen wrote a final report. Several of them give roadmaps and future plans for their product, and promise that work will continue after the project is finished. However, in every case, this has not happened – for whatever reason (I believe, for example, that one project owner became a father). Still, it’s interesting that once the money is paid, the drive seems to evaporate.

The difficulty for the mentors is that Google discouraged partial payments – it was all or nothing. So if the person had done some work, they were left with the difficult decision of over- or under-rewarding the person. I suspect that, in several cases, they erred on the side of generosity – after all, it wasn’t their money they were spending! Which perhaps is fair enough, given the setup.

I don’t want this post to be seen as bashing either SoCcers or mentors; they were offered a deal, and they took it. That’s fair enough. After all, I was a mentor – and there was nothing to force the SoCcers to work on after the period had finished. But if we want the Summer of Code to be useful in the future, it’s reasonable to look at our experience and their behaviour, and see if the goals were met and, if not, how to change things.

So what can we conclude? Here are some suggestions:

  1. Evaluators need to be very careful about which projects they approve, and carefully consider their relationship to other ongoing efforts.
  2. Evaluators should evaluate people as much as, or more than, projects, and give preference to people who are already committed to being part of the community.
  3. Evaluators should be careful not too choose projects which are too large, ill-defined or overly subjective.
  4. Mentors should be permitted to pay out partial amounts. This gives SoCcers a much greater incentive to do a good job.
  5. Mentors must be ruthless about paying out in proportion to the amount of available, working, useful and well-documented code.
  6. The success of the SoC needs to be measured over the long term, not just at the end of the coding period.

If anyone reads this who was involved in running the SoC for other projects, I would be interested to see similar analyses from there.

16 thoughts on “Summer Of Code – Six Months On

  1. That’s terribly depressing. How does this compare to other Summer of Code projects involving other open source projects? The only one of these I was following was Firepuddle, and I noticed it died pretty quickly.

    I think you’re right in saying that lots of these projects were way too big for one person over one summer. I think a program like this needs to have specific deliverables, with measurable outcomes. For example, the Firepuddle code should have been able to be compiled from CVS, and installable in Firefox 1.5. It looks like a lot of decent code was written, but due to it not being usable out of the box, it will probably just sit in obscurity.

  2. Well, I think the idea attracts a different set of coders than open-source projects do. This is basically a contract-hire deal. It’s like the many psychology studies that rewards children for, say, deciding to draw in an activity room. Afterwards these children are found to turn away from the activity that they originally undertook for fun but which ended up giving them a treat. It became work, and they decided later to stop drawing because work without rewards isn’t very motivating.

    Best,
    Tsee

  3. Thanks for the insightful analysis. I have been thinking about Summer of Code and meant to look at results from last year.

  4. As a SoC participant who has not worked on his project since October, I understand what you mean. We’ve been discussing this somewhat on the SoC mailing list. For many busy students, the only way to afford spending time on free software was to receive a stipend for it (they would otherwise have used that time working minimum wage jobs). In my case, I had just graduated and had lots of time in the summer while I was looking for jobs. I started my job at the end of July, and though I faithfully worked myself to death through the month of August, once the SoC deadline rolled around I found I had little time to work on my project (which had no users anyway). I have thus reverted to a more passive role in the open source community, mostly participating in mailing lists and forums, and submitting bug reports. Work, wife, dog, and volunteering just eat up too much of my time for anything more active than that. It’s a shame, but unless I work somewhere that gives 20% time like Google, it’s all I can really do right now. I wish the program had existed earlier in my university career when I could have devoted more of myself to it.

  5. We found something like 30% of the students stuck with their groups post SOC. This is more than we expected! Remember that many people have school/lives to go back to. While we were happy to see those that stuck around post SOC, we are very happy to have exposed people to the open source world, and the open source world to new people.

    The comments about the selection process and the rest have -not- fallen on deaf ears, we’re making a lot of changes for a potential 2006 program. It was our first year and such and we knew it would be a little rocky.

    Chris DiBona

  6. Chris: thanks for stopping by :-) Has Google done any research into SoC follow-up that it’s willing to share? It’s interesting that you had expectations for “stickability” and other metrics – it would be great to see those too.

    I realise that people have lives to go back to. But perhaps my idea of the aim of the SoC and your idea are different. I thought it was about getting work done, yes, but also in helping people to become long-term contributors rather than just doing work-for-hire. And so I thought about success in those terms. However, you seem to see it just as a way of getting people introduced to the idea. Of course, as it’s your program, you get to say what it’s for :-)

    As for your last paragraph (with the “-not-“), I hope you don’t think I posted this because I think Google is not listening, or that you messed up in some way. You did an excellent job, it was a great program, and anyone who gives away $2M to get free software written deserves credit :-) I just wanted to provide some constructive input into whatever process you are going through.

    I’m glad that there’s a possibility of a 2006 program, and the Mozilla project would love to be involved again. It would also be great if you could publish or circulate your proposed new process for comments and suggestions.

  7. Hi !

    Just thought I’d tell about my own experience. I was a SoCer in 2005 for Ubuntu, and since then I have been working with Ubuntu all the time, on my SoC project or on other enhancements (see http://www.manucornet.net/ubuntu ). I have become an official Ubuntu member last January and plan to keep working with them (and GNOME) for a long time.

    According to Corey Burger’s blog (http://www.advogato.org/person/Burgundavia/diary.html?start=70 ) I seem to be the only Ubuntu SoCer who’s still working with Ubuntu right now.

    I agree, the percentage seems low. But people who actually plan to keep working for Open Source, even if that’s only 25 or 60%, already make a *big* difference in my opinion. Therefore, I still consider SoC 2005 as a success.

    I really hope Google is going to make this happen again in 2006, and I’ll be glad to submit one or two projects for Ubuntu, GNOME, or other organizations!

  8. Nah, I didn’t thnk that you thought I messed up (I mean, Idid in some ways, but that’s okay, nothing/nobody is perfect, not first time round). The SOC had a lot of goals, including:

    Introducing students to open source.
    Introducing open source to new students.
    Pfroviding a realisitic summer experience to burgeoning developers.
    Bringing attention and money to projects.

    And more.

    If there was one state goal internal to google it was:

    “Flip bits not burgers!”

    :-)3

    Chris

  9. I can’t speak for anyone else, but the money was a plus, not the main motivation. I would love to continue Event Logger, but you’d going to have to find someone to finish the second year of my degree if you would like to see any immediate work on it.

    Summer is really the only time I have, besides trying to maintain my half-dozen other extensions and keep up my other interests. The lack of community and MoCo interest is also a downer. I’m not expecting people to care en-masse, but I started the project originally to aid the QA process. So far, I’ve not even had one reply to several emails asking people to evaluate the usefulness of such a project.

  10. Ben: but isn’t that part of the problem? We failed when we approved a project proposal from you that, in fact, no-one except yourself was interested in? (That’s not meant to sound harsh; it just seems a reasonable conclusion from the lack of interest you have seen.) This isn’t your fault – we need to work out how we can encourage and approve proposals which will be most useful.

    Gerv

  11. There is little hope in a process that is driven by giving money away. It has no cycle of feedback built into it, so is unlikely to be appropriately directed. I believe the best way to fund OS is to most closely approximate the market, which succeeds because of the built-in cycle of usefulness feedback.

    For example, choosing already useful projects and improving them. Find successful projects and ask the leaders what they would do if given X students and Y budget for summer. Then finance them into a summer long development ‘push’ or ‘sprint’. In this way, you are not paying them for the results of the summer, you are rewarding them for past success – which is a feedback action and therefore can be positive.

    This of course leaves aside the happy chaos of the bazaar, the place where the new and useful projects spring from – but it is not really possible to simulate chaos with money and concentration. Better to play to the strengths that you do have, and bring projects together that can really benefit from the money and concentration.

  12. first of all these are students not part of a corporate machine. Summer of code seemed like a good idea to me but having read some of these posts you should be ashamed of yourself. All or nothing… screw you screw goog. Money was never the incentive, and if you told me you got started in open source because of the money then I really know you’re a jackass. More to the point the open source community will gravitate to ideas worth pursuing, and the only thing Mozilla is good for is browsing porn because it will name your files automatically. It’s really not that exciting.