Community Size, and Bug Quality

I’ve just finished watching Diederik van Liere’s fascinating presentation (Ogg video, 20 mins), linked to by Mark, about his work with bugzilla.mozilla.org data. Download the full slide deck – you’ll need it to follow along with my comments, which should be of interest to the community-minded.

Slides 9 and 10, the community dynamics slides, are worrying. Diederik explains the decline in the size of the community as “increased product maturity”, but I’m not sure that’s true. If it were, we would expect to see the number of people entering the community decreasing. However, that number has stayed roughly constant, with peaks surrounding a release as we reach out to beta testers, and as new users report bugs. It’s the number of people leaving which seems to be increasing. Fewer people are hanging around once they get here. And I don’t know why.

Slide 11, about what the final resolutions ended up being of bugs filed in each year, is fascinating. It would be really interesting to see that graphed at a greater granularity.

We have made significant progress in reducing the percentage of duplicate bugs filed. People complain about Bugzilla’s search, but the percentage was a flat 25% from 2002-2004, and it’s now a flat 13% – basically half what it was. Also, from a low of 34% of bugs filed being FIXED, we are now up at 56%.

There are no doubt lots of factors involved in this effect, including the reduction in the size of the community mentioned in the earlier slides. I’m really interested in working out what we might have done which has improved the situation. I have an utterly open mind on this. here are some significant events I can think of which may be relevant:

Can anyone else think of other events or software deployments which may be relevant? I’m happy to do the research to find out when they happened.

7 thoughts on “Community Size, and Bug Quality

  1. > Diederik explains the decline in the size of the community
    > as “increased product maturity”, but I’m not sure that’s true.

    I’m sure it’s at least part of it. The product became essentially mature several years ago (arguably before the Firefox name was introduced, although different people will no doubt draw the line at different places, because product maturity is something you can really only approach asymptotically). As the browser already does more and more of what people want, they feel less and less need to influence its future direction. This is normal.

    Haven’t you also seen an increasing lag in the promptness with which people upgrade when new versions are released? I mean, back in the 0.9.x days, nobody wanted to use a three-month-old version of Mozilla. Remember that? Back then, three-month-old version was a crashy bug-ridden feature-impoverished piece of flotsam compared to the latest release. Today, a three-YEAR-old version is so similar to the current one, a normal user is unlikely to notice the difference even if they use the one version at work and the other at home every single day.

    This is not a bad thing. It just means the browser already does most of what we want it to do. There have been some improvements, sure. Performance optimization and text-shadow support spring to mind. And there have been some gratuitous “let’s change it just to change it” changes (e.g., the Awesomeness bar). And there’ve even been some new bugs (like, opening a tabset now causes the page in the current tab to be overwritten/lost). But on the whole these are all pretty small things for most users.

    It’s not like you just introduced a major new component, and it’s got all kinds of bugs and design issues that still need to be worked out. I mean, when *was* the last major new (user-visible) component introduced? The last one I can think of off the top of my head is the extensions mechanism. When that first went in, it was buggier than week-old garbage. Installing extensions was a pain, and there was no easy way to upgrade them, and every single time you upgraded the browser they were all gone and you had to start over again. (Personally, I kept the suite’s Navigator as my primary browser, even though I didn’t use any of the other suite components, for the first while, because a lot of the features Firebird users needed extensions for were built into the suite in the first place. I finally switched somewhere around Firefox 1.0, although I had Firefox installed and experimented with it much earlier, starting when it was still called Phoenix.) It took a while for the extension mechanism’s sharp edges to be rounded off, as it were.

    But now? Now the extension mechanism works more or less how we want it to work. (One minor remaining grievance is that there’s no obvious discoverable way for a sysadmin to install an extension for all users. But most users don’t need that.) Now, the product is essentially mature.

    > If it were, we would expect to see the number of
    > people entering the community decreasing.

    That would only make sense if the number of people being exposed to the product were constant (or declining). If the product is *gaining* marketshare, however, you would, all else being equal, expect to see an increase in the number of people entering the community.

    > It’s the number of people leaving which seems to be increasing. Fewer
    > people are hanging around once they get here. And I don’t know why.

    The reason they came (typically, some feature they wanted or bug that bothered them) is either addressed, and they’re satisfied, or it’s not, and they get discouraged with the process. (I’d imagine there’s some of each.) There aren’t eighty other things immediately springing to mind that they’d also like to see changed.

    I know my involvement in the Mozilla community has certainly decreased since 1999. It’s not like I’ve stopped being interested in Mozilla products and gone over to Opera or something. (On the contrary, I recently experimented with chrome and found that I don’t like it. I also experimented with Seamonkey, but it crashes even more now than it did in 2001. So I’m still using Firefox, still think it’s the best option, still am interested in it. But it just seems like there’s less to do now.

  2. I suspect it isn’t just maturity. When we were doing Evolution, we had a maximum user base of someone around 2-4 million, and yet had 15-20,000 users of our nightly builds. With a user base of 100 times that, and arguably a much more broadly relevant mission (open web v. free software), Firefox still has around the same number of users of nightly builds, and apparently even fewer people using the trivially easy-to-use TestPilot. I have not yet put my finger on the cause, but these bug numbers feel like another data point in the same story of low participation to me.

  3. I would not be surprised if the amount of hired developers coincides with the decline. There is little need to casual users to file bugs if fuzzers find them automatically or the tinderbox automation makes enormous pressure on regressions. For instance when did you see the last unusable nightly? And there is a lot of personal behind this. So my perception is that for outsiders it becomes increasingly difficult to compete with highly skilled QA people and automation and a armada of developers.
    Diederik made a jetpack that predicted the likelihood of fix. In my opinion he severely underestimated the mozilla factor (0.25), where he used a mozilla.com mail address as a indicator of working for mozilla. This is defeated by old bugzilla Ids of bugzilla veterans like bz and dbaron.
    So probably the chances that external people are filing new important bugs is rapidly decreasing and the feedback of a duplicate or invalid just educates them to stop filing.

  4. >Slides 9 and 10, the community dynamics slides, are worrying

    I think it has to do with the “size” of Mozilla.
    Mozilla started as small Opensource project, as underdog compared to MS/IE.
    People tried to help to get an alternative Browser.

    Mozilla.org is now known to be one of the bug Browser corporations and most people don’t think that Mozilla needs much help and there are alternatives to Mozilla.

    They now report their issue if they have one and that’s it.

    Another issue is that you get no feedback from “official” persons or developers. Ask or do something in bugzilla and the chance that anyone else will answer that is small.
    It takes in many cases years that a developer adds a comment if I move a bug in the “right” component. This can be a little bit frustrating as reporter and as triager.

  5. I think there’s a lot of things we could do with Bugzilla’s web interface that could encourage community growth by including a broader audience and rewarding participation. There are a lot of amazing ideas in sites like getsatisfaction and stackoverflow that emphasize simplicity and community.

    I’m currently hacking away on some of my ideas for bugzilla on top of your (awesome) RESTful API. If I come up with anything meaninful, I’ll let you know :)

  6. Benjamin: wait a sec, though. It’s often better to use a more suitable tool than to try and adapt an unsuitable one. For many reasons, Bugzilla is not a great place to take general user feedback and build community with users. We already have the problems that Matthias outlined. And this is why we have e.g. Hendrix. Thunderbird is already using getsatisfaction. Rather than build stuff on top of Bugzilla, might it not be better to decide to use a tool which is designed for this sort of thing from the ground up?

    Gerv

  7. Hi Gerv,

    I certainly agree for general user feedback, but I think there are a lot of hackers and testers that Bugzilla could target better. This is what I meant by a broader audience. Some thoughts:

    * Transparency. It’s tough for casual patch authors to get their thumb on what’s going on with the project. A dashboard with recent bug activity, statistics, and next-release information could help a casual contributor find their feet and figure out what’s important. It also gives them a great cue about who to ask about getting started.

    * Rewards. StackOverflow is a highly technical site who motivates its users to contribute with visibility and acknowledgment. We do this at a broader scale, but building it into Bugzilla puts it at the core of the way we communicate at Mozilla.

    * User experience. For me personally, it takes a lot of energy to search and file bugs; I sometimes feel like Bugzilla is fighting against me. Simplifying the experience and optimizing common use cases (search!) is not just for your average user. It works for developers too.

    I mean all of this as constructive–but I do think we should not be afraid to evolve this product. The RESTful API is fortunately a great way to experiment with new ideas and see if anything sticks; it’s my hope that I can get some of my ideas in concrete form, and then developers can judge for themselves.