The Mozilla project is groaning under the load of incoming bugs (see this thread in the newsgroups for some info and stats). Firefox is, I assert, not getting any more buggy (if anything, it’s getting less so), and yet the number of bugs filed is going up and up. It may well therefore be that a greater percentage of bugs filed are not actually useful to us.
There’s a good chance that any one particular filed bug is actually a bad thing – that is, its existence has a negative impact on the project. Each filed bug takes valuable contributor and/or developer time to triage and assess. If a layout developer spends 20 minutes looking into bug A and finding it’s a duplicate of bug B, he’s just acquired a small bit of information (“bug B may be more common than I thought”) at a very high cost. The time of layout developers is very valuable – there aren’t many of them.
More bugs is not necessarily a good thing. Bugs reduce bugzilla.mozilla.org performance, they clog up searches, they inflate numbers (which depresses people) and they tempt people to spend time looking at them.
So as a first step, let’s move away from the idea that every incoming bug report is valuable in and of itself, and we have to feel guilty if we don’t do anything with it. After all, we’ve already done this for broken website reports and crashes – all the reports go into Reporter and the crashes into Breakpad, but we aggregate them and look at the most frequent. That means that the reports of some people with rare crashes or with problems on not-popular websites will never be considered at all. No-one suggests that we are telling the reporters of such problems that they are not valuable because we do nothing with their information. So why should this be true of other bug types? Ignoring a bug is not a statement about the personal worth of the person who filed it.
We need to figure out where the greatest concentration of value is and focus on extracting it, rather than trying to answer the question “how do we extract all the value?” Trying to get everything is a Herculean task which will result in miserable and burned-out QA people.
So how do we extract the most value for the least effort?
Firstly, we need to make sure that everyone with a problem ends up in the right place. There are lots of different places they might need to be:
- Broken Website Reporter
- MozillaZine Forums
- Support Newsgroups/Mailing Lists
We need to figure out what the right places are and who needs to go to them when. E.g. when should someone be sent to SUMO, and when to the MozillaZine Forums, and when to the support newsgroups? Do we need to close or deprecate one or more of these support mechanisms in order to pool resources and have a consistent story? Or do they in fact serve different constituencies? If so, what words will direct people to the right one?
One suggestion would be have a well-known URL which explains what options there are, and why you might want each of them, and make sure everyone always links to that rather than any of the individual tools. The top of the front page of Hendrix or of b.m.o. actually looks a little bit like what such a page might be.
This not only benefits the people with a problem, but it benefits us, because we won’t get support requests in Bugzilla or bug reports in SUMO.
Once we’ve done that, there are several things we could do to cut down the flow of incoming bugs:
- Reduce the number of people who turn up to file (make the bug reporting mechanism harder to find, or direct people elsewhere)
- Reduce the conversion rate (i.e. scare people off during the process)
- Get more aggressive about resolving bugs INCOMPLETE
- Find a way to aggregate general bugs automatically
- Ignore more bugs (sort of what we are doing now)
I’m sure there are more things than that.
The question is: which of these will be most effective at retaining the valuable and discarding the less valuable? Different options will bias the bug filing process so that different sorts of people are more likely to be successful. For example, option 1) favours the inquisitive.
Here’s an important observation. Option 2) favours the persistent. If there is a correlation between that personality trait and bug quality – i.e. persistent people make better bug filers – then making the bug filing process easier to understand might actually be a bad move for the project. This seems counter-intuitive, but I think we need to grasp the idea that if there’s one resource that’s plentiful here, it’s the time of bug filers. And so optimising the process for their time, rather than QA time or developer time, is counter-productive. If it turns out that we can make all bug reports 10% better by adding another five compulsory questions to the Bugzilla Helper (for example) then that would be a good thing. Fewer bugs, better quality. (I’m not saying this is actually true – it’s a hypothetical.)
These are only some initial thoughts for discussion. This issue really needs a champion – someone who can look at all the possibilities, do some research and come up with a well-justified plan. Such a person would, I’m sure, be showered with thanks and praises by the whole community. I’m afraid I don’t have time. Do you?