The Mozilla project is groaning under the load of incoming bugs (see this thread in the newsgroups for some info and stats). Firefox is, I assert, not getting any more buggy (if anything, it’s getting less so), and yet the number of bugs filed is going up and up. It may well therefore be that a greater percentage of bugs filed are not actually useful to us.
There’s a good chance that any one particular filed bug is actually a bad thing – that is, its existence has a negative impact on the project. Each filed bug takes valuable contributor and/or developer time to triage and assess. If a layout developer spends 20 minutes looking into bug A and finding it’s a duplicate of bug B, he’s just acquired a small bit of information (“bug B may be more common than I thought”) at a very high cost. The time of layout developers is very valuable – there aren’t many of them.
More bugs is not necessarily a good thing. Bugs reduce bugzilla.mozilla.org performance, they clog up searches, they inflate numbers (which depresses people) and they tempt people to spend time looking at them.
So as a first step, let’s move away from the idea that every incoming bug report is valuable in and of itself, and we have to feel guilty if we don’t do anything with it. After all, we’ve already done this for broken website reports and crashes – all the reports go into Reporter and the crashes into Breakpad, but we aggregate them and look at the most frequent. That means that the reports of some people with rare crashes or with problems on not-popular websites will never be considered at all. No-one suggests that we are telling the reporters of such problems that they are not valuable because we do nothing with their information. So why should this be true of other bug types? Ignoring a bug is not a statement about the personal worth of the person who filed it.
We need to figure out where the greatest concentration of value is and focus on extracting it, rather than trying to answer the question “how do we extract all the value?” Trying to get everything is a Herculean task which will result in miserable and burned-out QA people.
So how do we extract the most value for the least effort?
Firstly, we need to make sure that everyone with a problem ends up in the right place. There are lots of different places they might need to be:
- Bugzilla
- Hendrix
- Broken Website Reporter
- SUMO
- MozillaZine Forums
- Support Newsgroups/Mailing Lists
We need to figure out what the right places are and who needs to go to them when. E.g. when should someone be sent to SUMO, and when to the MozillaZine Forums, and when to the support newsgroups? Do we need to close or deprecate one or more of these support mechanisms in order to pool resources and have a consistent story? Or do they in fact serve different constituencies? If so, what words will direct people to the right one?
One suggestion would be have a well-known URL which explains what options there are, and why you might want each of them, and make sure everyone always links to that rather than any of the individual tools. The top of the front page of Hendrix or of b.m.o. actually looks a little bit like what such a page might be.
This not only benefits the people with a problem, but it benefits us, because we won’t get support requests in Bugzilla or bug reports in SUMO.
Once we’ve done that, there are several things we could do to cut down the flow of incoming bugs:
- Reduce the number of people who turn up to file (make the bug reporting mechanism harder to find, or direct people elsewhere)
- Reduce the conversion rate (i.e. scare people off during the process)
- Get more aggressive about resolving bugs INCOMPLETE
- Find a way to aggregate general bugs automatically
- Ignore more bugs (sort of what we are doing now)
I’m sure there are more things than that.
The question is: which of these will be most effective at retaining the valuable and discarding the less valuable? Different options will bias the bug filing process so that different sorts of people are more likely to be successful. For example, option 1) favours the inquisitive.
Here’s an important observation. Option 2) favours the persistent. If there is a correlation between that personality trait and bug quality – i.e. persistent people make better bug filers – then making the bug filing process easier to understand might actually be a bad move for the project. This seems counter-intuitive, but I think we need to grasp the idea that if there’s one resource that’s plentiful here, it’s the time of bug filers. And so optimising the process for their time, rather than QA time or developer time, is counter-productive. If it turns out that we can make all bug reports 10% better by adding another five compulsory questions to the Bugzilla Helper (for example) then that would be a good thing. Fewer bugs, better quality. (I’m not saying this is actually true – it’s a hypothetical.)
These are only some initial thoughts for discussion. This issue really needs a champion – someone who can look at all the possibilities, do some research and come up with a well-justified plan. Such a person would, I’m sure, be showered with thanks and praises by the whole community. I’m afraid I don’t have time. Do you?
One aspect of bugzilla usage which I think should not be forgotten is that it has been (historically, at least) the gateway drug for getting involved in the Mozilla community. The classic path led from filing bugs to helping with QA to creating simple patches to …
I don’t know what distinguishes the people who walked that path from those who never go beyond filing bugs, but care should be taken not to scare off the first group. (Of course there’s probably a high correlation between them and those people who file high-quality bug reports, but at least it should help emphasize that those are the people who we really want to retain.)
A well known URL like http://www.mozilla.org/support/ perhaps?
A few comments on your numbered options:
1. This is good, but very very important is that when issues are getting brought up in SUMO or elsewhere that at some point the support teams realise they must pass these issues on to developers. I’m continually coming across issues that the support guys know about and how to solve and even a little about what causes it yet it’s a complete surprise to me and often there is even no bug filed about the issue, which makes the chances of it getting fixed pretty slim.
2. I hope by “scare people off” you mean “direct to a more appropriate forum for reporting the issue”, though that really seems basically the same as the previous point.
3. I’m already doing this, my component is down to 10 unconfirmed bugs which is a nice easy figure to maintain, it works pretty well.
4. I’m not sure you’ll ever get much here beyond triagers duping and adding meta bugs for related issues which already goes on anyway.
5. Not sure what bugs you are talking about. I think all newly filed bugs warrant some attention. It doesn’t take long to ask for clarification or reduced testcases. If the reporter can’t provide that then you make a judgement over whether it is worth your time to find out precisely what is going on based on the apparent severity of the issue.
– Get to Some well known url as you said mo.com/support.
– Ask following question
– Website Not working (Reporter)?
– Firefox crashes. (Explain how to report with Hendrix)
– Plugins (Explain how to get them)
If none of the above.
– Links to Forum / support
– Go to bugzilla
– on Bugzilla hide new Bug button on Home page :) yes seriously.
– Ask for the subject / module / version
– Search bugzilla.
Not found ?
Now show link for new Bug.
I think its better to have something like getsatisfaction. Contextual inline search. Fast and furious. :)
Is this essentially for the Firefox case or should it apply to other products?
I suspect others do not get same amount of bug or at least proportional figures. A variation may just be in bug reporter structure vs bug reports quality (say that maybe another product has 50% thorough reporters even if not fewer bugs or 10% of good reporters that file 90% of bugs).
The issue being that eventually each would have it’s own policy and approach, while on same bugzilla and probably some same qa and volunteers. This may seam subtle, but some schizo would be required to be harsher on FF than on another.
I can only think of better guiding people towards solving issues (and only eventually file bugs) rather than make it harder. Something like “help people help” rather than “stop them annoying”. So I’m mostly for the first part, getting people in the right place as I see support very much tied to bugs. Like first teach them to narrow issues, to ask around, to test and only the last step becomes bug/rfe. (http://tinyurl.com/5db7fb touches this, maybe bit OT)
Or maybe people should land on bugzilla only after some guidance from others..
[I’ve been mostly in TB area and some calendar and I have seen huge difference in all aspects of bug work, from numbers to responsiveness, from people involved to how drastic the result. For obvious reasons.]
This sounds like an area ripe for some automated analysis, as in
– user types in bug report
– compare bug report to existing known bug reports with similar characteristics
– display list of suggestions
Basically a vastly improved search – the existing search is pretty useless unless you’re really familiar with how the mozilla project works because of the prevalence of insider-speak in the bugs.
A specialised lexicon of search-synonyms for mozilla would help.
The current search results page only displays the summary/title, which is not enough information to know if the bug is the same as mine or not.
I could be wrong, but I don’t think the current search does full-text search by default, which limits its usability. The eclipse bugzilla is much better in this regard. In fact, I can recommend their set of “entry” pages as being much easier to use.
Does Bugzilla currently have any way to detect duplicates or common issues? I’ve seen some support ticket systems that analyze the content/keywords of the text you’ve entered and attempt to match it with existing knowledgebase articles.
So like when someone clicks submit on their bug report, Bugzilla would first take them to a screen that presents a couple SUMO articles that may match up to the issue they’re reporting, and to a few existing bugs that may describe the same problem. Or if it appears to be a single broken site, it could suggest using the reporter instead. If none of the presented choices match their issue, then they can go ahead and submit their report. A bit more time for the submitter, but as you said, their time is not the commodity that’s in short supply.
From a different interface, the same mechanism could also be used by volunteers who want to help with finding existing duplicates and tagging them as such.
How about adding a multiple choice “test” at the beginning of bug reporting ?
Pointing to a site for example that doesn’t rendering well in Firefox doesn’t mean that this is a Gecko issue. Send the reporter only to bugzilla if the site works well in another version of Firefox or Opera/Safari or if he can show a violation of a specification including URL. If he wants to report a crash he should either include a breakpad ID or a stack trace with symbols.
If he can provide such information point him to SUMO for example.
One major difference between Micro$oft and Mozilla is that Micro$oft’s bug process is closed (restricted to Micro$oft personnel) while Mozilla’s is open. This difference makes Mozilla products attractive to knowledgeable users.
It appears that a major similarity between the two organizations is that enhancements often have priority over bugs. After all, doing something new is much more interesting than repairing something old. This might actually be worse with Mozilla because of how much the organization depends on unpaid volunteers.
I believe there would be a significant reduction in duplicate bug reports if more attention were placed on repairing some old things. (Older bug reports seem to accumulate the most duplicates.) This becomes a management problem. Since I retired, I’ve been doing volunteer work in various venues. I might not be a paid employee, but I still have to follow the directives (orders) given by the paid managers. Perhaps, the Mozilla managers need to direct more effort towards fixing discrepancies and less effort towards RFEs.
When I was a software test engineer on a very large, long-running project, our bug reporting was open. 90% of the valid bug reports were generated by the testers, 6% were generated by the developers, 4% were generated by the end users. If an overwhelming proportion of the valid Mozilla bug reports are not generated by QA and developers, something is very wrong with your QA and development processes.
I’ve long thought long time users should be funneled through a process to report issues. For example only those with canconfirm can deposit a bug directly in bugzilla. Otherwise you’d have to go through the end user process:
1. Go to support site.
2. Select from list of options “site doesn’t work”, “feedback”, “application bug”, etc.
3. Submit info.
That way the info goes to the right locations. Ideally bugzilla could do some fancy logic and try to find duplicates before submission (kind of like what Digg does with submissions). But that might cause more confusion than anything else.
I think we need to make a distinction between “site doesn’t work” and “(my) site doesn’t work and I know exactly what the issue is and am trying to tell you about it” at filing time. The latter bug is a LOT more valuable, on average.
I think we also need to make sure that regression bugs (site works in fx2 but not fx3) make it through more than vanilla “site doesn’t work” bugs.
I don’t have a solution, but I notice that there are an awful lot of “me too” comments that sometimes gum up the works. Theoretically, I suppose voting for bugs is supposed to take care of this.
I think a combination of things would help.
Launchpad has excellent duplicate detection when you enter a bug, and Kiko said that he would share the algorithm with me so that we could implement it in Bugzilla.
I think an analysis of exactly *what* is unhelpful about the majority of bugs would be ideal. What is the missing information that causes you to ask a question back to the reporter? We could adjust the bug entry form accordingly, and possibly even re-work its UI so that it’s easier for the reporter to provide the requested information. If the problem is mostly duplicates, the above solution would probably help a lot.
Possibly a brief questionnaire before bug-filing would help direct people to the appropriate resource instead of Bugzilla, but perhaps it should be cookied or stored in the database that the questionnaire has been answered, so people know. Or just show them a page with some explanations on it and a five-second or 10-second timer on the “next” button so that they’re encouraged to read it.
Finally, MoCo may want to actually hire triagers. I think it’s a valid software engineering role. Also, putting a lot more attention on bugday and triage again (perhaps even some “Help Out Firefox” links from the front of mozilla.com) would help recruit volunteer manpower.
-Max
“So as a first step, let’s move away from the idea that every incoming bug report is valuable in and of itself”
Considering that there are bug reports in BugZilla that are open for more than TEN YEARS, you guys obviously “moved away” right from the start.
GM;
Part of this may simply traced to both the number of contributors and also how contributors do or approach things – now – versus in the past.
While I have used Mozilla Feedhouse as my homepage for quite some time – I now find all too frequently in Firefox 2x – particularly in recent weeks and months – nagging issues or “inconveniences” that were less “often” or completely absent in the past.
Why should Firefox users now be getting prompted via a password prompt (Master Password prompt window) when they load feedhouse – just to scan recent posts from a variety of blogs – this happens simply because someone has included – say a “litmus” image in their blog post – ala Chris Cooper – and that – for some reason which escapes me – this Litmus image is/was linked via https – all for an “image” to be displayed publicly.
Now I see in recent days where some element of your post or one of the others on feedhouse is now also blessed with some “bugzilla” encrypted link included in their blog posts – again triggering a Master Password prompt – for users that rely on this feature – again simply to load the feedhouse page and scan the blog posts. Certainly this is not required and/or should not be required or even prompted – just for a visitor to read feedhouse – but because of the design of the browser and components of a feed content – individuals are now being required to “clear” a prompt – which in my view should never have triggered in the first place.
Quite simply these types of “lack of developmental oversight” and what simply may be “inadequate use of common sense” were less prevalent in the past – could be yet another reason for the “increase” in bug submissions – though I have not yet – submitted this/these particular issues as bugs – others may have.
Not really a “browser bug” – but I am confident that individuals that must deal with these prompts are certainly “bugged” by now having to deal with them!!!
My take…..
Mike
Mike, OK, I’ll be the bad guy here. Gervase Markham’s blog is neither Bugzilla nor user support. Try support.mozilla.com or forums.mozillazine.org .
New bugs are possible at any time, but much of the time problems are probably caused by user actions. My hunch is that you are prefetching. User support can help you sort this out, and can also help you to write a useful bug report in the unlikely event that you have actually found a bug.
Is this mostly about Firefox? Cause in other products things may look different. For one thing in bug and reporter numbers and also in approach.
I say this cause eventually a per product policy will naturally arise (and already has ..). At least I can say that about TB and calendar where things are already different from FF and each other, for obvious reasons. But where some contributors are not. And thus, how schizo one can be .. Well this is mostly about the latter indications, aggressive invalid, ignore etc.
+1 for get people in the right place. I’d say that a guided support-to-bug process and directions would be next step. Support is much tied to this and one may be better directed to narrow issue, understand feature, ask around, only eventually ending in bugzilla. That is “help to help”.
just some thoughts ..
When we discussed this topic six years ago, I suggested: “First, stopping the effects of the swamping (bugmails), by getting rid of component owners. Second, making existing bugs easier for beginners to find. Third, making new bugs more difficult for beginners to file.”
The first of these has since been done, albeit hackishly, by setting a dummy owner for all Firefox components.
For the second, as Max Kanat-Alexander mentioned, you could provide passive search to reduce duplicates. (And not just for the summary, but also live updating suggestions while the description is being filled in.) This would require more database power, but would save time both for reporters and for QA staff.
The third matches up with your #2, and it’s something that’s niggled away at me for the past three years while I’ve been on the Launchpad team, as Ubuntu’s bug report volume has grown to dwarf Mozilla’s. One possibility that’s occurred to me is splitting the “Description” field into multiple compulsory fields: for example, the traditional “Steps to reproduce”, “What happens”, and “What should happen”. This would seldom trouble good bug reporters, who routinely provide all that information anyway. But it would achieve the dual purpose of increasing the difficulty for less serious reporters, and making the resulting reports more informative and easier to scan.
(For Launchpad it wouldn’t be as simple as that, because we cater not just for Ubuntu but also for less popular projects that want bug reporting to be really simple. So we’d need to make it configurable per-project, but that probably wouldn’t be necessary for bugzilla.mozilla.org.)
The Ubuntu Bug Squad is (at least in my experience) also pretty keen on #3. Often they mark a bug report as Incomplete if a new alpha/beta version has been released, and rely on the reporter to reopen the report if the bug persists.
mpt: I am coming to the idea that we need a “Help/Feedback Wizard”, which guides people to support, to the broken website tool, to Hendrix or to Bugzilla as appropriate and, if the answer is Bugzilla, then seamlessly takes them through a guided process with lots of searching and checking – like Launchpad’s but with a slightly less odd and jarring UI. (Click radio button, “Yes, it’s a new bug” – page suddenly scrolls on you. Ick!)
As for your last para, I’ve been on the end of the Ubuntu Bug Squad’s aggressive bug closing, and I personally think it’s too aggressive. I’ve never seen any evidence that they’ve attempted to reproduce any of the issues I’ve raised, even if they are issues of principle. Example:
“Battery Monitor should use absolute remaining time rather than % remaining time to calculate colour. My 6hr lifetime Thinkpad goes red with 1.5 hours left, which is more than some laptops when they are full.”
<later>
“Please can you tell us if this bug is still reproducible on the latest alpha?”
“Well, of course it will be, unless you’ve changed how the code works…”
Gerv
I like the ideas surrounding some kind of feeback wizard that walks people through a process to direct them to the right place for their feedback.
I think we can also change the front end of the bugzilla helper to ask better questions on a per product basis, so that other products aren’t affected by this (because I think they have very different requirements than Firefox does).
But I think that bugzilla is a great place where several of our contributors first start contributing, and making it significantly harder to use places yet another barrier of entry to the project, so I wouldn’t support anything that would cause that to happen.
I too like the idea of a guided help/feedback process. One tricky issue in doing that, though, would be preventing Google from leading people straight past your carefully-designed process directly to the bug tracker. (bugzilla.mozilla.org’s robots.txt actually helps a bit here, though I know that’s not why it was put in place.)
True, that scrolling is nasty. Reported. :-)
I wasn’t saying the Ubuntu Bug Squad’s behavior was good, just pointing it out as an emergent effect of Ubuntu’s bug volume. It’s annoyed me when I’ve reported bugs, too.