Automated UNCONFIRMED resolution

This is what the “Bugzilla Data Mining” entries have been leading up to. We’ve decided to attempt to use a semi-automated procedure to resolve some of the older untriaged UNCONFIRMED bugs. Here’s the plan – please let us know what you think.

  • Once a date is chosen, we raise awareness of the following procedure using blogs, newsgroups and websites. We don’t want anyone to get surprised.
  • At the appointed time, we turn off Bugzilla access to all except the person doing the change. This is to prevent things changing under us.
  • We search for all bugs in the Browser, Firefox, MailNews, and Thunderbird products[0] in state UNCONFIRMED where Last Changed Date is more than two months before the present.

    (The data mining used “bugs not confirmed after two months” – this is a subset of that. We are using “bugs not confirmed, and untouched in the last two months”, because then we won’t accidentally catch ones which are being worked on, but taking a long time to get confirmed.)

  • We then Change Multiple Bugs, and add the following comment[1]:

    This is an automated message.

    This bug has had no comments for a long time. Statistically, we have found that bug reports that have not been confirmed by a second user after two months are highly unlikely to ever actually be fixed.

    If you can still reproduce this problem in the latest version of the product (available from http://www.mozilla.org/products/) or, for feature requests, if you still believe we should implement this feature, please visit the URL of this bug (given in this mail) and add a comment to that effect, giving more reproduction information if you have it.

    If it is not a problem any longer, you need take no action. If this bug is not changed in any way in the next two weeks, it will be automatically resolved. Thank you for your help in this matter.

  • Two weeks later, we make a buglist of all bugs in the original list which have not changed since the date we added the above comment.
  • Do Change Multiple Bugs, add a “robo-resolved” keyword, and resolve them as WORKSFORME[2] with the following comment:

    “This bug has been automatically resolved after a certain period of inactivity (see above comment). Anyone who thinks this is incorrect should feel free to reopen it.”

So that’s our best shot. Any such process will never be perfect – but, in a perfect world, we’d have the manpower to triage all the UNCONFIRMEDs properly. But we don’t. So, we are looking for something which keeps as high a percentage of the good ones, and throws away as high a percentage of the bad ones as possible.

Here’s the logic. Bugs which haven’t been confirmed after two months are very unlikely to end up as FIXED – either they get fixed via another bug, or get fixed ‘accidentally’, or the code gets rewritten, or something else. We could just throw the lot away and not lose too much. But we want to do better, so we try and find the ones which might, by getting the reporters to reconfirm them. Getting reporters to reconfirm bugs is good because it tells us that they still see it and, just as importantly, they are still interested in helping us fix it.

Once this procedure is completed, we would use BugDays and other mechanisms to attack the bugs which had received comments, with the aim of resolving them or confirming them, starting with the oldest. This way, we hope to make a significant dent in the UNCO pile, and turn up the needles in our giant haystack.

[0] If you are the module owner of another product, and want to opt-in, shout.

[1] People who are not interested in reading these mails can create a filter and use Run Filters On Folder in Mozilla or Thunderbird, targetting the phrase “This is an automated message”.

[2] We either resolve as WORKSFORME, or OUTOFDATE if we can get that resolution added to Bugzilla. This depends on whether anyone has time to make the modification.

52 thoughts on “Automated UNCONFIRMED resolution

  1. Adding a new resolution (couldn’t you just rename “LATER” or something?) would be good. WORKSFORME won’t be appropriate in all cases.

    Also, the wording assumes (or sounds like it does) that the bug hasn’t been confirmed by a second user, when what it means is that it hasn’t been marked as confirmed by someone with privileges – would be better to say what it means, wouldn’t it?

  2. What about filtering unsuported platforms first ?
    There are some mac os

    Do you mean don’t include UNCOs from unsupported platforms in the procedure? Or do you mean do them first?

    michaell: good thought on the wording.

    We’re trying to write an email that’ll be understood even by people not involved in Mozilla from day to day. So I think “not confirmed by a second user” means roughly the same thing, and is more understandable than “not marked as CONFIRMED by another user with sufficient privileges”.

  3. I’m actually more comfortable with 3 months, but hey; I have no reason :)

    In any case, this should really be made into an automated process. Bugs that are X months old now are just as “bad” as bugs that are X months old tomorrow.

    If the process is automated, I’d rather make it run every 6 months or so – then you’re pretty much “guaranteed” that there is a new version of the product available.

    I’d also like there to be a page containing something like the following text (which should be made available for translation):

    “We have found that most bugs reported in our tracking system are caused by one of the following reasons:

    1. You have installed one of our products on top of an old install. Please (1) uninstall both, (2) delete the installdirectory and (3) install the product again. You will *not* loose your data.
    2. One of your installed extensions are behaving badly. Please uninstall the extensions one at a time. Look _here_ (some link) for deeper instructions.

    We have also found these steps to work in most cases:
    1. Try creating a new profile and reproduce the bug. If you _can_ reproduce it with a new profile and the latest nightly/release, you can report the bug. If NOT, then try the following, in order:
    * In your profile folder, delete XUL.mf(as)l
    * Try copying chrome.rdf from the new profile to your old one.

    2. Download a ZIP build of the latest nightly or beta/alpha release. If it works there, use that build untill a new release is made.”

    Anyway, the above text should be enhanced *greatly* if ever put somewhere. It’s just a draft/proposal.
    Now I’m really rambling .. heh, sorry about the long comment ;) But I stand by it – something like the above text in your X-monthly comments would do wonders, I think.

  4. >[0] If you are the module owner of another product, and want to opt-in, shout.
    Is there also a [3] opportunity to opt out from this?

    This is the last thing I want to happen in layout-tables with unconfirmed bugs. I don’t want to loose https://bugzilla.mozilla.org/show_bug.cgi?id=235446 with a testcase from dbaron only because it did not change over the past two months. And I certainly don’t want to be forced to save the unconfirmed from such an action by confirming them. This problem will arise in nearly every underowned component.
    How does this action differ from Sipaq’s MacOS cleanup attempt, where he deservedly nearly lost his bugzilla privilegs?
    Btw. speaking of reorg, what happened to Hixies bugzilla reorg?

  5. One Bugzilla enhancement that could prevent the spruious reporting of already-reported (and already-fixed) bugs would be to put a big notice on the bug-reporting form if the selected product is a browser (Browser, Firefox, Camino) and UA is an old build. Do similar sniffing to the Seamonkey start page.

    That would at least keep people from reporting bugs with a weeks-old milestone, as happens a lot.

  6. Following up on Bernd’s comment: if there’s no opt-out, perhaps it would help if any bug in the Layout components with keyword TESTCASE on it was excluded by default.

  7. Gerv, what about BugDay(s) targeted on these bugs between steps “Two weeks later,” and “Do Change Multiple Bugs, add a robo-resolved keyword”?

    I think, that all these still-UNCONFIRMED bugs are problem of Mozilla community, not problem of reporter or bug itself. Many bugs was never triaged and reporter hasn’t any way to to confirm it by self.

  8. Gerv – on the wording – I wasn’t meaning to suggesting you used those exact words, as I understand it needs to be possible for people not familiar with bugzilla to understand. On the other hand, if there’s a bug (and there are some like this) where multiple people have reported seeing the problem, but it hasn’t been confirmed because the descriptions aren’t good enough to figure out the right component, or because it requires access to some closed site or obscure product.

    For simple but still accurate wording, why not just say “not confirmed after two months”. They can see the “UNCONFIRMED” status and understand the concept, even if they don’t know how bugs get confirmed.

    Also, looking at the comments here, and thinking back to the previous stats, we are talking about (guesstimating) at least 300 useful bugs getting this notification (and then hopefully being poked so they don’t get resolved), which is rather a lot of “noise”.

  9. If you put the unconfirmed bugs into “uncertain state” after 2-3 months, don’t you risk having a group too big too handle? (a quick search on unconfirmed bugs older than 3 months gives some 6000 bugs…

    What if we take the 1000 oldest unconfirmed bugs and mark these “uncertain”, put the automated text in the bug and sent out the mailing to the concerned people. A month later we do the same with the 1000 unconfirmed bugs who are oldest then. And we continue like this untill the pack of unconfirmed bugs is “under control”

    Three months after marking the first group “uncertain” we evaluate what ahs changed. If there was some action we mark the bug as “new” otherwise we sent out a reminder to the persons involved. If after one month still noa ction we close the bug and mark as “wontfix”.

    We start with marking the first batch in 2004-10 as “uncertain”. In 2005-01 there is a first evaluation and the first pack is re-arranged: “new”, “fixed”, “wontfix”, … Those who are still “uncertain” get one final call. In 2005-02 the remaining “uncertain” bugs are closed.
    Each month we start another pack of 1000bugs.

    (note: this is just a draft idea)

  10. bernd – going back to your first comment – about Hixie’s bugzilla reorg. The answer is that it needs Mozilla staff (Asa in particular) to have the time to do it and to manage the disruption and extra triaging that it will involve.

    I can quite imagine this idea not happening for similar reasons – whether it’s opt in or opt out or whatever, it’s going to involve some disruption and extra work for at least some staff/developers, which means they’ll want to put it off until everyone has some spare time, which is never going to happen :)

  11. That sounds like a pretty good idea to me, even though I’m not much of a BugZilla guy, it wounds like it would work to me.

  12. I like this a lot. If the reporter (or someone else; in particular anyone on the CC list) thinks the bug should be reopened, they can do so. That nicely adds some ‘real intelligence’ (as opposed to AI) to the system, using a very simple feedback mechanism.

    I’ll be happy to receive relevant spam, if only to make me recheck bugs i’ve forgotten about ;-)

    Actually, it might be useful to do something similar for NEW bugs — they too can be inadvertently fixed, but still stay open —

  13. Like Greg I think that there should be a more clear way to inform the bug submitter not to file bugs with old builds. Bugs filed under Firefox are a good example, most of them are not filed with nightly build, only milestones.

    If I remember correct Bugzilla used to have an alert saying that the build was over 2 weeks old.

  14. I don’t see why it is any better to have automatically resolved bugs with a special keyword compared to just having UNCONFIRMED bugs. When a bug wasn’t confirmed for a long time, then it should stay UNCONFIRMED, because this is the reality.

    Having one more type of resolution and a new keyword makes things only more complicated.

    Using WORKSFORME is even worse, because this resolution should only be used when some people actually tried to reproduce a bug and failed – something an automated script obviously can’t do.

    What’s wrong with old UNCONFIRMED bugs at all? If you don’t like them, just ignore them. I don’t think they hinder the work on Mozilla. And automatic resolution adds no value to them.

    Automatically generating a comment in an UNCONFIRMED bug, that was inactive for a long time, is probably a good idea, because this actually saves some work (manually asking if there are any news about the reported issue). But I’m against automatic resolution.

  15. And I certainly don’t want to be forced to save the unconfirmed from such an action by confirming them.

    But that’s what confirming is for. If a bug is valid and has a testcase, it should be confirmed. We need to use our tools correctly if we are going to get maximum value out of them.

    Having said that, I suppose we could exclude bugs with a testcase keyword (good idea, Bill). But that should still be confirmed.

    Btw. speaking of reorg, what happened to Hixies bugzilla reorg?

    Unfortunately, the view among staff is that we’ve missed the boat here – it’s just too disruptive this close to Aviary 1.0 release. But we understand that a number of people consider this very important, and our intention is to do it ASAP after both Firefox and Thunderbird have released 1.0.

  16. What’s wrong with old UNCONFIRMED bugs at all? If you don’t like them, just ignore them.

    Having them open is bad in a few ways. If you are a new QA person, and are trying to triage UNCOs, you have 14,000 to choose from. Where do you start? Now, we happen to know that the older ones are much less likely to be useful than the newer ones. We should leverage that knowledge to whittle down the pile, and give people a practical and psychological boost in getting on top of the problem.

    Ideally, we’d keep the UNCO count stable or falling, and we’ve managed that in the past, but it’s got out of control. We need to bring it back into control, and then keep it there (using other measures).

    Automatically generating a comment in an UNCONFIRMED bug, that was inactive for a long time, is probably a good idea, because this actually saves some work (manually asking if there are any news about the reported issue). But I’m against automatic resolution.

    Say you have a six-month old UNCO. You add a comment, and no-one replies (not even the reporter). Is it best to spend limited QA resource trying to reproduce and triage that unclear bug, or to work on a just-filed one which has a much higher chance of being useful? So no-one should ever work on that bug, because there’s always something better to do. So why keep it open?

    In a perfect world, we’d triage them all. But we can’t, and it seems pretty clear which ones get the chop.

  17. Stefan, your comment leaves out that the bug got someone’s attention because you decided (apparently) to do some triage on it, and that it (as noted in the bug) appears to be a dup of a 2-year old bug marked NEW. So this proposed UNCO sweep wouldn’t have deleted the issue.

  18. Vague summaries

    Bug reports with a vague summary may remain uncofirmed:

    Users who observe the bug never find the unconfirmed bug in their search for previous reports.

    Developers/QA consolidating bugs on a particular topic never find the bug as relavant to the topic.

    So suggest adding something like the following to the ‘bug expiring’ warning message.

    “Make sure the bug summary mentions specific words from a relevant error message, command, or label so others who experience this bug can easily find this error report.”

  19. Note: the rate of UNCONFIRMED bugs may be unnaturally high because of bug 240093 (“Unable to confirm other people’s bugs, even with the canconfirm permission”). It looks like bug 240093 won’t be fixed until b.m.o upgrades to bugzilla 2.18.

  20. Hello all,

    I’m all for robo-resolving inactive unconfirmed bugfiles the way it was introduced here for many reasons. There are now over 8,000 UNCONFIRMED bugfiles at bugzilla (for Browser and Firefox products) and even if all trunks and builds were frozen forever, it would take at least 4 years to the current volunteers doing bugzilla triaging, confirming, etc..

    Many of old unconfirmed bugs are unconfirmable to begin with:
    – poor description, summary
    – vague, abstract data, absence or lack of specifics
    – very often with no testcase because creating a testcase is often impossible and here, I’m not even speaking of a reduced or minimal testcase
    – too many report/file a new bug and then run, remaining silent to comments or questions or ignoring bugmail, unaware of bugmail
    – sometimes, no url provided
    – if an url is provided, then you get to see
    o- a very long web page (over 50KB),
    o- with insane amounts of nested tables,
    o- tag soup,
    o- invalid markup/CSS code,
    o- IE-specific code,
    o- hair-rising amount of javascript code,
    o- scripts which were copied and pasted (so even the “webmaster” can’t even understand what he stole here and there)

    There is a need for robo-resolving inactive unconfirmed bugs because the mass of unconfirmed bugs is already overwhelming and dealing with many unconfirmed bugs is very much time-consuming.

    Whatever is decided about this robo-resolving inactive unconfirmed bugs, I know that there is (and will be) a great imbalance between the efforts of reporting a new bug and the efforts of trying to resolve each bug (whatever the solution: fixed, duplicate, invalid, wontfix). And with the upcoming of Firefox 1.0, the number of people willing to report a “bug” just can increase faster than the number of people willing to volunteer their time to resolve them. So you can and should expect that imbalance, that gap to grow even bigger with time.

    There has to be an “expiration date” set somewhat for these things.

  21. > We’ve decided

    Who is we?
    If that are drivers, did David accept a robot going through layout bugs?

    I don’t believe that the same approach that is good for mailnews, which has a long tradition of beeing swamped with bugs and not beeing able to handle this should apply to layout-tables. More than this there are some really delicate components like RDF, why on earth should we allow a robot go through this, because we have to many bugs in firefox-general? Please target the components that need this treatment like browser-general but don’t mess with the highly specialized components, where people spent already work on moving them into the probably right component.
    As I don’t tolerate people messing around with the bugs without understanding I will apply the same bad feeling to the kickout bot which is nothing else than somebody who kicks bug without having any clue.

    >If you are a new QA person, and are trying to triage UNCOs, you have 14,000 to choose from. Where do you start?
    This is a typical strawmen argument. First you choose one component where you would like to work. If you would like to confirm layout bugs you need firm knowledge of the corresponding specifications. Then the amount of available unconfirmed bugs, where you can apply your knowledge, shrinks immeadetely. Second believe it or not you can order bugs by number, so you will all the times know that you start on the fresh ones.

    I believe the problem is more that mozilla.org is not able to ensure that people will take responsibility over certain components and exercise ownership over this component. This problem increased since the netscape QA got dissolved. Instead of whining about the amount of unconfirmed one should ask do we have strong QA’s who will not accept a situation like this in their component. I certainly believe in ownership, where people take the responsibility instead of bots.

    The testcase keyword is not enough, the following query shows all unco table bugs that have a html attachment, did not change over 2 month and don’t have the testcase keyword.

    https://bugzilla.mozilla.org/buglist.cgi?query_format=&short_desc_type=allwordssubstr&short_desc=&component=Layout%3A+Tables&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=nowords&keywords=testcase&bug_status=UNCONFIRMED&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=exact&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=attachments.mimetype&type0-0-0=substring&value0-0-0=html&field0-1-0=%28to_days%28now%28%29%29+-+to_days%28bugs.delta_ts%29%29&type0-1-0=greaterthan&value0-1-0=60

    How would I feel if I attach a testcase to a bug and then comes some stupid robot and resolves the bug because I did not set the testcase keyword on the bug. It would be cool if one could add the testcase keyword when attaching a file. Or even better to have flag that this is a testcase which would then set the keyword automagically in the bug.

  22. Ludovic Hirlimann,
    there are no unconfirmed Mac OS classic bugs at the moment. We got rid of all of them and also of most of the confirmed bugs. At the moment there are still six open bugs assigned to system 8.5, 8.6 and 9.x

  23. Who is we? If that are drivers, did David accept a robot going through layout bugs?

    We is drivers, aviary and staff, all together. Everyone was CCed on the entire discussion, because it was felt to be very important.

    the following query shows all unco table bugs that have a html attachment, did not change over 2 month and don’t have the testcase keyword.

    And how many of those are people attaching a massive web page to a bug, where you’d spend an hour trying to make a reduced testcase only to find it was a duplicate of another bug which already had one?

    How would I feel if I attach a testcase to a bug and then comes some stupid robot and resolves the bug because I did not set the testcase keyword on the bug.

    For those small number of bugs, if it’s still UNCONFIRMED, you’d get two weeks to make a change (any change) to that bug. You can think of the incoming bugmail as a list of bugs to look at in the next two weeks :-)

    Gerv

  24. “This bug has had no comments for a long time. Statistically, we have found…”

    The message needs to be accurate in describing the scope of the sweep. If it is not, it will look like we’re throwing away all old bug reports, which will make bug reporters more angry than necessary. Replace the first sentence with something like “This bug has not been marked as confirmed and has had no comments for several months”.

    “Statistically, we have found that bug reports that have not been confirmed by a second user after two months are highly unlikely to ever actually be fixed”

    The message should say that the *report* is unlikely to lead to a fix, not that the bug is unlikely to be fixed. As you said, it is quite likely that the bug will be fixed or has been fixed elsewhere.

  25. “If a bug is valid and has a testcase, it should be confirmed. We need to use our tools correctly if we are going to get maximum value out of them.”

    But according to some folks, and some of the docs, it’s you that’s using bugzilla incorrectly. Some people/components (e.g. layout) operate on the principle that to be a valid bug, you have to know that there is something wrong in some particular area of code, rather than something that just “looks wrong”. When I was starting out with bugzilla, I was told that bugs shouldn’t be confirmed until they were moved into the correct product/component, and if it wasn’t clear exactly which bit of XP Apps or Layout or whatever it belonged in, then the report needed more clarification (or more expert triage) before it could be confirmed.

    I’m not saying that’s how things should be in the future (the “READY” state sounds like a solution to this issue), but we should recognise that the criteria for “NEW” aren’t well defined and different people are using it in different ways.

  26. Jesse: good points. I’ll clarify those things>

    michaell: also good points. I’ve come to realise that people are using the states somewhat differently (although I’d be interested to know of other components using it in the same way as Layout). And READY will hopefully help is to standardise usage more.

  27. I don’t know of other components that do it that way consistently (of course Layout is several components). I know timeless and ben chuang have objected to me/people confirming bugs (or attempting to resolve them from UNCO)

  28. >And how many of those are people attaching a massive web page to a bug, where you’d spend an hour trying to make a reduced testcase only to find it was a duplicate of another bug which already had one?

    in table layout: NONE, zero, Nullo

    I went trough all unconfirmed in layout-table that had a html attachment, and not the keyword testcase. 19 Bugs with 19 testcases without a single exception. Nineteen bugs that the robot would have swept. So practical evidence suggests that you need to refine your bot before it sweeps valid testcases. And it is also clear, who attaches html? The average user who is barely capable to work trough bugzilla? Not of course, that are triagers who know what is nessary in a bug, but forget to add the keyword testcase after they attached the file. A good query would in addition search for html attachments where the size is below 2k. I am not sure that bugzilla can search like this.

  29. Bernd:
    That’s 19 bugs you should add the keyword TESTCASE to RIGHT NOW! If you perform your own cleanup, then the robot will have nothing to do.

    Gerv:
    However, they do have a point. Trying to apply this to all components is likely to be a bit destructive. There are definitely some heavy-hitters that should go first. I think that this should be opt-in, with owners highly recommended to join.

    If you’re going to do a massive bugspam by adding comments to the bugs, won’t that change their date to be modified 2 weeks ago? How will you ensure that the comment is the last activity on that bug?

  30. You are proposing to close bug reports that report actual bugs. Just because there is no recent activity does not mean a report is bogus.

    I am tracking some 85 bug reports (just because I am interested in them). 27 of them are Unconfirmed. 5 of those have had no activity in 2004, but 3 of those are actual bugs (the other 2 being requests for enhancements).

    19 of the Unconfirmed bugs I am tracking have had no activity in more than two months. 6 of them are requests for enhancements; before they are closed, there should be an explanation of why the requested enhancements are rejected.

    I believe the proposal reflects a failure of the triage system. You will be declaring bugs fixed just to get them off the list depsite the fact that actual problems exist.

    Here is an example. Is bug #224335 not a real error merely because its last activity was almost a year ago?

  31. If you’re going to do a massive bugspam by adding comments to the bugs, won’t that change their date to be modified 2 weeks ago? How will you ensure that the comment is the last activity on that bug?

    By searching for all bugs in the original set which have not been modified since the date you added the comment. We can do this. The mechanics are the easy part :-)

    You are proposing to close bug reports that report actual bugs.

    They are certainly not necessarily actual bugs. Or are you asserting that all UNCOs are actual bugs?

    Just because there is no recent activity does not mean a report is bogus.

    Indeed not. Hence the request for people to say if a bug is still valid.

    I believe the proposal reflects a failure of the triage system.

    Absolutely. If we had enough manpower to triage them all, we’d definitely do that instead. But this chart says that we don’t. So this is the next best thing.

    Here is an example. Is bug #224335 not a real error merely because its last activity was almost a year ago?

    I don’t know. Does it still occur? If so, ask someone (perhaps the person who commented that he could see it) to confirm it. :-)

  32. BTW, I should add (in case it wasn’t clear from comments) that I’m quite happy to exclude all bugs with the testcase keyword from this sweep.

  33. Yes, bugs with “testcase” should not be automatically closed.

    I just reviewed my non-enhancement, Unconfirmed bug reports and retested five of them that had no action for well more than two months. These were all originally submitted against versions of Mozilla that are pre-1.7. All five are still problems. I then submitted comments that these are indeed problems for 1.7.3.

    I then reviewed the steps to reproduce the problems. In one case, I refined those steps. In the other four cases, the steps are still valid. Since I am a recently retired software test engineer, I believe those steps constitute test cases. Thus, I inserted the keyword “testcase” for each of the five reports. The comments that indicate the problems still exist also indicate the continued validity of these test cases.

    All of this does not address how “stale” bug reports requesting enhancements will be handled. The lapse of time does not render those requests invalid.

  34. David –
    If you’re able to reproduce them, then they’re confirmable as being bugs. If you can’t, try finding someone on IRC who will.

  35. I just reviewed my non-enhancement, Unconfirmed bug reports and retested five of them that had no action for well more than two months. These were all originally submitted against versions of Mozilla that are pre-1.7. All five are still problems. I then submitted comments that these are indeed problems for 1.7.3.

    Fantastic. The warning period we plan to have is hopefully to allow people still involved with the project to take action like this.

    Since I am a recently retired software test engineer, I believe those steps constitute test cases. Thus, I inserted the keyword “testcase” for each of the five reports.

    That’s actually rather an abuse of the word “testcase”, in my view. What you really need is the new “READY” state :-)

    All of this does not address how “stale” bug reports requesting enhancements will be handled. The lapse of time does not render those requests invalid.

    Well, sometimes it does. The feature to be enhanced may be gone completely, or the enhancement may be done in another way, or the reporter may have decided it’s not what they want, but not closed the bug.

    Asking people to re-verify they still want the feature is fair enough, IMO.

    BTW, David, I’ve given you the canconfirm and editbugs privileges. A retired software test engineer should certainly have these things :-) I’d be interested to know why you haven’t asked for them, because this probably represents a flaw in our mechanisms for “promoting” (I hope you don’t think using that word is patronising) people in Bugzilla.

  36. Gerv, I think that UNCO bugs that have never been looked at (eg with no comments other than by reporter) shouldn’t be “pinged” like this. They’ve clearly just been ignored by us altogether, and what you’re proposing would just add insult to injury from the reporter’s point of view.

    Perhaps we should do a sweep for such bugs first and evaluate them?

  37. Gerv I meant do them first. Get rid of the UNCO bugs targeted to unsuported platforms.

    I can’t see an advantage in doing them first – why not just let them get caught up in the sweep with the others?

    I think that UNCO bugs that have never been looked at (eg with no comments other than by reporter) shouldn’t be “pinged” like this. They’ve clearly just been ignored by us altogether, and what you’re proposing would just add insult to injury from the reporter’s point of view.

    Well, no comments does not necessarily mean never looked at; people may have read it, thought “what a terrible report; looks like a timesink” and moved on. But anyway, the point of doing this is that we don’t have the manpower to triage the massive backlog. If we did, we wouldn’t be here. I suspect untouched bugs actually make up the majority of the outstanding ones.

    I hope reporters won’t be insulted; if they respond, it will significantly increase the chances of someone actually looking at their bug.

  38. What is the goal here? What is this project trying to acheive for Mozilla.org?:

    * Is the goal to make Bugzilla.m.o research more useful, by improving the ‘signal-to-noise’ ratio?

    If so, I think this plan wouldn’t help much. These bugs are ‘low hanging fruit’, but only a small proportion of the ‘rotting’ fruit. I’d guess there’s just as much ‘noise’ or rot, if not more, in the (tens/hundreds) of thousands of stale bugs with status NEW

    We could accomplish the same thing by defaulting bmo queries to omit these ‘stale’ bugs. People using the ‘advanced’ query page can include them if they want. This would also address Boris’ concern (see above), that reporters’ efforts would be disregarded by mass resolving the bugs. Which leads to another consideration:

    * Is the goal to make triager time more productive?

    As Gerv said a few posts ago, Any bug which ends up FIXED is a great thing. Almost any other bug is a net drain on resources – we’d have been better off if it had never existed. I couldn’t agree more. I’ve stopped triaging bugs myself because so few get FIXED that it’s mostly a waste of my time.

    Unfortunately, there is and will be too much ‘noise’ in bmo no matter what we do. No automated solution can identify useful bugs — in part because what’s useful depends on what developers choose to work on — and there aren’t the hours to cull them manually. So how do we work in this ‘noisy’ environment?

    Direct triagers to specific, productive projects, instead of arbitrary, uncoordinated volunteer assistance: When a developer/drivers/etc, chooses a project, they could post a ‘help wanted’ for triagers to work on that particular project: research and compile info in bmo, build testcases, combine and cull duplicate and overlapping bugs, etc. The triager would deal with the ‘noise’; valuable developer time is wasted on triaging

    As I’ve suggested elsewhere, we could establish a simple webpage where triagers could find a list of what’s needed, posted by developers, drivers, etc. If I, for one, had a list like that, I might return to investing time in Mozilla.

    * Finally, is our goal is to make reporters’ time more productive?

    The fact is that these reporters’ efforts are already wasted — few of their bugs will lead to anything, whether we resolve them or not.

    We should deter all but the hard core from bmo and send what are essentially user RFEs and tech support requests to more appropriate forums: State much higher minimal standards for a bug: Steps to reproduce, on a current build, testcase, etc; state that bugs failing these standards will probably be marked INVALID.

    On one hand, we’d deter many bug reports and might miss something; OTOH, we don’t read all those bug reports anyway. Bugzilla ‘noise’ would be greatly reduced, and users would get better responses elsewhere to their RFE’s and support requests.

  39. > I suspect untouched bugs actually make up the
    > majority of the outstanding ones.

    You should check this (should be queryable). I agree that we have a lot of untouched bugs, but I doubt they are the majority.

    > I hope reporters won’t be insulted;

    What makes you have such hope? I know that I would be incredibly insulted by something like that….

  40. Gerv, just to clarify further, I’m very concerned about the ill-will this could generate. I really do think you and Asa are underestimating the impact of that….

  41. To illustrate what a wrong comment can do please read:

    http://bugzilla.gnome.org/show_bug.cgi?id=145283
    then the corresponding blog entry
    http://jwz.livejournal.com/361609.html
    Jamies blog seems to be pretty popular…
    and now look at https://bugzilla.mozilla.org/show_bug.cgi?id=235179
    and try to estimate how long this bug did not get any attention.
    And once you have seen this, please make an estimate what kind of friendly comments in the blog scene this action may create.

  42. I would like to suggest an alternative for clearing stale Unconfirmed bug reports.

    1. Pick a product (e.g., Browser) and create a query to list all Unconfirmed, non-enhancement bug reports with no changes over that past 6 months. If that is too many, pick a product/component combination (e.g., Browser with all variations of Layout). You need a query that will return a large but not overwhelming list. The product/component combination in my example has 118 bugs.

    2. Identify a triage team within the Mozilla “regulars”. The team’s members should be added to the cc: lists of all bugs in the product/component combination during the review period.

    3. Issue an E-mail call for reviewers to all individuals who have Mozilla.org Bugzilla accounts but who are NOT regular developers or testers. Let the reviewers know the URL for running the query.

    4. Reviewers are requested to review any bug as long as that person did not report the bug. A review should involve the following:
    (a) Is the description sufficiently clear?
    (b) Does the description define a real problem, does it describe something inherent in the design, or is this really a request for an enhancement?
    (c) Is there sufficient description (e.g., Steps to Reproduce) to allow someone to develop a test of any fix to the problem?
    (d) Can the problem actually be reproduced?
    Not all four of these need to be addressed, but at least two should be.

    5. As a reviewer finishes a review, the bug report is updated to comment on the review results. This removes the bug report from the query (new activity) so that other reviewers don’t review it.

    6. Triage team members will automatically receive messages with the review comments. They can then make informed judgements whether a stale bug can be marked Resolved or New.

  43. David: what you are basically saying is “get people to triage them”. However, we’ve proved conclusively that we don’t have the manpower at the moment – and all the organisational procedure won’t help that.

    In addition, I have attempted to show statistically that the bugs we are targetting with this effort are those least likely to be useful. If there were such an effort as you describe (and Bug Days are not far from it) then we’d be targetting the bugs most likely to be useful, not least.

    That’s not to say we shouldn’t try to get more people involved – we should. This is how we keep the number under control when we’ve got it down. But we shouldn’t start new volunteers off wading through 10,000 mostly-rubbish bugs either.

  44. I definitely think the UNCO bug cleanup is a good start. The wording of the message, however, needs to be carefully crafted, taking into account the above comments.

    “This is an automated process meant to clean up bugs in an Unconfirmed status that have grown stale. Please review this issue and determine if you would like it to remain open. You may close the bug yourself, let the automated process handle it, add supplemental information, or get someone else to confirm your issue and mark the bug as New.

    Your input is very important to us, but unfortunately our resources are limited to us. We greatly appreciate your assistance in focusing our efforts. Thank you.”

  45. alanjstr,
    I personally feel that the wrong wording would hurt us a great deal. And to be honest your wording sounds very useful to me. Though we need to have the automated message mention “the url in this message” imo, since many will receive the message through e-mail, and wont know how to access the bug easily.