Automatic Resolution of UNCONFIRMED Bugs

We are going ahead with the plan to automatically resolve some old UNCONFIRMED bugs. We will be taking the first step (issuing the warning) on the morning of Sunday 10th April, UK time (the middle of the night in the US).

This idea has been discussed previously on this blog. We’ve managed to address the major issues raised in that discussion; we’ve got a new EXPIRED resolution in Bugzilla, the language of the comments has been improved, and we’re excluding bugs with the ‘testcase’ keyword. I’m open to further suggestions, as long as they don’t delay the process. In the end, even if we can’t get it 100% right, the pros outweigh the cons.

20 thoughts on “Automatic Resolution of UNCONFIRMED Bugs

  1. Let’s say I have filed a good bug report, with testcase, talkback, the works. My bug then gets closed as a duplicate of another bug filed much earlier. This other bug however doesn’t have a testacase. This bug would get marked as expired, even though I feel it shouldn’t.

    Now, the questions is how many bugs have duplicates of them and remain unconfirmed? Mind you that this probably shouldn’t happen, and so a script marking any bug that has a duplicate and is unconfirmed should probably be marked as new instead.

  2. It would be nice if UNCOs which depend on another bug could be excluded. But probably this won’t work (at least not with the UI), since you can’t search for those type of bugs, see Bug 103866. Probably we should confirm those UNCOs for now then?

  3. Gerv,

    “and which have not changed since the date the comment was added.” With regards to that comment, what if for some reason a person does something like remove a CC (for instance, “I don’t want to get email on this anymore”), but take no other action. Will that bug still be auto-resolved?

  4. mvirkkil:

    Bugs with “testcases, talkback and the works” should not be duped to bugs without. If one bug has “the works” and the other is older but unconfirmed, then the older one should be duped forward.

    In any case, I’m sure that such cases are rare. As Gerv has said, it won’t be 100% perfect, but the benefits absolutely outweigh the drawbacks.

  5. The info on the page doesn’t say, but I think you should exclude all bug in “enhancement” severity.

    I have seen many enhancements lying around for ages since:
    a) developers tend to look at confirmed bugs first

    b) enhancement request tend to come from “new” users that does not have the “canconfirm” right

    c) “non core” QA people leave them for the developers to say if this is fix or won’t fix, and therefor leave them in uncomfirmed.

  6. Gerv, the proposed comment text talks about 3-month comment cycles and points to /products as the place to get the “latest version”… but for Firefox the versions available there are about a year old in Gecko terms. Perhaps we should only do bugs that are more than a year old in the first sweep? Or point to a reasonable place to get trunk Firefox builds?

  7. Boris has basically beaten me to the comment I was going to make. 3 month product cycles were a Seamonkey thing until 1.7 – they don’t apply to anything now, and never have applied to the other products.

    People filing bugs against the latest releases are (sometimes) getting told they should try a nightly – the automatic comment should probably do the same thing. There is already plenty of confusion where people are watching “FIXED” bugs and finding them not fixed when they install a 1.0.x security release.

  8. I’m not saying the idea here is necessarily bad, but the statistics are lacking in terms of showing that this would be effective. Specifically, I would expect the important metric to be “what fraction of X-month/year old UNCONFIRMED bugs wind up with a FIXED resolution?” The statistics you’ve collected are “how long [FIXED bugs] took to get confirmed”. I can’t come up with a solid relationship between what you’re measuring and what I would expect to be important. It could be that the metric I mentioned would also support doing robo-resolution (maybe even more than the one you used), but many filed-as-unconfirmed bugs get duped quickly, so it’s possible that old unconfirmed reports actually have a higher probably of being valid than new ones.

  9. Also, related to previous comments about the 3 month product cycles, users will not find any build useful for testing the bug again at the URL you’re giving them,

    Suite 1.7.6 and FF/TB 1.0.2 are probably what many of the reporters were using when they filed the bug. They need Suite 1.8b1 or a nightly aviary build (since no milestones have been released since 1.0).

  10. In terms of warning, will you be placing a notice in bmo a few days ahead of time indicating that bmo will be closed for t length of time to do this? I imagine that not all Bugzilla users, and probably most “affiliated” with the projects not in the auto-resolve set, don’t follow the blogs and newsgroups where this will be announced. Just a line or two like when Bugzilla was upgraded…?

    (There are only two UNCOs in Camino unchanged in the past three months, and two-thirds have been changed in the past week, thank-you-very-much :D)

  11. mvirkkil: There are 461 UNCONFIRMED bugs in the relevant products which have had dupes marked of them. However, having reviewed a few of them, I don’t agree that because another bug has been marked as a dupe of a bug a long time ago, it means that bug is much more likely to be useful now.

    mcsmurf: if an UNCO depends on another bug, then someone obviously thinks it’s a real bug, and so it should be confirmed. Remember, the first thing that happens is a warning asking people to retriage.

    Neil: No, such bugs would not be auto-resolved. But, given that such bugs would not have changed for the past three months, the chances of anyone getting dissatisfied with bugspam from them is fairly minimal!

    Robin: We don’t know, because we don’t know how many of the ones we initially target will get confirmed/commented upon. If we did it today, the initial target comment would be put on 9848 bugs.

    Henrik: I think it’s reasonable to ask people to confirm that their enhancement still applies. After all, many things could change to make the request unnecessary. And enhancement bugs have a tendency to hang around for ages, because only the reporter can really close them. FYI, 1181 of the aforementioned bugs are enhancements.

    Smokey: we’re still working out the best way to disable Bugzilla in order to do this, but yes, we will put up some warning – just like with any downtime. BTW, after I emailed him, pink asked that Camino not be included in this – probably due to your excellent work ;-).

  12. I’d say wait for the preview releases – if you’re going to do something this big, you may as well do it properly… ;-)

    I’m also predicting that whoever does this, is going to receive a lot of hate mail over it.

  13. Glad to hear there’ll be a warning in Bugzilla, too; I haven’t been around long enough to be familiar with other downtimes besides the latest upgrade :-)

    (And, lest anyone be confused by my earlier attempt at humor, it’s not only me attacking those Camino UNCOs; there’s a nice group of four or five of us who go through serially and triage….)

  14. It would seem to make sense to track the end user product release dates: Do bugs from before June 2004 (before either 0.9) on the first run, and do another for bugs before November 2004 (before either 1.0) run after the 1.1 releases.

    …which comes to about the one year mark. Guess I like to think conservative when it comes to mass changes…

    As an aside, I tend to read “This bug has had no comments for a long time” as an invitation to spam to keep bugs alive :\

  15. I think this is a great idea and one that should have been enacted as a policy precedure and done, minimally, on a yearly basis. Without doing this periodically I’ve often wondered how MoFo QA ever had any confidence in assessing the quality of the product(s) with all the stale UNCONFIRMS (UNC)tilting the bug part of the quality assessment in the negative direction. Perhaps UNCs were left out in these statistic gatherings, in which case, tilts the quality assessment incorrectly in the positive direction.

    One point, however. There are whole categories of bugs that very often never get reviewed because they are considered very low on the priority scheme, skinability bugs are an example of these. As a theme writer, I’ve reported several of them (some several years old) that were never reviewed yet are still very much a problem (at least the last time I looked.)

    Getting this message on these bugs will provide me the incentive to re-test them to insure that they are still relevent, however, my expectation is that if I re-confirm the validity of the bug (via the url provided in the message), then it will not be closed nor will I, or others, be required to fill out a new bug.

  16. This should happen to “new” bugs after a year! I have a few bugs in my list, which are new but WFM and realy old. Unfortunately I do not have the right to close them and the reporter doesn’t answer. Pinging bugs regularly would be a great thing. Uncos after 3 month and new after a year or so. Doing this automatic in bugzilla would be eaven better. ;-)
    Sorry if this is rubbish. I’m only a pro-user – no dev.

  17. Option 2 sounds like the best idea to me, provided that it doesn’t mean that the whole thing fails to happen because of the release crunch (i.e. drivers and people managing the releases should be signed up to this beforehand). I think either 1 or 4 would also be ok – waiting for final releases would mean a long delay, and if people are involved enough to be reporting bugs it’s probably reasonable to expect them to use alphas/betas.