There’s been a bit of confusion about the point of the process for auto-resolving old UNCONFIRMED bugs. This is mostly my fault, as is what people see as the lack of notice – we’d been discussing this for so long that when it finally happened, I didn’t lift my eyes up enough to see that I should have made more noise about it than just posting on my blog. Apologies for that.
So, why are we doing it? The logic goes as follows:
- We don’t have enough manpower to triage all the bugs in Bugzilla. This is abundantly clear from the charts. So, we have to concentrate our efforts on a subset.
- Bugs rot. They become fixed by some other change, the product changes so the enhancement is no longer relevant, the reporter discovers it was his configuration, people change their minds about an enhancement, etc. etc.
- Statistical analysis showed that if a bug is not confirmed after three months, it’s highly unlikely to end up FIXED. We’ve got 96% of the value out of the bug pile by this point. So the remaining bugs, as a group, are the least useful bugs in the system.
- Given point 1, the bugs from point 3 are therefore the ones QA should not be spending time on. To triage them by hand is an inefficient use of time.
- However, we want to try and extract some value from those bugs also. So, the plan was to offload the work by getting the reporters to tell us if the bugs are still valid. Hence the auto-UNCO proposal.
So what does this mean?
- There are currently 18,000 UNCO bugs. We are doing the process on 12,500. If, say, 11,000 of them get closed, we’ve focussed QA effort in on what is roughly the best 2/5 of the UNCOs.
- If you are a longtime contributor or module owner, and you’ve got a lot of bugmail from this process, it’s not good to spend a lot of time carefully triaging each bug. The entire point of the process is to avoid that. If your pile is manageable and you want to whiz through them quickly, go for it. But don’t let this land you with a load of extra work.
- People will comment to say “this bug is still good”. If this happens, it’s a good thing, because it means that bug is far more worth spending triage time on than either other auto-UNCO bugs or even new UNCO bugs. It helps focus QA effort.
- Yes, the process may result in several bugs being eventually closed which are good reports of actual problems. But, given the lack of manpower, and the benefits of being able to focus on a small and better quality bug set, I think that’s a good trade-off. After all, without this, those bugs would have just sat there unlooked-at anyway.
- Once the process is complete, QA should spend time looking at the bugs people have commented on; because these are much more likely to be good than the ones people haven’t commented on.
One quick clarification: the bugs will be RESOLVED EXPIRED, so they are easy to find if you need to.
As a side effect, the implementation of the first part of the plan has (re)thrown up a need for workflow improvements to Bugzilla. Ideas for this are being tracked in the wiki. It’s now part of my day job to make developers’ lives easier, and so I’m going to be looking at those suggestions very carefully indeed, with a view to doing something about them. Please contribute to that page if you have ideas.
I want to have a good go at convincing skeptic module owners that this is a good thing. So email me if you think it isn’t, and I will. However, if I can’t convince you, then I will exclude your module(s) from the second part of the plan. After all, they are your modules.
Gerv,
Thank you for posting this information. I’m pleased to learn that bugs closed by this process will be RESOLVED EXPIRED. It addresses my main concern with this process as I will still have a way to search the entire set of “open” bugs.
Gerv, I think the focus on UNCO is one part of this plan that confuses me… The average quality of NEW bugs that haven’t really been looked at (and we have plenty of those) and UNCO bugs that are still open after a few years is about the same, in my experience…
I completely second what Boris said above. I just don’t see the point. It makes no difference from my point of view to have UNCO “cannot in editor move caret from right to left of image using arrow key” or NEW “cannot in editor move caret from right to left of image using arrow key”.
The bug is here, visible from both the developers and the users, and that’s what finally matters.
As you said, we lack manpower. Not only on QA. It means that the triage we make, and the priorities/milestones we set are very often approximative, to say the least.
I got a rough 200 bugmails in less than 24 hours, coming from both your automaton and answers. This is not manageable. We’re not all Hixie-observes-everything-reads-everything ;-)
Speaking of Hixie, has anyone seen him recently? I think he is under that huge pile of bugmail ;-)
Daniel: the big difference between the two bugs you mention is that the NEW one has been lifted out of the pile of rubbish consisting of all the other UNCO bugs. If you know that bug is a real bug, it should be NEW (possible process improvements and new states aside).
Bugzilla has a number of states for a reason – to help us track and manage bugs. the UNCO state is for bugs which are filed by people who aren’t regular bug filers (and so who don’t have canconfirm) and so has (or should have) a much higher proportion of rubbish bugs – dupes, invalids, what-on-earth-are-you-talking-abouts. The original idea was that developers would not concern themselves with UNCO at all, and just look at NEW bugs – it was a way of reducing their workload. This distinction seems to have been forgotten along the way.
the next time you add comments like this, can you make it clearer that people should _not_ test using the latest stable release? in various bugs, the reporter confirmed that the behaviour still exists in 1.0.7/1.7.12, which is really a pointless exercise.
biesi: I did add specific download links to 1.5b2 at the bottom of the email… There’s not much more I can do! But I’ll try and tighten up the wording.
> the big difference between the two bugs you mention is that the NEW one has
> been lifted out of the pile of rubbish
That would be true if people with canconfirm didn’t file their rubbish as NEW by default. Given that they do, there’s really not much distinction between a NEW no one competent has looked at and an UNCO no one competent has looked at.
> and so has (or should have) a much higher proportion of rubbish bugs
That’s been pretty much false every single time Asa has run the numbers. The differences between general UNCO and general NEW were a few percent at best. The differences between the best and UNCO bug filers were about 15-20%, with the best people having 50% “real” bugs while the UNCO was at about 30%. Did this change significantly with Firefox? I’d love to see the new numbers (which we can get out of bugzilla) instead of vague guesses based on should-bes. Here by “rubbish” I mean things that end up dupe, invalid, etc. I’ll agree that UNCO has a somewhat higher incidence of the “what are you talking about?” bugs, but even then it might not be by that much — we have plenty of people with canconfirm (some of them even developers) filing such bugs with various frequencies.
Name names! Or even bug numbers. By email if you like.
Asa’s numbers are related to recently-filed bugs. The two sets of stats are not comparable. My point is that, after three months, almost all of the decent bugs in the UNCO pile have been confirmed. See my stats, linked, for details – but basically, the chance of a bug ending up FIXED after it’s been UNCO for more than three months is pretty tiny.
Gerv, almost every single person I know of who has canconfirm has been guilty of filing crap bugs that really should have been filed as UNCO at one time or another (myself included here). The one exception might be timeless, who makes an effort to file bugs as UNCO when he really doesn’t have enough information.
If you really want me to spend a few days digging up specific bugs, I can do that. I’m just not sure what that would help.
And yes, I did see your data on UNCO. What I’m saying is that the data for NEW would likely look similar — the chance of a bug that’s been NEW for over three months being fixed is not so hot if you just look at the numbers.
As for good UNCO bugs being confirmed, that depends on the component. Again, in all the layout components we have actively been discouraging QA from confirming bugs unless they have a minimal testase. This has been by far the best use of the UNCO status we could make in those components. But as a result there are plenty of real bugs that are UNCO just because they have no testcase yet.
Boris: I can see your point about the UNCO focus; however, we need to start somewhere. Once we have a READY state and it’s been in use for a few months, it may be appropriate to do a similar thing for NEWs which are very old – say, nine months old.
I have been a frequent bug filer, but have never got the canconfirm or if I did it was taken away from me.
There are several types of bugs that may be unlikely to get confirmed:
* web evangelism bugs
* bugs that have unique configuration issues but still should be fixed (ie. a specific operating system or with specific operating system feature, in certain networked enviroments)
* bugs without sufficient details
I think you should leave the evangelism componet alone.
–Sam
I have to say, from a non-BugZilla perspective, and just from a tech support and issue triage perspective, this is a perfectly logical move, and I applaud it. “Old unresolved” is one of the hardest classes of issue to fix, for all of the reasons you suggest. If you don’t pester the poster about the report, it really will sit there forever. This is a manpower-relieving way of group-pestering and getting useful feedback from the opt-back-in process, both on which bugs are worth fixing, and on which heuristics need adjustment in determining genuine orphaned issues. Learn by doing.
Excellent resource management. :)
Two questions:
1. If a bug is updated later on the same day that you add your automated comment, is that considered “changed since the date the comment was added”?
2. For bug reports that request enhancements, why should a test case be necessary?
Gerv: Inre: your comment to Daniel,
“the big difference between the two bugs you mention is that the NEW one has been lifted out of the pile of rubbish”
Just exactly who makes that determination as to what is garbage is a very important point. Many unconfirmed bugs IMO are the result of someone defacto acknowleging the bug, but determining that his prefs are such that that particular capability should not even be there in the first place. When in fact the bug is indeed valid.
Case in point: https://bugzilla.mozilla.org/show_bug.cgi?id=253435
This should have been marked as New then perhaps WONTFIXME
If that is even an official category.
Sam: Apply for canconfirm. We didn’t do Tech Evangelism; I believe they opted out.
David: 1) Yes. 2) It’s not – who said it was?
Re: Test cases for RFEs.
The criterion for closing unconfirmed bugs indicates that such bugs will not be closed automatically if they have the keyword “testcase”. In general, this means that unconfirmed RFEs will be automatically closed since RFEs usually do not have test cases.
Perhaps RFEs should be handled with different criteria.
As one who has received three of these notices, I also applaud this effort. I was able to re-check these bugs and determine that they are no longer valid due to evolution of the browser.
Good plan, IMO.
I’d like to apologize for the semi-templated comments I made to some b.m.o bugs that were resolved as FIXED as, I thought, a results of this automated process. I didn’t realise (until after I’d dealt with several of these) that it wasn’t your automated process that was resolving the bugs, but other people in response to the automated process putting its comments in place. (I’d said that, perhaps, the bugs should be automatically resolved as WORKSFORME instead.) Having a completely different automatic resolution (EXPIRED) is a good thing and clearly dilineates between that and other resolutions.
This is all a good thing. It only server to prompt people to say a “negelected bug is either “still alive” or to deal with it properly.
“It only server”? Too late to correct the typo! I can only laugh at myself now. (I’ll try not to notice the other typos, of which there is at least one more, since they aren’t actually funny…)
Nice nice. A lot of useful here. good work
Thanks for all.
online
I’m with DR – “Perhaps RFEs should be handled with different criteria”.
For bug reports that request enhancements, why should a test case be necessary?
hi i want to make following changes in Bugzilla.
1) Versions option should have active version as 1.3 , 1.1, 1.2 etc versions
should be available only for searching purpose in reverce order (should not be available for creating new bugs or modifying old bugs).
2) “Target Milestone” date should always be >= the current date. A user should not be allowed to create bugs with “Target Milestone”
Arpan: See the Bugzilla support pages for the right place to ask such questions.