You may recall we are doing some work to try and clear the backlog of old requests (review, super-review, feedback etc.) This first involved sticking a CSV template on bugzilla.mozilla.org so we could monitor the number and age of outstanding requests. Since mid-December, I’ve been downloading a full dump of the outstanding reviews each day. Then, in mid-January, we implemented a gently nagging email, to be sent every week, reminding people about their requests older then 7 days. I have now written a script to analyse that data, to see if the email has had any effect.
Here’s what happened, looking at the most common type of request (“review”), in terms of the number of review-days outstanding:
The first noticeable downtick in the graph is just after the first nagging email got sent out, and as you can see, the effect continued weekly for about a month, before vanishing, and the level plateaued again. I would interpret that as an indication that all the low-hanging fruit is now gone. And there’s still a lot of outstanding reviews – 928 older than 90 days (down from 1192).
We reduced the number of outstanding review-days. But was this due to people dealing with some old ones, or were people getting more timely with newer ones? If we do the graph again, this time ignoring all reviews older then 30 days, we can see that (if we assume the number of new review requests per day is roughly constant, which as far as I can tell it is) there’s not been much reduction in waiting time:
So what do we do now?
I think we want to get to a place where no request is outstanding more than 30 days, and where someone with contributor engagement can monitor requests as they cross that threshold and try and see why and when something went wrong. On average, 4.4 reviews a day cross the 30 -> 31 day threshold. As reviews are the most common type of request, I’d suspect it’s about 5 requests a day.
Here are some options to clarify the situation and make it better in the future (of which we could do more than one). Note that here I am talking about requests which have the ability to take an explicit requestee (such as “review”).
- Ban new requests “of the wind” (ones without a named requestee)
- Ban new requests of requestees who have not been active in Bugzilla for > N days (suggested N: 30)
- Cancel existing requests of requestees who have not been active in Bugzilla for > N days (suggested N: 30)
- Cancel existing requests on VERIFIED bugs (are there any which could validly be on such bugs?)
- Cancel existing requests on RESOLVED bugs (are there any which could validly be on such bugs?)
- Cancel existing requests which have been outstanding > N days (suggested N: 30)
- Find someone to examine requests which cross an N-day boundary and investigate them (suggested N: 30)
All bans would be accompanied by advice on finding an appropriate reviewer. All cancellations would be accompanied by an apologetic comment and useful advice for the patch submitter about steps to take next.
 Note: I’m no statistician. I’m happy to give my collected data to anyone else who wants to do some analysis. Just email me. Anyone who wants a CSV dump of all current requests, it’s here.