The Right Measure

Now this is the most sensible measure of the relative security of two browsers. Ask the question “For how many days was I at risk from a published exploit to a known bug?”.

In 2006, the score was Internet Explorer: 286 (78%); Firefox: 9 (2.5%).

7 thoughts on “The Right Measure

  1. To be fair, it would be interesting to have these metrics :
    – How much time between Microsoft being notified about a security issue and the said issue officially (or not) published.
    – Same for Firefox.

    Seeing how some security issues have been sitting in bugzilla for *years* (with restricted access), and seeing how people may be more inclined to release security issues for Microsoft products in the wild, even before Microsoft itself gets notified, that would make these results much less interesting.

    Firefox has this advantage that most of the security issues that each new version fixes are published the day the version fixing the issues is released.

    If Firefox security bugs were getting published as Microsoft security bugs were, I doubt the gap would be so large…

  2. I agree, but when people say “IE bugs have been published”, there are two levels : a publication by a third party (hacker site, blog, etc) and a publication by Microsoft. Clearly, in case Microsoft publishes first, there’s no third party publication to take into account.

    It’s the same for Mozilla development (including Firefox) : anyone can publish exploits about Mozilla (it’s been the case recently on the myspace.com form issue), but if no third party publishes anything, then Mozilla developers are free to reveal exploits or not.

  3. Xandrex: Yes, other metrics would be interesting:
    – Number of bugs published in the wild by hackers and alike for IE, and how much of the total number of bugs published by Microsoft it represents.
    – Same for Firefox.

    I bet the gap between the two resulting percentages is huge.

  4. glandium: “bugs which have been sitting in Bugzilla for years” are no threat to anyone until and unless they are independently discovered by a black hat and they begin to exploit them – at which point, the statistics above would start counting “vulnerable days” immediately.

    I don’t understand the distinction you are trying to draw between the Firefox and Mozilla processes. Both companies ship browsers. Both companies have some bugs reported directly to them, and both companies find out about other bugs by third party publication. (Bugs reported directly to the company can also be published publically later if the original discoverer is dissatisfied with progress.) Crackers attempt to develop exploits to published bugs in both browsers. Both companies ship fixes to bugs. Presumably, both companies prioritise fixes to holes being actively exploited.

    Yet the number of “insecure days”, between publication by someone and a fix being released, is 286:9.

  5. Further, wouldn’t the gap grow if we take into account what glandium is saying? How many bugs in a closed system such as MS that hackers do not have access to the source code are in MS’s IE tracking system and have never been fixed or published? Who knows but my guess is that it is measurably worse than the amount for Mozilla (not a difficult guess judging by their track record). I think Mozilla would welcome an audit of how long vulnerability bugs have been sitting on their system v. similar ones on MS. But since that will never happen, published exploits is the only way it can really be audited by an independent party.

  6. I think what glandium is saying is that he thinks people are more likely to privately report security bugs to Mozilla because of the reputation that Mozilla fixes bugs quickly — is there a gap because IE has more bugs and MS fixes them more slowly, or is it because people are more likely to publicly disclose IE security vulnerabilities.

    Another point is that the 9-day window for Firefox was for a bug that never actually had a publicly disclosed exploit. The DOS was publicly known, but I still know of no exploit to execute arbitrary code (which is what the security advisories claim). Or perhaps it’s related to bug 331981? It was mentioned in bug 334515 as being very similar and to have been fixed by the same patch. But 331981 is still closed for reasons I won’t be able to understand…

  7. If Microsoft has a poor reputation in the security community and therefore people are more likely to publicly disclose IE bugs than report them quietly, that’s their problem. They’ve made their bed, now they have to lie in it.

    That’s why I, at any rate, would take very seriously a suggestion from a bug reporter that he or she was poorly treated by the Mozilla security process.