On Monday August 29th at 6.30pm BST Mozilla was informed by Google about a misissued certificate for *.google.com which was being used in active attacks on users in Iran. This certificate was chained to the root of the Dutch CA “DigiNotar”. Since that notification, I have been part of the Mozilla team working on our response.
DigiNotar discovered evidence of compromise of their systems on the 19th of July, but decided not to inform embedders of their root, including Mozilla. We have now been given details of 247 certificates, covering 23 CNs, which were misissued around this time, from a number of different DigiNotar intermediate certificates, including their EV intermediate. (These are the ones Chrome has explicitly blacklisted.)
Mozilla has a spreadsheet of certificate data from the certificates, but not copies of the certificates themselves. It seems that the attackers tried to make their certificates as like the genuine ones as possible by filling in the correct company names and locations. This data suggests that misissuances continued to occur for at least 1 day after the date DigiNotar tell us the compromise was discovered. These 247 certificates were all short-life, and expired by the 19th of August. The data we have contains revocation dates (in some cases, from as late as 27th July), but we have not been able to confirm that the certificates actually were revoked. Given that they have now expired, the revocations could perhaps have been removed from the CRLs. However, a survey of existing certificates on the web suggests that it was standard DigiNotar practice, for all their certificates except for their EV certificates, not to put any revocation information in their certs – either CRL Distribution Points or OCSP servers. Given that the attackers may well have used standard DigiNotar certificate profiles to generate the certificates, it is highly likely that many of the rogue certificates did not contain revocation information. However, DigiNotar have been unresponsive to requests for clarification on this point.
The CNs concerned were as follows:
DigiCert Root CA
Equifax Root CA
Thawte Root CA
VeriSign Root CA
If that had been it, we might never have known. However, it has now emerged that DigiNotar had not noticed the full extent of the compromise, because for one particular intermediate certificate (the DigiNotar Public 2025 CA, the “2025 intermediate”), the attackers had managed to hide the traces of the misissuance – perhaps by corrupting log files. It is from this intermediate that the *.google.com certificate which recently came to light was issued. This certificate was issued on the 10th of July, a week before the 247, and is not a short-life cert. Up to now, due to the lack of logging, DigiNotar has been unable to determine how many certificates were misissued from this intermediate, or what their CNs or serial numbers were. (And not knowing their serial numbers makes it impossible to revoke them.) From looking at OCSP requests for unknown serial numbers, it seems there are at least 4, but there could be many more. This, to me, shows a greater level of sophistication; it is at least possible (but entirely speculative) that an initial competent attacker has had access to their systems for an unknown amount of time, and a second attacker gained access more recently and their less subtle bull-in-a-china shop approach in issuing the 247 certificates triggered the alarms. [Update 2011-09-05 18:10: given the patterns of issuance emerging, this now seems less likely.]
After we were informed by Google of the problem, we attempted to contact DigiNotar. We were unable to open a channel of communication with them until the middle of the following morning, European time. When I called them, I was not particularly impressed at being told I would be called back, and then not being, and also at being referred to a PR person for Vasco, DigiNotar’s parent company. Incidentally, it is my personal view that public statements by both Vasco and DigiNotar (translation) about the extent of and effect of this compromise have been, at best, incomplete and at worst actively misleading.
After consultation among the Mozilla security group, it was decided to remove the DigiNotar root certificate from our trusted root store, and make sure that all certificates issued by them were no longer trusted. We decided to take this action based on the fact that we no longer had sufficient confidence in its trustworthiness. A number of factors over the past two days have contributed to that loss of trust:
- We were not informed when the breach and misissuance was first discovered in mid-July
- Misissuance occurred from multiple intermediates, suggesting a wide compromise
- Dating evidence suggests that DigiNotar’s response to this initial compromise was not sufficiently robust or complete
- Six weeks of investigation between then and now were not enough for DigiNotar to discover the additional misissuances from the 2025 intermediate; it was revealed only by an active attack
- Their internal systems do not have sufficient logging to know the number of misissued certificates from the 2025 intermediate, or their serial numbers, or the names of the sites concerned
- Probable lack of revocation information in some of the certificates
- Their website was hacked 2 years ago, multiple times, and they didn’t notice
- They have issued a press release containing statements which were, at best, misleading, concerning the extent of the breach (it does not mention the initial breach, only the issues with the 2025 intermediate)
- One statement told their users it is 99.9% safe to override browser certificate warnings
Various DigiNotar intermediate certificates had been cross-signed by other trusted CAs. In order to achieve full blocking, we implemented code which checks for DigiNotar’s name in the certificate chain. Any certificate issued after 1st July 2011 is blocked with no possibility of override. Any certificate issued before that date is blocked, but the user has the option to override the block and visit the site anyway. Those other CAs have now revoked the intermediates, and our upcoming patch will specifically add them as “untrusted” in our certificate store.
DigiNotar is also a participant in the “PKIoverheid” (“PKIgovernment”) PKI run by the Dutch Government, the certificates for which chain up to one of two root certs owned by that government – the “Staat der Nederlanden” roots. These are also trusted by Firefox, having gone through the inclusion process in their own right. DigiNotar controlled two intermediate certificates in this hierarchy (one for each root), and issued SSL and personal certificates from them.
Our initial patch also blocked DigiNotar-issued certificates below this root. However GovCERT, the Dutch CERT (“Computer Emergency Response Team”), indicated that they had made an assessment of the DigiNotar systems and it was their belief that the machines used for issuing certs in this PKI were not compromised. Therefore, the Dutch government requested that these certificates in their PKI continue to be trusted, and assumed responsibility for that decision. We decided to grant their request, and so added a few lines of code to exempt certificates in this separate hierarchy. (In fact, this code had a bug, and did not cover both of the roots in question.) This action was in line with what Google and Microsoft were doing at the time, although, because they used different technical means for the block, they did not need to write specific code for the exemption.
It has recently emerged, due to a report from a 3rd party security company, Fox-IT, that this confidence was misplaced, and the next Mozilla security update will remove this exemption. There are questions here for the Dutch government and, particularly, GovCERT, as to how they felt confident to give us the assurances they did. We were offered some of them in writing; in hindsight, we should have taken them up on that offer.
In order to reduce the impact of this removal, the EFF SSL Observatory data was used to gather a list of sites who might be using affected certificates from the DigiNotar root, and crowdsourced an effort to contact the webmasters and send them a warning email. Thanks to the Mozilla Dutch localization team and Bits of Freedom in the Netherlands for their help with publicising this. By the time we decided to extend the block to the Staat der Nederlanden roots, DigiNotar and the Dutch government had already started contacting affected sites, so we have not expanded this effort.
Note: I am at a wedding today and so may not be around to answer questions about this before Monday.