Data Backup Digest

Microsoft were recently left red faced when they had to reach out to their certificate authority affiliates and ask them to send in their own copies of audit data. This was following a Microsoft system crash which resulted in data loss.

In total, Microsoft lost audit data for 147 roots following a system crash. This then meant that a number of affiliates were temporarily without certification, meaning that automatic query emails were sent out to them to warn that they were to be removed.

In its initial steps to try and repair the situation, they sent out an email to all of their certificate authority affiliates to let them know the situation.

“As many of you may have just noticed, our system just generated a bunch of emails informing many of you that you are subject to removal because Microsoft does not have evidence of a qualifying audit on file. This is likely an error on our side, but we need your help,” the statement read.

“Our CRM system suffered a data loss, and it looks like it rolled back to an old backup. As a result, we lost audit data for about 147 roots. If you received a message, please don't panic. Instead, please just send Microsoft your most-recent audit data, and we will update our records. Sorry for the confusion.”

To offer some background on this situation, Microsoft’s Trusted Root program requires that certificate authorities must give a copy of their audit data every year, as part of the compliance requirements of the program.

Microsoft stores this data internally in a standalone tool and it’s with tool that the error occurred. Though Microsoft’s statement say that it then rolled back to an old backup, it makes you question why they don’t have more recent backups available.

This is slightly embarrassing for a company that has Microsoft Azure, a backup and cloud data storage solution. Having to reach out to their affiliates to ask them to provide their data again, in lieu of a recent backup on Microsoft’s side, doesn’t create the greatest confidence.

While it’s unknown how recently those 147 roots were submitted, it can only be assumed that Microsoft aren’t backing up their data often enough. Even if their primary copy of the data fails, they should have a replica available to fall back on.

Microsoft would be wise to review their disaster recovery plan across all their business streams if this example is any indication of the type of protection that surrounds some of their data.

As a consumer, it’s always wise to have redundancy in mind. A good way to keep to this is to follow the 3-2-1 backup rule. This means that you have at least three copies of your data, in two different formats, with one of those copies off-site. Also, backing up incrementally, that is as the data has any changes made to it, will mean that you’ll always have the latest versions of your data available in a crisis – and you won’t have to reach out to anyone to get it back!