Backup lessons from a cloud-storage disaster

The largest cloud provider based in Europe, OVHcloud, suffered a catastrophic fire last month that destroyed one of its data centers and smoke-damaged a neighboring one. OVHcloud customers with data in the burned-out data cener who had their own disaster recovery measures in place or who purchased the off-site backup and disaster-recovery services offered by OVHcloud have been able to resume operations. Those who did not lost data that will never come back.

Some losses were complete, such as those described on Twitter by rounq.com who is still waiting for backups and redundancy that he thought were already in place, according to his tweets. Companies that had some type of off-site backup seemed to be up and running again, such as Centre Pompidou

There also appear to be companies somewhere in the middle that have resumed operations but have also acknowledged losing data. One of them is Facepunch, providers of the dystopian game Rust that involves players creating their own virtual environments that are stored as files on a server. If the environments that they built were stored in the destroyed data center, it appears to be gone.

”There is no cloud”

Nothing sums up the reality of cloud computing more than this: There is no cloud; there is only someone else’s computer. And if that someone else is your cloud provider, and it’s computers are not properly protected, it can harm you.

Remember that the cloud is not magic, and that nothing stops the fundamental laws of physics and chaos. It appears that OVHcloud had everything it needed to prevent and stop a data-center fire, but as it turned out, it wasn’t up to the task.

Remember that the cloud service you use, whether it is IaaS, PaaS, or SaaS, is just another data center like one you might build yourself—only it’s in another place run by other people. They may be the best at what they do, but neither they nor the systems they employ are infallible. That is why you should always have backup and disaster-recovery plans in place. Hope you never have to use them, but have them ready if you do.

Another part of the OVHcloud story that drives this home is how much time it’s taking the provider to bring additional capacity online. They are the largest cloud provider in Europe, and they are still struggling to replace the lost computing and storage capacity they lost over a month ago. It looks like they’re doing their best, but they can build and provision servers only so fast. They’re just another company trying to build a data center. There is no cloud; there is only someone else’s computer.

Follow the 3-2-1 rule

The incredibly basic 3-2-1 rule states that you should have at least three versions of your data on two different media, one of which is off-site. And the rule applies to data stored in a public cloud.

Many tweets from OVH customers said they stored their backups on another server in the same data center that burned, which means their primary and backup data were destroyed by the fire. Others felt it was OVH’s responsibility to protect their data from a data-center fire, so they made no provisions at all for backups. (At least one Twitter handle seemed unsympathetic toward those who would blame OVH. @kalle_sintonen has repeatedly told people that you get what you deserve when you pay for an inexpensive VM with no redundancy without also paying for an optional offsite backup service offered by the provider.)

The optional service that OVH offered made sure that backups were copied to another data center. Customers who opted for that service have been able to order new servers and restore their operations from this other backup. Some customers did not use it, and, unless they provided other backup, their data is gone. The consequences of ignoring the 3-2-1 rule are unforgiving when something like this happens.

Know whether service agreements ensure backup

You should know how resources are backed up in your private data center, but do you know how your cloud resources are protected? Are backups stored in redundant storage in an additional location than the resource that they are backing up? You need to know, and you that backup needs to meet the requirements of the 3-2-1 rule.

Read your service agreement to see what protections it includes. Does it even mention backup? Does it talk about disaster recovery? Most cloud contracts do not, and if they do, they specify that backups and DR are your responsibility, not theirs. If they offer an optional backup service, it is your responsibility to opt in. Make sure that everything you think they are providing is guaranteed in writing. Remember: if it’s not in writing, it doesn’t exist.

If your contract does include backup, does it explicitly say how the vendor stores backup data and whether it is stored in a completely different system in a completely different region and account?  How will their backup system protect you in a disaster like the OVH fire?  Could the backups be in a neighboring data center that could be damaged by the same fire? If your provider doesn’t have good answers, demand better ones or change providers.

It appears that OVH is bending over backwards to help customers as much as it can. But in the end, if your primary and backup data were both stored in the destroyed data center, your data is gone forever. Even if a customer stored an additional copy in the neighboring data center, it also might be gone due to smoke damage. OVH is literally scrubbing the insides of smoke-damaged computers to bring them back online, but some of these systems might not recover, and data stored on them could be gone forever. 

Some cloud-storage vendors are transparent about data protection. If you search for backup and recovery on their websites, you might find a webpage that states backup of your data is your responsibility. Or it might detail what kind of backups they do provide and where that data is stored.

Other vendors are opaque and don’t make that information readily available. Try getting a straightforward answer from them on whether backup and recovery is your responsibility. If a vendor representative tells you that your data is safe and that you don’t need to worry about backup and recovery, get the details in writing. If you do have a backup and DR service, make sure it protects against all disasters including electronic attacks. Something that physically destroys the data center is not the only thing you need to worry about. You also need protection from threats like ransomware and hackers deleting your account.

Short of well-documented backup and disaster-recovery protection included in your SLA, make sure you perform your own backups. It will cost some money, but it’s a good bet the companies that lost everything last month wish they had done so.