Backup lessons learned from 10 major cloud outages

I’ve always called for independent backups of everything. Whether it’s in your data center, an IaaS vendor like AWS, or a SaaS vendor like Microsoft 365, it needs independent backup. However, some people believe we should trust the cloud vendors to do the right thing. What’s the worst that could happen?

Fifteen years of cloud data-loss incidents gives a pretty good idea of how bad a cloud outage can be. On my podcast, The Backup Wrap-up, we recently reviewed 10 cloud catastrophes that occurred over the last 15 years (listed alphabetically):

• Carbonite (2009): Due to lack of redundancy and the use of consumer-grade storage arrays, Carbonite lost the backup data of thousands of customers in a major data loss incident. They blamed their storage vendor rather than taking responsibility.
• Code Spaces (2014): A hacker gained access and deleted all customer data and backups from Code Spaces’ AWS environment. Code Spaces went out of business as a result.
• Dedoose (2014): A service failure took out both Dedoose’s primary research database and its backups, which were only done monthly. This resulted in over a month’s worth of data loss for many researchers.
• KPMG (2020): An admin accidentally changed a Microsoft Teams retention policy, permanently deleting chat data and files for over 145,000 employees. Native retention policies in Microsoft 365 did not allow recovery; in fact, they were the cause of the data loss.
• Musey/Moss (2019): A startup accidentally deleted their entire organization’s Google account, instantly losing over $1M worth of data and IP. Google could not restore the data as no independent backup existed.
• OVH (2021): A fire destroyed servers at OVH’s Strasbourg data centers. Many customers lost data as OVH’s included backup service stored backups in the same data centers.
• Rackspace (2022): Rackspace’s hosted Exchange environment was hit with ransomware. Slow patching allowed the attack, and recovery took months even with included backups. Rackspace ultimately shut down the hosted Exchange business.
• Salesforce (2019): A faulty script gave all Salesforce users full-modify permissions until fixed. Salesforce’s backups did not allow quick restoration of proper permissions, demonstrating the need for independent SaaS backups.
• StorageCraft (2014): During a cloud migration, StorageCraft accidentally decommissioned a server early, losing customer backup metadata and rendering their backups useless. The CEO took full responsibility and worked to help customers re-seed backups.
• UniSuper/Google Cloud (2024): Google accidentally deleted UniSuper’s entire cloud environment across regions due to a configuration error. However, UniSuper’s third-party backups allowed full recovery within a week.

Lessons learned from cloud outages

Let’s take a moment to reflect on the hard lessons we can learn from these gut-wrenching tales of data loss and business disruption. First and foremost: The cloud is not a magical realm of infinite redundancy and automatic backups. It’s just someone else’s computer, and like any computer, things can and will go wrong. We’ve seen it time and time again – from the OVH data center fire to the Rackspace ransomware attack. Your data is only as safe as the precautions you and your cloud provider take to protect it.

So, what’s the most critical lesson here? Back up your cloud data! And I don’t just mean relying on your provider’s built-in backup services. As we saw with Carbonite, StorageCraft and OVH, those backups can evaporate along with your primary data if disaster strikes. You need to follow the 3-2-1 rule religiously: keep at least three copies of your data, on two different media, with one copy off-site. And in the context of the cloud, “different media” means not storing everything in the same type of system; use different failure domains. Also, “off-site” means in a completely separate cloud account or, even better, with a third-party backup provider.

[ Related: Keep up with cloud outages with Network World’s weekly global network outage report ]

But it’s not just about having backups; it’s about having the right kind of backups. Take the StorageCraft incident, for example. They lost customer backup metadata during a botched cloud migration, rendering those backups useless. This hammers home the importance of not only backing up your primary data but also maintaining the integrity and recoverability of your backup data itself.

Another hard truth: SaaS providers aren’t immune to data loss either. The Salesforce permissions fiasco and the KPMG Teams retention policy snafu prove that even the biggest names in SaaS can accidentally nuke your data. And as we saw with Dedoose, sometimes their recovery capabilities are severely limited. That’s why it’s crucial to back up your SaaS data independently, using a third-party solution that gives you control over your backups and recovery.

Now, I know what some of you might be thinking: “But Curtis, my cloud provider offers geo-redundancy and multi-region replication. Isn’t that enough?” Ask UniSuper how that worked out for them. Google accidentally deleted their entire cloud environment across multiple regions. If it wasn’t for UniSuper’s third-party backups, they would have been up the proverbial creek without a paddle.

Lastly, let’s talk about the human element. Many of these disasters, like the Code Spaces hack or the Musey Google account deletion, came down to human error or poor security practices. It’s a stark reminder that no matter how sophisticated your cloud infrastructure is, your data is only as secure as your weakest link. Educate your team, implement strong access controls and security measures, and always, always have a tested incident response plan.

I want to reiterate something I said earlier. In this list of 10 cloud disasters, only one company came out unscathed, and it’s the one that had a tested third-party backup of its cloud data. If that isn’t enough to convince you to do it, I don’t know what will.

The cloud is an incredibly powerful tool, but it’s not a silver bullet for data protection. Trust, but verify. Back up your data like your business depends on it because it does. Learn from the misfortunes of others and don’t let your organization become another cautionary tale. Remember, there are two types of people in this world: those who have lost data and those who will. Make sure you’re prepared for when that day comes.