W. Curtis Preston (Mr. Backup) and Ben Patridge talk about the things they learned while working in IT and backups at a bank.  This is the second Beer and Backups episode. We’re both 1 pint in w/a strong Belgian beer, so there might be some slurring here and there. 😉

Special guest on the podcast this week, Ben Patridge (Portland beer Ben), and W. Curtis Preston (Mr. Backup) discuss horror stories and other things gone wrong with backups, while drinking a couple of Trappist beers from Belgium.

(BTW, neither of us got the pronunciation of Westmalle correct.  Here it is if you’re interested: https://forvo.com/word/westmalle/).

After covering in the last episode why tape isn’t really good for backup anymore, W. Curtis Preston (Mr. Backup) and Prasanna Malaiyandi discuss the continued uses for tape. It’s not dead — even growing.

W. Curtis Preston (AKA Mr. Backup) explains to Prasanna Malaiyandi the early days of tape, and what happened to make backup and tape to become bad for each other.

Today we have a special guest on Restore it All, and it’s none other than Jeff Rochlin, an industry veteran and long-time friend. He’s going to be giving us the inside scoop about how they do storage and backups in Hollywood, where he has worked for over 40 years. He started out as an actor, and has run IT for the likes of Disney, Dreamworks, and is now head of IT at Fandango and Rotten Tomatoes. This episode is a little longer than most, but it’s full of good stuff. I hope you enjoy it.

W. Curtis Preston (Mr. Backup) and Prasanna Malaiyandi are joined by a guest, Matt Leib, and they discuss how Microsoft disabled the registry backup in Windows 10 and told no one.

W. Curtis Preston (Mr. Backup) and Prasanna Malayaindi interview an IT director from a tech company in the bay area. (Company and full name withheld for privacy reasons.) . We discuss what it’s like to be an IT and security manager in today’s world.

Two events from two major software vendors less than a month apart have proven my point that SaaS companies should not be trusted to backup your data. That is because Salesforce and Microsoft both committed acts that as a backup person I find simply hard to fathom. I will explain both in detail but suffice it to say that both have proven that they do not understand backup and recovery, and should therefore not be trusted to perform that service on your behalf.

Your SaaS vendors are not backing up your data

The shocking truth behind most SaaS vendors is that they are not backing up your data in a way that would be helpful to you in a major outage caused by you or malware attacking your company. They do have disaster recovery copies of your data, but that is only to be used in case of disaster – theirs not yours. They also might have data protection features designed to help address some user mistakes, but none of the major services are backing up your data in a way that you can easily use to address a major attack. If you want to debate this point, please read your service level agreement for your favorite SaaS service and try to find the backup service that you think exists. Good luck with that. 

Yes, I am aware of Salesforce’s $10,000 restore service, and the fact that Microsoft support will restore your Onedrive account if it goes completely bonkers. (There is no equivalent for Exchange Online.) I am also aware that neither of these “services” is mentioned in your service agreement, and both of them are offered with a “best effort” disclaimer. They are also both “all or nothing” services, which means you can’t restore part of it and you’ll lose any data since the last backup, and it can take many days to restore your data. Like I said: your SaaS vendors are not backing up your data.

Even if SaaS vendors were backing up, I wouldn’t trust them

Just because you’re paranoid doesn’t mean nobody is out to get you. Just because I see malware on every corner, and rogue admins lurking in every data center doesn’t mean they don’t exist. Ransomware has run amok; just last week two cities in Florida paid over $1 million in ransom to get their data back. Don’t tell me I shouldn’t be worried about it.

Last week I was a guest on a podcast that discussed whether or not SaaS vendors should be responsible for backing up your data. My first point was, of course, can we all agree that they’re currently not doing that? I think we agreed on that point, which means that companies using SaaS services should really back up their data. But I disagreed with my fellow podcasters about whether or not SaaS vendors should be expected to backup data for their customers. My opinion is that even if they offered such a service, I feel it’s asking them to go above and beyond the call and to participate in something they do not have a core competency in. The events of the last 30 days have proven my point. Two major software vendors have shown that they know nothing about backup. 

Salesforce corrupts its customers’ data, tells them to fix it

Salesforce ran a database update script that accidentally corrupted the user access permissions of thousands of Salesforce users – accidentally giving everyone access to everything. After shutting down the service, Salesforce took the unusual step of telling customers to fix it themselves. No, I am not exaggerating or making this up. 

They published a blog post explaining what happened and telling customers how they could fix the corrupted data: use an updated (but not too updated) sandbox copy, or manually reset the user access permissions of all your Salesforce users. Despite the gall of telling customers they should fix the data that Salesforce corrupted, they never even suggested backup as an alternative. It’s like they didn’t know it exists. Many Salesforce customers back up their data to third party services, and others conduct periodic manual backup by downloading data directly from Salesforce. How could Salesforce not even think to mention to someone that if they had a backup they can fix this in minutes? I have various theories but none of them are good. 

Microsoft secretly disables registry backup for over a year 

This story is even more unbelievable. For those of you unfamiliar with the Microsoft Windows platform, it uses something called the registry to keep track of everything. The registry holds a number of vital pieces of information, including all software that is installed and many configuration settings. Historically, Microsoft has automatically kicked off a registry backup before any major event, such as loading of patches. If something goes horribly wrong, you could roll back to the last registry backup and all would be well – unless you’re running a recent version of Windows 10.

Someone at Microsoft felt that the 50 to 100 MB each registry backup took up was adding too much to the footprint of Windows – so they disabled it. Secretly. Over a year ago. In a way that made it look like nothing happened. The backup process would continue to run and it would say that it was successful! But the backup would be zero bytes. That bites.

Due to the stealth nature of how they did it, the only time you would know this had happened was if you needed to restore your registry. You would do what you always did to tell Windows to go back to a different recovery point. That’s when you would find out that there were no previous recovery points, because Microsoft disabled them in April 2018! 

Tell me again how I’m supposed to trust them to do my backups?

Backup is a weird discipline. Backup people think about things differently; we are much more concerned with making sure data is safe than if the latest feature in your app has been rolled out. We find ourselves constantly asking if the new app or the new database for the new server is being backed up – because often it is not. Backup mentality is important but very few people specialize in it. Even I fell into it accidentally and tried to get out of it before I realized what an advantage it was. People get a CS degree thinking you’re going to be a programmer, or a SysAdmin, or a network admin – No one goes to college thinking they’re going to be a backup admin. This is why I think it’s a core competency that should not be trusted to the average software company. If you are not currently backing up your SaaS apps, please look into it.  And please stop telling me I should trust Microsoft or Salesforce to do it for me.