Addressing Spectre/Meltdown in your Backup System

Your backup server might be the biggest vulnerability in your datacenter, as I already discussed in my previous blog post. Which means that you should have patched it first, but I’m betting that you haven’t patched it yet. If you don’t know why I feel this is a problem, go check out the previous post.

How are you responding to the Spectre & Meltdown vulnerabilities with regards to your backup infrastructure?  What kind of week you’ve had depends on what type of backup infrastructure you have.

Bare Metal Backup Server

This includes bare-metal Linux & Windows servers, and backup servers running in VMs in the cloud. You need to find the appropriate patches for your backup server’s OS, test them, and install them.  Here’s a good list of those patches. I’m guessing you probably don’t have the time to test them to see what kind of performance impact they might have on your backup system.

Reports of the performance impact of various patches include everything from “no noticeable impact” to “50% performance loss.”  Unfortunately for you, it seems that the more I/O intensive your workload, the greater the impact on performance. So you might install (or have installed) the patches and then run/ran your next set of backups — only to find out that they don’t complete anywhere nearly as fast as they used to.

If that’s the case for you, then you’re having to figure out how to respond to this performance loss. If your backup server is running in a VM, you might be able to just upgrade to a bigger VM.  You’ll have a little downtime, but that’s a small price to pay.

If you have a bare metal server, which is far more likely, you might find yourself in a situation of needing to do an emergency upgrade to the backup server.  Some systems run in a cluster and can be scaled by just buying another node in the cluster, but others will require a forklift upgrade of the backup server.  Either way, you may be looking at an emergency order of a new server or two. In short, you might be having a very difficult week.  It’s a good week to be a server vendor, though.

Virtualized Backup Server

If your backup server is running inside a VM, you’ve had even more interesting week. In addition to everything mentioned above, you also need to deal with microcode updates from VMware or Microsoft.

VMware got a lot of credit for responding to Spectre/Meltdown very quickly, as they issued patched pretty quickly. Unfortunately, the patches were apparently causing spontaneous reboots, so they pulled them almost as fast. Check out this page for the latest info on this.

Once these patches are available again, you’ll need to test and install them. And, of course, you will also need to patch the guest operating systems just as you would if they were bare metal.

Hyper-V customers need to do the same thing.  Here’s the latest information from them.

The performance impact of these patches is no more known than the performance impact of the previously mentioned OS patches. Which means you might find yourself having to upgrade the underlying hardware, or at the very least increasing the power of any VMs to compensate for the performance loss.  Again, it’s a good week to sell servers, not such a good week for those buying them.

Cloud-native Backup Service

If you are using a cloud-native backup service, you don’t have to do anything.  A cloud native service means you are not responsible for the VMs offering such a service. Those VMs are not your problem.  The most you might want to do is contact your backup service vendor and ask them if they have patched their systems to address any vulnerabilities.

When the backup service installs the appropriate patches in the backend, there might indeed be an impact to the performance of each VM. But if it’s a scalable cloud service, it should be able to easily compensate for any performance loss by adding additional compute resources.  This should not be something you should have to worry about.

Cloud means never having to say you’re sorry

A true cloud service should not require you to have to worry about the infrastructure.  (Which is why I feel the word “cloud” does mean something, @mattwbaker.) There are other backup systems out there that are actually quite good – but they’re not cloud native. If your backup app requires you to create VMs in the cloud to install your backup server software in, they’re not really a cloud app.  They’re cloud washing. (Honestly, taking a product designed for physical nodes in a datacenter and installing it in VMs in the cloud is a perfect example of how not to use the cloud.)

If your backup service is actually a cloud backup service, you should not have to worry about the security of your backup system – it should be automatically taken care of.  If you’re having to take care of it, perhaps you should consider a different system.

----- Signature and Disclaimer -----

Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Evangelist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.