Did you know there have been 7870 public data breaches since 2005? Your company’s data is under attack. Like terrorism, the attackers only have to be successful once. You have to be successful 100% of the time.
Which is why its important to patch your systems regularly and keep abreast of any security vulnerabilities your company’s backup product may have. But have you ever thought about how much of a security risk the backup server is? It’s a risk for three reasons: the value of what it has, the typical experience level of its admins, and lack of attention.
The backup system has all the marbles
Did you ever think about the fact that the backup system is the most sensitive server in your environment? It’s sensitive because it has everything and it can do everything.
First, the backup system has a copy of everything! All the data in your environment resides on disks or tapes it controls. While some data may be stored offsite and is effectively out of reach, most current data is immediately available via a few simple commands. Sometimes the backup data is available via other mechanisms, such as a web or NFS server, which is why a vulnerability in those products could give a malicious user access to anything he/she wants.
The backup system can read and write every piece of data in your datacenter. In order to backup data, it must be able to read it. To be able to read it, the backup system is given superuser privileges. Unix/Linux backup software runs as root, and Windows systems tend to run as Administrator. That means it can read or write any file in the environment.
Most backup software also has the ability to run scripts before and after the backup, and those scripts run as the privileged user. Combine that with the ability to backup and restore files, and you have a scary situation. A malicious user that gains backup admin privileges can write a malicious script, back it up, restore it to the appropriate location, then execute the script using a privileged user. Just let that sink in for a minute.
The backup admins are often very junior
My first job in tech was the “backup guy” for a huge credit card company. I barely knew how to spell Unix, and a few days into my job I was given the keys to the kingdom: the root password to the backup system and every server in the datacenter. (We didn’t have the concept of role-based admin in those days, so anything you did with backups, you did as root.)
My story is not unique. Backups are often given to the FNG. He or she takes the gig because it gets them the job, but it’s the job that nobody wants. As soon as you get some experience under your belt, they do their best to pass off this very difficult job to anyone else. This has been true of backups for years, and this revolving door usually results in very junior people running the backup system.
I know I wanted to get out of backups back then, but I went from being the backup guy to being in charge of the backup team. Three years later, I was still the main point of contact for the backup system. Working for me were several people who were just as junior as I was when I started, all of whom had root privileges to the entire bank. Without going into details, I’ll just say that not everyone that worked for me should have been given the keys to the kingdom like that.
The most sensitive system in your environment is being handed over to the most junior person you have. Again… let that sink in a little bit.
The backup server doesn’t receive enough attention
The security team always made sure the database servers & file servers were patched. But I don’t recall ever getting a call from them about the backup server. That meant it was up to the most junior person in the environment to make sure the most sensitive server in the environment was being regularly patched and secured against attacks. That makes perfect sense. Not.
Another way this manifests itself is in the backup software. Many companies making backup products rely on external products (e.g. Apache) to augment their functionality (e.g. web access to your backup server). The thinking is to use publicly available tools instead of building their own. They’re a backup company, after all, not a web server company.
But unfortunately, embedded software like this often gets patched later than it should. When an Apache vulnerability is discovered, people who know they are running Apache tend to patch it. But what if it’s inside your backup software? You rely on the backup vendor to know that and to patch it appropriately. But the inattention I’m referring to also sometimes applies to embedded components inside a backup system. It make take weeks or months before the vulnerability is patched in the backup software. This ArsTechnica article discusses a recently patched vulnerability in a backup software package where there was a three month delay between the initial discovery of the vulnerability and the creation of a patch for all related systems.
Choice 1: Secure your onsite backup system
You can do a number of things to secure your onsite system, starting with recognizing how much of a vulnerability it is. You can harden the system itself, patch the backup system, and do your best to limit the powers of your backup admin.
Harden the backup system
Firewall it off, using a software firewall running in the system or an actual firewall in front of the system — preferably the latter. Make it so that you can only administer the system via a particular VPN, and that admins must authenticate to the VPN prior to administering the backup system. This also addresses another vulnerability, which is that some backup systems send their commands in plain text.
Make sure that the backup server is running the most secure version of the operating system you have.
Run the backup software via a separate privileged account, not the privileged account. Run it with an account called backupadmin with userid 0, or with Administrator privileges. Do not run it as root or Admininistrator. Then use your ITD software to watch that account like a hawk.
If your backup admin needs root privileges on Unix systems, force them to use sudo.
Require Windows backup admins to use their non-privileged account, and “Run as administrator” when they need to do something special.
Make sure the backup system is continually updated to the latest patch level. It should be the first system you patch, not the last.
If your backup software supports two-factor authentication, use it.
If you are writing backup data to a deduplication appliance across Ethernet, you need to harden and separate that interface as well. For example, do not allow direct access to any of its data via NFS/SMB. A physically separate Ethernet connection between the backup server and any backup storage would be preferred.
Limit backup admin powers
If your backup system supports the concept or role-based admin, do whatever you can to limit the power of the backup admin. Maybe give them the power to do backups but not restores. Or they can run backups, but not configure backups. Restores and configuration changes could/should be done by a separate account that requires a separate login with strong two-factor authentication.
Choice 2: Get rid of your backup server
What if you got rid of your backup server altogether? There’s nothing more secure than something that doesn’t exist! You could do this by using a backup system with a service-based public cloud architecture. Backup services that backup directly to the cloud offer a number of security advantages over those that use backup servers.
Front end designed for direct Internet access
Traditional backup systems are designed to be run inside an already-secure datacenter, where there is an expectation that direct attacks will be lower. Cloud backup systems are designed with harder front ends because they acknowledge they will be directly connected to the Internet. A lot of the basic security changes suggested above would be considered table stakes to any Internet-facing service.
Continuous security monitoring
Backup services run in a cloud like AWS are continually monitored for attempted intrusion. (Again, this is table stakes for such a service.) You get best of breed security simply by using the service.
Any embedded systems constantly & automatically patched
The operating systems and applications supporting any backup service are automatically and immediately patched to the latest available patches. The infrastructure is so huge that this has to be automated; you don’t have to do anything to make it happen.
Backup data not exposed to anyone
A good cloud backup system also segregates your actual backup data from the rest of the network, just like I was suggesting for your onsite backup server. But in this case, that’s already one. No one is getting to your backup data except through the authorized backup system.
Summary: Lock it up or give it up
Once you recognize what an incredibly vulnerable thing your backup server is, your choices are simple: lock it up very tight or get rid of it. I think most companies would be served well by the latter. Given the advent of really good dedupe and replication, only the biggest companies are not able to take cloud-based backup systems.
----- Signature and Disclaimer -----
Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technologist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.