OK, encrypt your disks after you've done everything else

The other day I wrote a blog entry that said encrypt your tapes but not your disks.  My fundamental premise was that encrypting data at rest in your disk drives only protects from the thing that will never happen: someone walking out with an entire disk array under their arm.  Single disk drives yanked out of the array (more likely) were not going to be any use to anyone even if you didn’t encrypt them.

Turns out I was wrrrmph.

Turns out that the most sensitive data is probably very recoverable from a RAID-ed disk drive.  A whole lot of 1K database rows can be stored in a 64K block of data stored on an individual disk drive in a parity-protected disk array.  (See the comments from my previous post for details.)  And it turns out that you can’t degauss hard drives and return them, so there’s also the exposure of what happens when you return a disk drive to the manufacturer.

I was wrong about the risk, but I still think there are bigger fish to fry in the datacenter.  Sticking with just my world, we’ve got companies that:

  1. Don’t copy their backups (they keep only one copy of every disk or tape they make)
  2. Don’t send their backups offsite
  3. Wait a week or two before sending their backups offsite
  4. Don’t back up their laptops
  5. Back up their remote offices using tapes that aren’t copied and/or aren’t ever sent anywhere

If you’ve got data that isn’t being backed up and isn’t being stored in a different location than it was backed up, you will lose data.  This isn’t a “maybe some guy might steal a disk drive and if he does he might be able to read some data on it.”  Every company in the world has lost a disk drive somewhere in their environment.  I’m a very small company and I lost four this year alone.

The number one reason people telling me they’re on the list above is money.  So my point is that if you’re spending money on encrypting your disks, but you’re not backing your stuff up in the first place — you’ve got your priorities all wrong.

I have the same opinion when I see people spending money make their backup server highly available, but they don’t have money to make a second copy of their backups.  Who cares if your backup server goes down for an hour?  It’s a big deal, but the only app that’s down is backup — not production.  But the chances of you losing data because you had a failed tape and no copies is much higher.  Save the money on the HA software for the backup server and spend it on something that actually makes your backups better.

I also think you can minimize this risk by doing a few things, all of which are cheaper than full disk encryption:

  1. Strong physical security in the data center.  Plenty of good things you can do.
  2. Video surveillance in the data center
  3. Identify really sensitive data and encrypt it in the application
  4. Strong physical security (locks) on the disk arrays themselves.  Prevent someone from grabbing a disk drive.
  5. Monitoring on same.  If a disk drive is taken, you should be immediately notified.

Like I said, there are lots of things you can do (and should do) that don’t cost near as much as full disk encryption and most of which you should be doing anyway.

Continue reading

Announcing Backup Central Live! Q2 cities & dates

After our very successful five-city tour in Q1, we are now announcing cities, dates, and locations for our Q2 events.  Those of you that live in Raleigh, Boston, Philadelphia, Dallas, & Minneapolis are the next folks to be able to attend a Backup Central Live event.   In addition, those of you interested in deduplication, continuous protection of servers, and backup of laptop data have three webinars to choose from next month.

Here are all of our upcoming events and where you can register for them.

Raleigh Apr 26 Register Now
Boston Apr 28 Register Now
Philadelphia May 17 Register Now
Dallas May 19 Register Now
Minneapolis May 24 Register Now
Better Backup: Strategies for Better Protecting Your Data, Your Time, and Your IT Budget Apr 19 12p ET Register Now
Top 10 Backup and Disaster Recovery Secrets You Can’t Afford Not to Know Apr 20 12p ET Register Now
Enterprise Laptop Backup: Protecting Users At The Edge Apr 27 1p ET Register Now

See you there!  If you have any questions about events, feel free to contact us.

Continue reading

Encrypt your tapes but not your disks

Update: My opinions on this have changed due to the comments written below.  Feel free to read this post, but make sure you read the follow-up post as well where I change my tune a bit.

Steve Duplessie wrote a blog post inspired by the RSA hack . His post isn’t about that hack at all.  But for the record, I agree with this guy who says “RSA Silent About Compromise For 7 Days – Assume SecurID Is Broken.

Steve’s blog post said that the lesson we should learn from the RSA hack is that anyone can get hacked.  I would agree with him.  He said that your security system should be based on that assumption.  I would agree with that.  He said:

Your security strategy should be based on the assumption that you WILL lose your backup tapes.  You will be hacked.  You will have your customer’s name, SS numbers, and bank account information published on a website.

He then goes on to say, “…if your binary data is going to go missing, it had best be encrypted. Encrypt it at rest, in flight, on the truck, on the disk, in the lab, in the warehouse, everywhere.  Encrypt it so when you lose it, it gets stolen, or Chuck leaves the tape on his dashboard while at the bar, it can’t do you any harm.”

Let me start where I do agree.  Encrypt your backups tapes. Encrypt your backup tapes. Let me say it again: Encrypt your backup tapes! With in-drive encryption built into any tape drive worth its salt, it’s a no-brainer.  (You do need to make sure you have a good key management system.)

Where I don’t agree with Steve is when he recommends that you should encrypt your disk drives.  (BTW, I respect Steve a lot and I’m sure he’ll appreciate this blog post as much as the next guy.)  I will go with his assumption (that you’ve been hacked) and explain why encrypting data on disks at rest wouldn’t help.

If the host storing the data has been hacked.  The hacker is accessing your system like any other user.  Data encrypted at the drive level is automatically unencrypted for the host that is reading the data.  It’s as if you aren’t encrypting — otherwise the apps reading the data wouldn’t be able to read it.  Data encrypted at the application level doesn’t protect you if the server has been hacked, either, because the hacker can just become the appropriate user that runs the app, and voila!  He sees the data unencrypted.  What if the server isn’t hacked, but the SAN is? If the SAN is hacked between the host and the encryption device (assuming in SAN encryption, not host-based encryption), such as via WWN spoofing or the like, then they will be able to read the data as well, so you’re not protecting against that. 

Let’s say you didn’t encrypt, and someone grabs a disk out of RAID array and runs off with it.  They would only have part of the picture and they wouldn’t  be able to read any data off that.  (Update: Greg pointed out that this doesn’t apply if we’re talking single disk solutions, or one half of a mirrored double disk solution.  He is right.  This comment only applies to RAID 0+1,10, 5, 6, etc. where multiple disks are required to create a volume.) 

The ONLY scenario that encrypting data at rest on disk protects you from is someone literally walking out of your datacenter with the entire disk array on their back and no one seeing it — AND that same person being dumb enough NOT to bring the (much smaller) system that can unencrypt the data with him.  Yeah, that’s gonna happen.

Should every laptop hard drive be encrypted?  Yup.  Should every backup tape be encrypted?  Yup.  Should your smartphone have remote wipe and a really good way to prevent people from accessing it as well?  You bet. They are way too mobile and have way too much sensitive data on them.

But I’m still not sold on storing data at rest on disk in encrypted form.  But I honestly would love for someone to explain to me why I’m wrong.

Continue reading

Is SNW good for end users?

Steve Duplessie tweeted today that he thought that SNW was a waste for end-users.  Historically I would agree with him, but I thought I’d give the question a fresh look.  Sometimes end users ask me, “If you were me and you could only go to one seminar a year, which one would you go to?”

First, let’s consider the vendor-specific shows: EMC World, CA World, VM World, Mac World, Symantec Vision, etc.  I assume you know what you’re going to get when you go to those shows.  The answer to every problem you have is a product from that company, or at the very most, a product from a partner of that company that probably isn’t a competitor of that company.  I remember once when I saw an exhibitor at a vendor-specific show ignore this and it did not go over well.  So if you want to learn all about vendor A and why they’re so great — and never get any

Continue reading

My thoughts on the gmail software update bug of 2011

It’s the biggest thing that’s happened in backup and recovery in a long time.  I can’t imagine being “the backup guy” on the other end of this story.  Can you imagine the stress of being the last line of defense for gmail?  Wow.

We all know the story, right?  A software update bug caused somewhere between 150,000 and 500,000 gmail users (which they said was .02% of their user base) were greeted with an empty inbox one morning.  Google took a few days to get everything back, and in the end, they had to resort to tapes to do it.

I’m no Google lover.  I’m a fan of google.com.  I used to use gmail and Google Apps to host my email, but I’ve since moved off and went with hosted Exchange.  So I don’t want anyone accusing me of being a Google fanboi, OK?  So when I start talking about my thoughts, please don’t suggest that the praise I send Google’s way is due to any sort of loyalty, alright?

Here’s what I learned via this outage:

Google is backing up gmail

I spent some time at a very large ISP a few years ago and was shocked to learn that they were not backing up user’s email account.  These were paid ISP subscribers’ accounts and they were not backing them up.  “It’s just email,” they told me.  “Do you know how much it would cost to back that up?”

So I find it admirable that one of the things that came out of this story is that Google is backing up gmail — even free gmail.  There were no comments that said something like “Pro accounts were restored, but free gmail users were not.”  They backed it all up and they restored it all.

Google is backing up gmail to tape

In this world of cloud backup and disk backup, it was interesting to see that Google’s last line of defense was still tape.  They replicate things to multiple data centers, but at some point they back it up.  And when they do, they do it to tape.   The biggest reason that I can think of is that with the sheer volume of data they are dealing with, tape is absolutely the cheapest way to go.

Let me state this again: a company who is notorious for rolling their own and could totally code their own backup application and take advantage of dedupe, etc, is backing the world’s most popular cloud service to tape.

It think both of these things I learned are huge.  How about you?

Continue reading

Backup Central Live! survey yielded interesting results

Tape is still hot, cloud is getting hotter, but companies are still stagnated when it comes to capital purchases, according to a recent survey of 156 IT professionals during the Backup Central Live! seminar series.

I purchased an audience response system from Option Technologies for our Backup Central Live! seminar series.  (We will be in 20 North American cities and several cities abroad this year.) Each attendee is given a keypad that allows them to answer questions displayed on the screen during our seminars.  The system displays what everyone’s answers are and stores them in a database for later retrieval.  This increases audience participation, provides some interesting live verification to the audience of what the speaker is saying, and can provide some interesting marketing information when you compile everything together.

While not every attendee pushed the buttons on their keypads, we got a much larger percentage of response than you would get from any email-type survey. Such surveys yield single-digit percentages at best.  In contrast, the vast majority of our attendees responded to the live survey, giving us a total of 156 respondents.  (3 of them walked off with the keypad!  We know who you are…)

The biggest surprises were the number of people that are still doing things exactly the opposite of what we’ve been telling people to do for years.  (This brings up another advantages of these systems.  The perceived anonymity allows people to be much more candid in their responses.)

Given that I’ve been preaching for years that the one thing that people needed to stop doing was backing up directly to tape across the network, it was surprising for me to learn that 49.1% of them are still doing exactly that — backing up directly to tape with no disk buffer.  By the way, I don’t have a problem using tape at all.  I just believe you should stage backups to disk prior to sending them to tape.  The mismatch between the speed of the backups and the speed of tape devices is the number one cause of backup system failure today.  Every backup system specialist I know has been saying this for nearly a decade, so it’s surprising that half of the respondents are still backing up directly to tape.  A related statistic showed that even though many people have moved to disk staging or deduplicated targets, 88% still use tape as their final destination for backups.  Only 13% of the respondents have gone tapeless.  (61% of the respondents said they are not using dedupe at all.)

Another big surprise came when we talked about long term retention (greater than 1 year) and how it is being accomplished.  The only reason to store backups for multiple years is e-discovery and space reclamation.  Backup is lousy at both of them, so backup software shouldn’t be used to meet this requirement.  This is why it was a surprise to learn that 60% of the respondents were still using backup software that had no e-discovery capabilities to meet their e-discovery requirements!  Couple that with the fact that 24% of the respondents also said their retention on their backups was infinite, or that 79% of the respondents said they retain backups for longer than a year, and you have a “disaster” waiting to happen.  All it will take is one e-discovery request to cost each of these companies far more money than a full archiving system would have cost.  Then they’ll wish they had done something different.

42% of the audience felt they do not have a good understanding of the amount of data they are managing. This is a very common problem as well, leading to difficulty in planning for the capacity of the primary and backup systems.  Storage management software can help, but very few people use it.

My final surprise came when we talked about the cloud.  For those who think that the cloud is all hype and no purchases, consider this.  For something that’s only a few years old, I think it’s impressive that over 10% of the respondents said they were storing reference data in the public cloud, 10% said they were storing primary data in the public cloud, and 10.6% said they were storing backups in the cloud.  78.9% of the respondents are not using the public cloud at all, so there’s certainly room for growth.  One reason for the popularity of the cloud could be that 41% of the respondents said that their company is not making any capital purchases right now. This was cited as the number one barrier to deploying new technology.  Since cloud is all about a monthly bill and not a large capital purchase, it is very compatible with the way many IT departments are now being forced to conduct themselves.

Like I said, tape is still hot, cloud is getting hotter, and many people are not making capital purchases.  Very interesting stuff, if you ask me.

Continue reading

Back up Gmail/Facebook/Flickr

Gmail’s outage today made me think about something I’ve had in the back of my brain for a while.  While I’m a big fan of the cloud (as long as it makes sense for you), one of the things we say is that you can outsource the management of your data, but not the responsibility for it.  I’ve also written that for personal data, I do not believe in using a free (or incredibly cheap) cloud service provider to store your only copy of data. For example, I don’t believe that the only copy of your pictures should be on Flickr.

But now there’s a way to have two copies of data in the cloud: cloud backups for cloud storage.  Backupify.com and backupmyblog.com/backupmypics.com/backupmytweets.com/backupmymail.com will back up various parts of your online life for a reasonable fee.

Backupify.com has a free plan that will back up 2 GB of online data from Facebook, Twitter, Gmail and more.  The Pro account includes support for Google Apps and handles up to 20 GB of online data from 25 online accounts for $4.99/month.  Finally, they have the Pro 500 account that handles an unlimited number of accounts with an unlimited amount of storage for $19.95/mth.

Backupmyblog.com/backupmypics.com/backupmytweets.com/backupmymymail.com has a free 1 GB account and a basic account that cost $19.95/year for up to 1 GB of online data.  Additional storage is $2.95/GB/yr.  It is unclear if someone who wanted to back up all of those things would need four accounts or one account.

I have not used either of these (yet), but I’m surely thinking about it.

Continue reading