Backup Central

My thoughts on the gmail software update bug of 2011

Written by W. Curtis Preston Tuesday, 15 March 2011 01:28

It's the biggest thing that's happened in backup and recovery in a long time.  I can't imagine being "the backup guy" on the other end of this story.  Can you imagine the stress of being the last line of defense for gmail?  Wow.

We all know the story, right?  A software update bug caused somewhere between 150,000 and 500,000 gmail users (which they said was .02% of their user base) were greeted with an empty inbox one morning.  Google took a few days to get everything back, and in the end, they had to resort to tapes to do it.

I'm no Google lover.  I'm a fan of google.com.  I used to use gmail and Google Apps to host my email, but I've since moved off and went with hosted Exchange.  So I don't want anyone accusing me of being a Google fanboi, OK?  So when I start talking about my thoughts, please don't suggest that the praise I send Google's way is due to any sort of loyalty, alright?

Here's what I learned via this outage:

Google is backing up gmail

I spent some time at a very large ISP a few years ago and was shocked to learn that they were not backing up user's email account.  These were paid ISP subscribers' accounts and they were not backing them up.  "It's just email," they told me.  "Do you know how much it would cost to back that up?"

So I find it admirable that one of the things that came out of this story is that Google is backing up gmail -- even free gmail.  There were no comments that said something like "Pro accounts were restored, but free gmail users were not."  They backed it all up and they restored it all.

Google is backing up gmail to tape

In this world of cloud backup and disk backup, it was interesting to see that Google's last line of defense was still tape.  They replicate things to multiple data centers, but at some point they back it up.  And when they do, they do it to tape.   The biggest reason that I can think of is that with the sheer volume of data they are dealing with, tape is absolutely the cheapest way to go.

Let me state this again: a company who is notorious for rolling their own and could totally code their own backup application and take advantage of dedupe, etc, is backing the world's most popular cloud service to tape.

It think both of these things I learned are huge.  How about you?

Comments  

 

0 #10 Jean 2011-03-26 13:25

I used to be hard disk technician and I can tell you that I can read lots of data of single disk pulled from and RAID. This is because most RAID technologies are using large block (64-256K). Yes partial files are often seen but I might be able to pull thousand of names with SS in the block. I can also see partial word, excel, PowerPoint documents within respected size.

This is where on-board disk encryption emerged. Like tape drives they use imbedded chip to do it without speed impact.

So when technician received these disks they cannot see the data anymore.

Quote

 

 

0 #9 DougCP 2011-03-17 15:31

Good to know Google won't lose all that information they have on you.

Quote

 

 

0 #8 rworman 2011-03-15 17:56

For the sake of posterity, here is a link to the story (and mea culpa) on the official Google blog:

http://gmailblog.blogspot.com/2011/02/gmail-back-soon-for-everyone.html

Quote

 

 

0 #7 Ewan 2011-03-15 14:35

I'd love to know their rollback capabilities, rentention periods, stuff like that - I assume they've written something which handles Gmail's own storage formats, rather than a plugin to something like TSM, and I assume they do constant streaming of data to tape in the background, rather than run a specific window - I'm sure it's all technically very interesting.

On the use of tape itself vs disk, I think you're right it's really just another statement that tape's not dead yet, especially in high-end environments where you never want to read the data back (because that means you've got a very big issue).

Quote

 

 

0 #6 Nick W. 2011-03-15 14:22

Thanks for this, Curtis, I had similar thoughts when this came out even though all I heard around me was how could Google have lost the data in the first place. Considering it was a fairly low likelihood of occurrence, I applaud Google for being able to recover fully - even if it did take a few days.

Also very nice to see a giant use tape when other options are clearly available for them, maybe this will cause a few of the disk-only proponents to think a bit.

Quote

 

 

0 #5 Rowan Gillson 2011-03-15 06:48

Sorry, should have qualified that we are big proponents of backup to disk, I've supported this methodology for at least 6-7 years now since I started working with data protection products. What I was trying to elocute was that I don't think vendors who keep pushing disk only solutions have the customers best interests at heart.

Hope that makes sense.

Quote

 

 

0 #4 W. Curtis Preston 2011-03-15 06:31

I'm saying that there are perfectly valid reasons for using tape instead of disk in a typical backup environment. It is much more appropriate as the initial target for backups than tape; tape is impossible to make happy when backing up directly to it from a source filesystem, over the network, using an incremental backup (what ppl do most of the time). So I do NOT agree that the addition of disk in backup architecture was just a scheme to sell more disk.

Having said that, there is also a very valid place for tape in almost all backup environments.

Quote

 

 

0 #3 Rowan Gillson 2011-03-15 05:54

Quoting W. Curtis Preston:

I agree with everything you said except for the part about disk being just something the vendors are pushing to sell more disk. (Not in most situations anyway...)

Why wouldn't they? It's in their best interest to sell you more disk even if you could use more efficient, reliable or cost effective tech I would have thought. It's a strategy I've seen quite frequently by a number of vendors pushing disk only solutions, without naming anyone specific :)

Isn't that why storage vendors pushed VTL so hard? Anyway, it's a theory of mine that I'm happy to have rebutted

Quote

 

 

0 #2 W. Curtis Preston 2011-03-15 05:06

I agree with everything you said except for the part about disk being just something the vendors are pushing to sell more disk. (Not in most situations anyway...)

Quote

 

 

0 #1 Rowan Gillson 2011-03-15 02:25

Curtis, I couldn't agree more strongly that tape still has it's place in a data protection strategy. It's the last line of defense when your disk based copies have gone the way of the dodo.

As Public Enemy put so succinctly, "Don't Believe The Hype". The push to implement disk only solutions is a strategy the vendors use to push more disk sales on the customer, but when all is said and done, it doesn't matter how many copies of the data you have, it only takes a few lines of bad code to lose decades of valuable information.

On the flipside, tape should be used only after consideration of what it is you actually need to protect and retain given your state or national legal obligations for data retention. It's unrealistic to expect a business to put every byte of data down to physical media designed to last 30 years in controlled conditions.

My 2 cents :)

Quote

 

Refresh comments list
RSS feed for comments to this post

Add comment

Name (required)

E-mail (required, but will not display)

Website

Notify me of follow-up comments

Refresh

Send

Cancel

JComments