Include All Files; Reject Some

Written by W. Curtis Preston Wednesday, 29 June 2011 16:57

I had a twitter chat with @JLivens the other day where the question was "what do you back up?"  My first response was to, of course, say that I back up everything - thrice - cause I'm me. If you're curious, my critical personal data is synced on multiple computers with history using Dropbox (which I'm reconsidering based on how things have been going over there lately), then it's backed up with the free version of CrashPlan to another computer that isn't at my house, AND I can't resist the urge to throw in a Time Machine backup every once in a while.  You know what?  I haven't done one of those in a week or so.  Just a second.  My little Time Machine icon is spinning now. Ah, there I feel better.

Side note: for all my talk about tape lately, you'll notice that I don't have any tape in my setup for now.  I am about to embark on a project that may make me reconsider that as I might have an archiving need soon.  Can't keep it all on spinning disk!

Alright, back to the topic at hand.  What do I back up?  I actually do back up everything, but that is not the point I wanted to get across in this post. 

It's easy to come up with a list of directories you don't want to back up.  Your /tmp folder, your "Temporary Internet Files," your folder on your work laptop that contains the illegally downloaded movies that you should have never downloaded in the first place.  Yeah, I'm talking to you.  Pay for the media/software you consume.

But what I wanted to talk about was how to make your backup selection if you want to exclude things.  What I've found is that the human tendency is to say "just backup the Documents folder," or something like that.  And that is what I really want to talk you out of.  There is too much risk doing it this way.  You could accidentally put some important data in a directory you're not backing up.  You could create a whole other directory that contains really important data and forget to add it to the list.  The risk outweighs the benefit of excluding the other data.

If your backup software has the ability, please have it autoselect both filesystems/drives and folders/directories.  If it supports it and if you want to do so, you can also create an exclude list of the directories you definitely don't want to back up.

And that's what I came to say: backup up everything, but exclude what you don't want.  Hopefully the title makes sense now.

Schedule Tweets from the command-line

Written by W. Curtis Preston Tuesday, 21 June 2011 07:29

We at Truth in IT have several events that we need to invite people to, and twitter is one of the ways we do that.  Scheduling such tweets in advance is a great way to make sure you send the right tweet at the right time, and Twuffer.com (short for Twitter Buffer) is an easy way to make such automated tweets happen.  The only problem is that each scheduled tweet in twuffer.com takes several mouse clicks, each of which is followed by a screen refresh.

I wondered if there was an easier way.  I'm proficient in old-style Bourne shell programming in Unix/Linux (never did get very good at Perl, but I rock at Bourne Shell) and I know how to use cron, so if I could just find a way to tweet from the Linux command-line I figured I could make my own twuffer.

An Internet search for "tweet from the command line" turned up this and this article.  I got all excited, then disappointed once I realized those were using basic authentication, which was disabled in June of last year.  It was replaced by oauth authentication, allowing you to authorize an app to use your twitter account without giving them your twitter password.

A google search for "oauth twitter commandline post" turned up this post from Joe Chung's "Nothing of Value" blog called "Twitter OAuth Example."  He explains a series of separate PHP scripts that, if run and edited in the proper order, will result in you having a script called twitter.php that is actually your own properly registered and authorized twitter app that can send tweets from the command line.

While I was able to figure out Joe Chung's instructions (and I'm incredibly thankful for them and the code that comes with them), I wanted to adapt his code and instructions a little bit for those who may not be as adept at coding.  And I've also added my own code around the final tweet.php script to support scheduled tweets.

Before You Start

If you want to understand more about Oauth and how it works, you should read the original blog post.  Each major step below is also a link to the original instructions from twitter.

What You'll Need

You will need a Unix/Linux command line (or something like it), php and cron to make all of this code work.  If you don't have cron or something like it, you won't be able to send scheduled tweets, but you will still be able to send tweets from the command line.  You'll also need to have a basic understanding of the command line.  Unlike the original code from Joe, though, you won't have to edit any of the PHP scripts.

Step 0: Download my modified code

You can download all of my source files here: http://www.backupcentral.com/twitterapp.zip
Unzip them into a directory then cd into that directory.  My first six steps of my post follow the ones from the original post.   I again urge you to read the original post, as he really deserves all the credit for figuring this out.  All I did was hack his scripts to behave differently.  If you want even more information, each step is a link to the original oauth spec from twitter.com.

Step 1: Register an application with twitter

Only registered apps can send tweets via Twitter's API.  So in order to send a tweet on the command line, you need to be your own app.  (Don't worry; the code is already written.  You just need to register the code you just downloaded as your own app.)  The first step in this process is to go to twitter.com and register your app.

Here are some pointers to help you fill out the form:

1. Whatever you put as the name of the Twitter App is what will show up when you send tweets in the "via" column.  For example, we named ours TruthinITApp, so our scheduled tweets say "via TruthinITApp" at the end.  You can name the app whatever you want, except that the name cannot have the word "twitter" in it
2. It doesn't matter what you put in the rest of the fields, although you should probably put a valid website, and a description of what you're up to.
3. I put Browser as my application type, but I'm not sure if that matters
4. Specify Read & Write or Read, Write & DM access
5. Use twitter for login

Once you have clicked Save, you will be presented with a results page.  You need to get two values from that page: Consumer Key & Consumer Secret.  (Record these values somewhere for later.)

Step 2: Get a request token

Now you're going to do the equivalent of a user using the app for the first time.  You will login to twitter, then try to use the app.  Twitter will ask if you authorize the app.  After you do that, it gives you another value you need.

1. Login to twitter as the user you wish to send tweets as
2. Run the following command, substituting the two values of consumer_key and consumer_secret you got in Step 1

$ php getreqtok.php consumer_key consumer_secret

This will display a URL followed by a command.  You will use those two strings in the next two steps.

Step 3: Authenticate the user and authorize the app to tweet for the user

Cut and paste the URL from the previous step into your browser.  (This is the equivalent of using the app for the first time as the user you want to tweet as.)  Once you click Authorize App, it will display a seven-digit number that will then append to the command displayed in the results of the previous command.  (Record the value for later.)

Step 4: Get the access token and secret

Now that the app has been authorized to tweet for the user, the app needs to establish a special key and secret (think username and password, but without actually giving them your password) that it will use each time it tweets on your behalf.  The command will look something like the following command, where consumer_key and consumer_secret are the values that you got when you registered your app, oauth_token and oauth_token_secret are the values the app was given when the app was authorized by the user, and authkey is the seven-digit value from the web page.

$php getacctok.php consumer_key consumer_secret oauth_token oauth_token_secret authkey

This command will display the next command that must be run, which is the actual twitter.php command, along with all the arguments you need to pass to it.  It will look something like the following, where access_token and access_token_secret are the values that the previous command got that are the unique username/password combo for this app and for this user. (Notice the access token actually starts with your twitter user ID -- the number, not the name.)

$ php tweet.php "Hello World..." access_token access_token_secret consumer_key consumer_secret

Step 5: Post a tweet on the command line

Start your twitter client or monitor twitter.com for the user you're going to send the tweet as.

Run the command above, and you should see a bunch of text fly by.  As long as you don't see errors like "Invalid Token" or anything like that, your tweet should have gone through.  

You just sent your first command-line tweet!

Scheduling tweets using cron and tweet.sh

In addition to the code above that was written by Joe Chung, I wrote twitter.sh, that uses twitter.conf and twitter.txt to automate the sending of tweets using cron.  The rest of this blog post is about how to use those tools, which are also in the code you downloaded in Step 0.

Step 6: Edit tweet.conf with the appropriate keys and secrets

Put the values of consumer_key and consumer_key secret as the second and third field in the consumer_key line:

consumer_key:<consumer_key>:<consumer_key_secret>

Create a line for each user that you have authorized using the steps above and insert the appropriate values for:

username:<access_key>:<access_key_secret>

Step 7: Put a cron job that will run tweet.sh every minute for you:

* * * * * /workingdirectory/tweet.sh workingdirectory >/tmp/tweet.out 2>&1

Where workingdirectory is the directory where you installed the code.

Step 8: Edit tweet.txt and put a tweet sometime in the near future. 

The format for tweets is as follows (where "|" is the field separator):

MON DD HH:MM|username|Tweet goes here

Here's an example.  First, get the current date

$ date
Tue Jun 21 03:20:22 EDT 2011

(Yes, I'm up a little late working on this post...)

Second, add a tweet to the file for a few minutes from now

$ echo "Jun 21 03:22|testuser|Test tweet1" >>tweet.txt

Please note that I used "|" as the field separator.  This means you cannot use the "|" character in any of your tweets.  One other note: Twitter will not let you send the same tweet twice, so you will need to change your tweet phrase if you want to do more testing.

When Jun 21, 03:22 rolls around, it will send your tweet.  If tweet.php returns successfully (indicating a successful tweet), it removes it from tweet.txt and appends it to completedtweets.txt.  If there was a problem sending your tweet (such as it being a duplicate), then it leaves it in the tweet.txt file.

That's it.  All you need to do to send tweets in the future is to add them to tweet.txt and they will magically happen.  You can put blank lines, comments, or whatever other formatting you want in tweet.txt, as long as the actual tweet lines follow the format in step 8.

Please let me know if this post was helpful.  Also please post any suggestions on how to make the code better.  If I can make it work, I'll update the code and the post.

Tape more reliable than disk for long term storage

Written by W. Curtis Preston Thursday, 02 June 2011 00:46

Tape is inherently a more stable magnetic medium than disk when used to store data for long periods of time.  This is simply "recording physics 101," according to Joe Jurneke of Applied Engineering Science, Inc. 

I had heard rumblings of this before, but it was Joe that finally explained it in almost plain English in a post to this thread from hell on LinkedIn.  Here's the core of his argument:

By the way, the time dependent change in magnetization of any magnetic recording is exponentially related to a term known as KuV/kt. This relates the "blocking energy" (KuV) which attempts to keep magnetization stable, driven by particle volume (V) and particle anisotropy (Ku) to the destabilizing force (kt) the temperature in degrees kelvin (t) and Boltzmans constant (k).  Modern disk systems have KuV/kt ratios of approximately 45-60. Modern production tape systems have ratios between 80 and 150. As stated earlier, it is exponentially related. The higher the ratio, the longer the magnetization is stable, and the more difficult it is to switch state.....Recording Physics 101....

I had to call him to get more information.  He explained how this came about.  Disk drives have been pushed for greater and greater densities, which caused their vendors to create a much tighter "areal density."  Tape, on the other hand, mainly got longer and fatter to accomodate more data in the same physical space.  (Yes, it increased areal density, too, but nowhere near as much as the disk drive folks did.)  The result is that the tape folks have more room to play, allowing them to use magnetic particles with a bigger particle volume (the V in the equation).  The bigger the particle volume, the more stable the magnetism is, according to the KuV/kt equation.  In addition, tapes are generally stored outside of the drive, which means their temperature is lower than disk drives.  That means they have a lower k volume (degrees kelvin), which is one of the "bad" numbers in the KuV/kt equation.  Having a higher V value and a lower t value is what translates into tape systems having ratios of 80-150, vs disk systems that have ratios of approximately 45-60. While I don't have an exact cite to point to in order to show these exact values, what he's describing makes perfect sense to me.
 

Add to this the fact that tape drives also have a lower bit error rate than disk.  SATA disk is 1:10^14, FC disk is 1:10^15, LTO is 1:10^16, and IBM 3xx0 and Oracle T10000s are 1:10^17.

Add to this the fact that tape drives always do a read after write, where disk drives do not always do this.

Sooo...

Tape drives:

1. Write data more reliably than disk
2. Read it after they've written it to make sure they did (where disks often don't do that)
3. Have significantly less "bit rot" or "bit flip" than disk drives over time.

Like I said in a previous post, I think we've put these guys out to pasture a little too soon.

My Detente With EMC's DD Archiver

Written by W. Curtis Preston Friday, 20 May 2011 19:12

When I first heard about the EMC disk archiver, I blew my stack.  I don't remember exactly how it was presented to me, but what I heard was that EMC was coming out with a disk product that was designed to hold backups for seven years or more.  Since storing backups for seven years or more is fundamentally wrong (and no one -- and I mean no one -- argues with that), the idea that EMC was coming out with a product that was designed specifically to do that angered me.  Brian Biles, VP of Product Management for EMC's BRS division, said with a wry smile, "so you're saying we've become a tobacco company."

I replied saying, "No, you've become a cigarette case manufacturer.  You shouldn't smoke, kids, but here's a really pretty gold case to hold your ciggies in."  I had a similar conversation with Mark Twomey (@storagezilla) on Twitter.

Since that time, I have come to a detente.  I still wouldn't buy one of these for my long term storage needs, but I can see why some other people might want to do so -- and I don't think those people are wrong or committing evil or data treason. This blog post is about how I got here from there.

Here were my arguments against this product:

There's no way that this could cost less than tape

Some of the messaging that I saw for the Archiver suggested that it was as affordable as tape.  That's simply not possible.  First, let's talk about what we're competing with. (For these comparisons, I am assuming you have either a tape system or a Data Domain box, and that what we're talking about is adding the cost of extra capacity to support long term storage of backups or archives.)

A backup or archive that is kept for that long is not kept in the tape library; it's put on a shelf.  (This is because chances are that it's never going to be read from.)  Therefore, the cost for tape is about $.02/GB, which is the cost of an LTO-5 tape cartridge.  The daily operational cost of that tape's existence is negligible, assuming it's onsite.

The last time I checked target dedupe appliances, they were about $1/GB after discounting.  I also saw a slide that this archiver is supposed to be about 20% cheaper than a regular Data Domain.  That puts it at around $.80/GB -- 40 times greater than the cost of a tape on a shelf.  And the daily operational cost of that disk is higher than the tape because it is going to be powered on.  (The Archiver does not currently support powering down unused shelves, although it may in the future.)

Then there is the issue of dedupe ratio.  The deduped disk price above is assuming a 20:1 dedupe ratio.  Dedupe ratios do not go up over time; they actually decrease.  This is because eventually we start making new data.  (The full backup you take today is going to contain quite a bit of new data when compared to the full backup from a year ago.)  Then there's the fact that the Archiver needs to start each tier (a collection of disks) with a new full backup, thus decreasing the overall dedupe ratio of the entire unit.  (It must do this in order to keep each tier self-contained.)  The result is that you will probably get a much lower dedupe ratio on your long term data than on your short-term data.  This increases your cost.

If you're going to do the right thing and use archive software to store data for several years (instead of backup software), any good archive software has single-instance-storage.  So if you're using archive software, you're going to get an even lower dedupe ratio.

Which brings me back to my belief that there is no way this can be anywhere near as inexpensive as tape.

The good news is that I didn't hear EMC saying that the Archiver is as cheap as tape when I saw them speak about it at EMC World.  When I talked to the EMC people at the show, I told them I had heard stories of EMC sales reps showing this unit cheaper than tape by using dedupe ratios of 100:1.  (The idea is that you're going to store 100 copies of the same full backups.)  They told me that any sales rep quoting ratios like is not speaking on behalf of EMC and talking out of his ...  Well, you know.

There's nothing that this unit offers that justifies that difference in price

Disk offers a lot of advantages when used for day-to-day backups.  It's a whole lot easier to stream during both backups and restores.  There is no question that it adds a lot of value there.  However, the idea of backups or archives that are stored long term is that no one reads them.  If they are reading them, it's for an electronic discovery request, where the amount of time you have to retrieve that is much greater than the time you typically have for a restore.  This increased amount of time is easily met with tape as your storage medium.  Disk offers no real advantage here.

When I said this, Mark Twomey pointed out that this unit offers regular data integrity checking of backups stored on it.  I informed him that if this were important, there are now two tape library manufacturers (Quantum & Spectralogic) that will be glad to do this for your tapes.

I will concede that disk does offer an advantage if you're using backups as your archives.  Having backups that will load instantly helps mitigate the issue of how many restores you're going to be doing to satisfy a complicated ediscovery request.

It's just wrong to store backups for many years

You should not be using your backups as archives.  You should not be using backups as archives.  If you ever get an ediscovery request for all of Joe Smith's emails for the last seven years -- and you happen to have a weekly full for each of the 364 weeks of that time frame -- you will remember what I said.

The thing is that EMC agrees. In fact, the EMC Archiver presentation starts with a few slides about how you should be doing real archiving; you should not be using your backups as archives.

They also said that they see this device as a transition device that can store both backups and archives.  Just because this device can store backups doesn't mean you have to store backups on it.  You can use proper archive software.  (But, if you did, I once again point out that your dedupe ratio will go down and therefore your effective cost per GB will go up.)

So what's changed, then?

I had a number of good conversations with EMC folks at last week's EMC World.  (Which, for the record, was a really big show.)  Some of those comments are above.  They know that this is not going to be cheaper than tape, and they're saying that anyone that is saying that is not being truthful.  They know that storing backups for years is wrong; they also know that more than half of the world does it that way.

The reason for the detente, however, is that I realize that many people hate tape.  I think they're wrong, as I've stated more than a few times.  There are plenty of IT departments that have a "get rid of tape" edict.  If the goal is to get rid of tape, the fact that the alternatives are much more expensive is not really an issue.  And if you're going to store backups for a really long time on disk, then at least EMC put some thought into what a disk system would need to do in order to do that right.  This includes things like fault isolation. If you lose one tier for whatever reason, you only lose the data on that array.  It includes things like scanning data occasionally to make sure it's still good.

Finally, Index Engines also announced an important product at EMC World that will help increase the value of the Archiver for those using it to store backups.  They already have a box that can scan tape backups and basically turn them into archives.  (One of the coolest products I've ever seen, BTW.)  They now support NFS, so you can point an Index Engines box at a DD Archiver and voila!  Those backups that you are storing on disk magically become fully searchable, ediscovery-ready archives.

Summary

Don't use your backups as archives.  Use archive software instead.  Tape is still the most economical destination for long term storage of backups or archives, and it's a pretty reliable one, too.  However, if you're going to store your backups or archives on disk for many years, there are worse places to put them than the EMC Data Domain Archiver.

Server virtualization does NOT cause storage explosion

Written by W. Curtis Preston Friday, 06 May 2011 16:55

Server virtualization doesn't kill storage.  People kill storage.  That's all I'm saying.

I get hot under the collar when I hear people say things like "server virtualization increases storage requirements by huge amounts."  They slam server virtualization with this comment, as if changing a server from being a physical one to being a virtual one somehow magically increases its size.  They list it as a reason that you shouldn't use server virtualization.

So I got a little irked when I heard the CEO of Symantec, Enrique Salem, say something like it in his keynote this week at Symantec Vision. (It was a great show, by the way.)  "Server virtualization increases storage use by 200% - 800%," he said.  When we had the media Q&A with him, this was the first question out of my mouth.  "What about moving a server from being physical to being virtual increases storage requirements?"  I asked a similar question of every other Symantec person I met with that day, as well as when I met VMware CTO, Steve Herrod.

In retrospect, I was probably a little hard on Mr. Salem during my Q&A.  Even Steve Herrod from VMware verified that the typical VMware customer does see such a storage explosion.  However, I still stand by my statement that this is not VMware's fault.  Moving to VMware does not cause your storage to magically explode.  Moving to VMware probably does "help" it happen, though.  Here are my thoughts on that.

VMware's design actually reduces storage use

The average virtual machine image (VMDK in VMware speak) is significantly smaller than the smallest disk drive you can buy to put into a server.  The smallest hard drive I can configure in a Dell server is 250 GB. You can create a thin-provisioned VMDK and it will consume only as much storage as it needs to, which is going to be far less than 250 GB.  I don't know Hyper-V as well as I do VMware, but I'm guessing it's similar.  I would also say that moving servers into VMware/Hyper-V means that you can put all those very duplicated images on a single storage volume that supports deduplication, removing that huge storage explosion.  You can't do that if you're using physical servers with discrete hard drives.

Many people buy their first "real" storage array when they buy VMware/Hyper-V

They may feel that this "forces" them to increase their storage costs, because they're used to just buying discrete hard drives -- often with no RAID or monitoring.  They then blame this increase in cost on VMware/Hyper-V.  I don't buy that either.  First, they didn't have to do that.  They could have bought a nice HP/Dell/IBM server with internal storage and run VMware on that.  The decision to buy a storage array is a second decision.  Second, if VMware "forces" them into the 21st century as far as storage management is concerned, so be it.  It's about time they have real storage.

Server virtualization often means a lot of test/dev VMs

This was Mr. Salem's point.  VMware/Hyper-V makes it really easy to have many, many different images of different configurations, so people create dozens or hundreds of VMs in their test/dev environment, and that causes a huge increase in storage.  I again say that you could continue to do in your dev/test lab whatever it was you did before you had VMware/Hyper-V, so it isn't VMware/Hyper-V's fault that you lab now uses 10 times more storage than it used to.  But it sure does make it easy, though, doesn't it?  I would also say that this increase in storage is accompanied by a huge increase in usability of the lab.

VM sprawl is evil and real and it eats up storage

This was the universal comment from most everyone I talked to.  When we step out of the test/dev world, it is a reality that when you are buying physical servers, there tends to be much more of an approval process.  When all you have to do to create a new server is click the right button on your mouse, you tend to create new "servers" very quickly.  Next thing you know, you have a whole lot more servers (and images of Windows/Linux) than you ever would have had if you had physical servers. VM sprawl is real, and it should be addressed with process and procedure.

VMware and Hyper-V are not the problem here.  What we do with it is the problem.  Yes, they make it much easier to do dumb things like VM sprawl, but blaming VMware and Hyper-V on your storage explosion is like blaming Ferrari for your tickets.  Just saying.

Someone in Ohio should have been fired

Written by W. Curtis Preston Friday, 22 April 2011 18:50

[This story originally happened in 2007, but I just learned about it, so I blogged about it.  Then I learned that it was a four-year-old story.    Everything here still applies, even if the actual story is old.  But I did re-edit the story and change it's title because the original wording seems a bit odd four years later. ]

Someone in the office of the State of Ohio should have been fired, and it isn't the guy who already got fired.  He should get his job back.  This story has me fuming.  I don't often write blogs like this, but here it goes.

The story as it was published in 2007 was "Intern loses backup tape with 800,000 SSNs on it. Intern fired."  The real story, in my opinion is what led up to this.  I read this article and this statement from the intern, and learned that the following allegedly happened in the State of Ohio:

1. The State of Ohio used (and may still use) unencrypted backup tapes to store SSNs and names

If your company or government entity is currently making tapes of any kind with SSNs on them then fix it.  Fixing this costs so little now that it is simply unforgivable not to be encrypting your backups tapes -- especially if you're handing them to a dude in a truck.  If you're handing them to an intern to take them home in a car… well, I really don't know what to say.

This is not a new problem.  It's not like we haven't had hundreds -- hundreds -- of exposures over the past 10 years that show how bad this practice is.  Ignorance of this problem simply isn't possible at this point.

2. Employees of The State of Ohio wanted to cover this up

They told the intern to not tell the police that one of the things stolen was a tape with sensitive data on it.  Seriously.  This tells me, of course, that they knew their unencrypted backup tape was a bad idea, and that they needed to keep others from knowing what they were doing.  It also tells me that they were liars.

3. The State of Ohio (a $52B/yr enterprise) had the money to hire $150/hr and $200/hr contractors full time, but didn't have the money to hire Iron Mountain (and still may not have it)

Seriously.  It had been the practice for apparently 10 years or more for someone to take the backup tapes home in their car.  Do I really need to say why this was stupid?  A hot car is not where tapes should ever be stored -- ever.  Asking someone who is off the clock to handle company property of any kind is also wrong.  Tapes -- especially unencrypted tapes -- should only be handled by professionals with procedures and policies to do such things.

No one ever told this young man what to do with this tape other than to bring it back the next day.  So not only was the practice to have him take it home, the practice was not to even give him any special instructions on how to handle the tape. Wow.

4. These same employees and their lawyer were bullies who needed a scapegoat and found one

The story about how they bullied this young intern into signing a resignation is just tragic.  He asked for an hour to think it over and they said no.  He asked for 20 minutes.  No.  He asked for 10 and they said no.  Just sign the paper.

Jared, if you're reading this, I would gladly act as an expert witness on your behalf for any kind of wrongful termination lawsuit you want to file. (I know this offer is a little late, but it's still out there.)

Someone in Ohio should have opened an investigation about the lack of security of taxpayers' personal information, as well as the details behind this story.  But if that never happened (and I can't find any evidence that it did), it's probably too late now.

Have we put tape out to pasture too soon?

Written by W. Curtis Preston Thursday, 21 April 2011 23:33

A week at NABShow (National Association of Broadcasters) and two days at Tape Summit last week have given me a chance to revisit my thoughts on tape.  Here's a brief summary of how my opinion of tape has changed over the years:

Stage 1: Tape was it.  It was all I knew. Backing up to disk was crazy, as it was too expensive.  (early 90s)
Stage 2: Tape was still it, but tape drives were getting too fast.  Multiplexing or disk staging was starting to be required.  Disk was too expensive to hold backups long term.
Stage 3: The dedupe craze hit.  It was both theoretically possible, as well as financially feasible (for some) to store all backups on disk -- and still have an offsite copy.
Stage 4: (Pretty recently).  I compared the pricing of today's dedupe systems to similarly-sized tape systems.  I was shocked at how expensive disk still was (4x-8x the price of tape).
Stage 5: (Today) I think we have unsuccessfully put a very good backup and archive target out to pasture and we should really reconsider that.

First, let me state that I am not saying that we should not have disk in a backup system, or that deduped systems are over-rated. What I am saying is that tape has more to offer than we've been giving it credit for lately. Here are some factors that came into my mind while considering this:

It costs 4-8 times more to acquire a disk-based backup system than it does to acquire an automated tape system.

While I've heard this from multiple sources, let me give you a real-life example to drive home this point.  I recently priced tape libraries and dedupe disk systems for a 20 TB shop, and I was surprised to learn that disk was actually still way more than the price of tape -- even after dedupe.  The average street price of the tape libraries I was considering was about $15K, and the average price of the dedupe systems was about $60K.   Since the customer was getting rid of their (very old) tape library, their choices were:

A) Buy a new tape library, copy tapes and hand them to a dude in a truck  ($15K)

B) Buy a dedupe system AND a tape library.  Copy from the dedupe system to the tape library, and then hand tapes to a dude in a truck. ($60K + $15K)

C) Buy two dedupe systems and replicate between them (no truck needed) ($120K)

Option C was 8 times more expensive than Option A and was out of the question.  While it meant they could get rid of their Iron Mountain bill, they did not believe they could ever save enough money to recoup that additional $105K.  Option B offered no cost savings, so it was difficult to justify the additional $60K.  I pointed out that Option A (if done correctly) requires a disk cache in front of their tape library, but they informed me that they were already doing that.  (Based on their throughput requirements, though, adding a disk cache wouldn't have added that much to the price.)

You can undoubtedly make an argument that a backup-to-disk system is easier to manage than a hybrid tape system, but the simple fact is that the disk system will be more expensive to purchase.

Tape actually has a better bit error rate than disk

For those unfamiliar with the concept of bit error rate (BER), the following definition from Wikipedia should be helpful:

"The bit error rate or bit error ratio (BER) is the number of bit errors divided by the total number of transferred bits during a studied time interval. … The bit error probability p^e is the expectation value of the BER. The BER can be considered as an approximate estimate of the bit error probability. This estimate is accurate for a long time interval and a high number of bit errors."

LTO-5 has a bit error rate of 1:10^17.  The TS1130 from IBM & the T10000C from Oracle both have a BER of 1:10^19.  SATA disk has a BER of 1:10^15 for SATA (SAS/FC is 1:10^16 but no one is using that for backup or archive).  This will probably come as a surprise to many people.  Tape has actually gotten so good at writing data, it is more reliable at writing data than disk!

While 10^15 may look really close to 10^17, it's not.  When it's bits we're talking about, it's the difference between 113 TB and 11.1 PB!  It means you are 100 times more likely to have bad data on disk than you are on an LTO-5 tape drive, and 10,000 times more likely than if the data is stored on a T1000C or TS1130 drive!

Tape uses less power than disk

Every time I calculate power consumption for tape systems vs. disk systems, tape systems win.  The reason for this is that tapes in slots take up no power at all, tape drives use very little power while they're not doing anything, and you need far fewer tape drives than you need disk drives.  I recently did a comparison for a 20 TB shop that resulted in at least a 2X difference in power consumption, and that included enough disk to do disk staging before the tape system.  (I plan to publish this once I double/triple check my numbers, but right now I feel pretty safe in saying at least a 2X difference.)

You buy the system once; you power it all day long every day.

Longterm (5+ years) storage of data on disk is not compatible with the typical lifecycle of disk, but it is compatible with tape.

This one is something we don't talk about.  An individual tape is made to hold data much longer than an individual disk, and the lifecycle of most tapes is much longer than the lifecycle of most datasets.  You cannot say the same about disks.  Storing data on disks for more than 5 years automatically assumes that you're going to migrate data from one disk unit to another.

In addition to the media, it is also very common for tape libraries and tape drives to outlast the disk systems sitting next to them. Where most companies migrate data at the end of the depreciation cycle for disk, they tend to keep their tape libraries and drives much longer than that.  They also tend to swap out their drives in the tape libraries; the same is not true in disk units. If you find a disk system in your data center older than five years, I'd be shocked.

What's the problem then?

Let's throw out the claims I've heard:

1. Tape has bitrot

So does disk.  It's called magnetism.  It happens.  The chances of bitrot happening on tape are far less than the chances of it happening on disk.

2. Tape is flimsy

Tell you what.  Move disks around the way you move tapes around and see how flimsy they are.

3. 80% of tape restores fail.

This Gartner statistic has been thrown around so much and I really don't know where Gartner got this number from, but it's out there.  What I can tell you is that in my entire career of working with backups, I've only had one or two restores that failed due to an actual bad tape -- and that's why we make copies.  But I can tell you of dozens of situations where bad disk drives caused me all sorts of headaches.

I can also tell you that most of the restore failures I've seen have been caused by human error - not tape failure.

4. Tape is too slow

Baloney.  Check your facts again.  There isn't a disk drive alive that can keep up with the speed of today's tape drives.

5. Tape is hard to make happy during backups & restores

Agreed.  This is why I believe strongly in using at least disk caching.  I would never design a system that uses just tape to do backups at this point.  I'm actually OK with all of the designs mentioned above (in the A, B, C list).  I think dedupe systems are awesome, and the idea of replicating to another one is even better.  But I also know that doing this is more expensive than the alternative. The other thing I know is that it can't possibly be cheaper to store data on disk for many, many years, and it may even be risky to do so.  (See my comments on BER.

What I'm really making an argument for is the use of tape for long term archiving, and as a less expensive way of getting data offsite.  (Less expensive than having a second dedupe system and replicating to it.)

6. Tapes go bad sitting on the shelf and you never know they're bad until you need them

That is correct.  This is why both Spectralogic and Quantum have come up with products to proactively scan your old archives to find and fix any corruption issues before you need a given tape.  If it finds something wrong, it can be fixed by copying the other copy that you have.

Conclusion

Tape can be your friend for long term archives and cheap offsite storage.  Don't dismiss it so lightly.

SNW Reflections

Written by W. Curtis Preston Thursday, 07 April 2011 18:46

I attended my first SNW conference in over two years last week.  (My previous employer was really good at scheduling me for competing events, so I wasn't able to go.)  Most of my thoughts about what I saw go hand-in-hand with Don Jenning's thoughts here.  I agree that it was very cool to see The Cube guys and to know that Infosmack was recording in the social media room.  (I got to sit in on one of those recordings with none other than Dr. Dedupe, Larry Freeman.  I look forward to that podcast.)  And it was my first experience live tweeting from a big event that had a hashtag (#SNWUSA).  That was definitely very cool.  Here are my thoughts, some of which are about SNW, and others that are about the vendors.

First, SNW is still very much alive and well, but it's definitely not the show that it used to be.  There was a time when your company was considered dead or dying if your major reps were not in attendance.  That appears to be the case no longer.  Sure, there were a lot of companies there.  But there were a lot of companies not there as well.  Perhaps it's because there are other options now to talk to analysts and other vendors (like The Exec Event) that pull budget money away from this.  Perhaps its also because there is now a whole other group of people (bloggers) that you also need to reach out to.  Vendors are either doing their own thing, like HDS' recent event, or they're sponsoring one of the Tech Field Day events (which cost much less than doing it yourself).  Either way, that's more money being pulled away from this "industry event."

The other reason (if not the main reason) that vendors do a show like SNW is for new leads.  New leads means new end users to talk to.  I remain unconvinced that this is a show to find such people in significant numbers.  Any time I ever scanned the room, I saw way more vendor, analyst and press badges than end user badges.  I also kept seeing the same end users over and over.

As a former end-user, and someone who continues to see himself as an advocate for such, I am simply not drawn to the content offered by SNW.  With very few exceptions, it's one talk after another given either by a vendor or someone the vendor is paying to be there (either an analyst or an end user who was sponsored to the show by a vendor).  I do like the SNIA tutorials, BTW, because I know they try really hard to keep those vendor-neutral.  But every other presentation was (IMO) simply a marketing presentation.  They should all be titled: The Correct Way to do X, Which Can Only be Done by Buying Our Product.  I'm not saying that these sessions are bad, mind you.  I'm just saying that they don't draw me.

I'd much rather hear from independent thinkers that are mostly absent from this conference.  There's another conference that these people tend to speak at that tends to draw a much larger percentage of end users, and that is Storage Decisions.  I like that show for its content and the make-up of its audience.  It's a real shame that I seem to be uninvited.  This upcoming Storage Decisions is the first one I haven't spoken at in a really long time.  Although no official word was given, it's no coincidence that this happened right after I started hosting my own road show: Backup Central Live.  No worries.  I've got plenty of other things to do.

In my final thoughts on SNW, I want to pass out the worst-designed booth award.  Booth design experts tell you that you have 7 seconds to catch someone's attention.  Anobit's booth display was whitespace with their company name and the phrase "add another bit."  What are they, anti-dedupe software? Are they hardware, software, service, what?  Then there was RackSpace.  Their booth, which I took a picture of, said they were the world leader in hosting and cloud computing, but never said the name of their company.  This is something not lost on the people that worked the booth, because they took the ghetto little white sign (hung on the pipe and drape before the vendor gets there so they know which booth to go to) and hung it up and over their display. See this photo for what I'm talking about.  So we have a booth with a company name and no description, and a booth with a very big description but no company name.  I declare a tie.

I do have some thoughts to post about companies that I met and got briefings from.  They will come soon.

OK, encrypt your disks after you've done everything else

Written by W. Curtis Preston Thursday, 31 March 2011 20:53

The other day I wrote a blog entry that said encrypt your tapes but not your disks.  My fundamental premise was that encrypting data at rest in your disk drives only protects from the thing that will never happen: someone walking out with an entire disk array under their arm.  Single disk drives yanked out of the array (more likely) were not going to be any use to anyone even if you didn't encrypt them.

Turns out I was wrrrmph.

Turns out that the most sensitive data is probably very recoverable from a RAID-ed disk drive.  A whole lot of 1K database rows can be stored in a 64K block of data stored on an individual disk drive in a parity-protected disk array.  (See the comments from my previous post for details.)  And it turns out that you can't degauss hard drives and return them, so there's also the exposure of what happens when you return a disk drive to the manufacturer.

I was wrong about the risk, but I still think there are bigger fish to fry in the datacenter.  Sticking with just my world, we've got companies that:

1. Don't copy their backups (they keep only one copy of every disk or tape they make)
2. Don't send their backups offsite
3. Wait a week or two before sending their backups offsite
4. Don't back up their laptops
5. Back up their remote offices using tapes that aren't copied and/or aren't ever sent anywhere

If you've got data that isn't being backed up and isn't being stored in a different location than it was backed up, you will lose data.  This isn't a "maybe some guy might steal a disk drive and if he does he might be able to read some data on it."  Every company in the world has lost a disk drive somewhere in their environment.  I'm a very small company and I lost four this year alone.

The number one reason people telling me they're on the list above is money.  So my point is that if you're spending money on encrypting your disks, but you're not backing your stuff up in the first place -- you've got your priorities all wrong.

I have the same opinion when I see people spending money make their backup server highly available, but they don't have money to make a second copy of their backups.  Who cares if your backup server goes down for an hour?  It's a big deal, but the only app that's down is backup -- not production.  But the chances of you losing data because you had a failed tape and no copies is much higher.  Save the money on the HA software for the backup server and spend it on something that actually makes your backups better.

I also think you can minimize this risk by doing a few things, all of which are cheaper than full disk encryption:

1. Strong physical security in the data center.  Plenty of good things you can do.
2. Video surveillance in the data center
3. Identify really sensitive data and encrypt it in the application
4. Strong physical security (locks) on the disk arrays themselves.  Prevent someone from grabbing a disk drive.
5. Monitoring on same.  If a disk drive is taken, you should be immediately notified.

Like I said, there are lots of things you can do (and should do) that don't cost near as much as full disk encryption and most of which you should be doing anyway.

Announcing Backup Central Live! Q2 cities & dates

Written by W. Curtis Preston Wednesday, 30 March 2011 19:17

After our very successful five-city tour in Q1, we are now announcing cities, dates, and locations for our Q2 events.  Those of you that live in Raleigh, Boston, Philadelphia, Dallas, & Minneapolis are the next folks to be able to attend a Backup Central Live event.   In addition, those of you interested in deduplication, continuous protection of servers, and backup of laptop data have three webinars to choose from next month.

Here are all of our upcoming events and where you can register for them.

See you there!  If you have any questions about events, feel free to This e-mail address is being protected from spambots. You need JavaScript enabled to view it .