Have we put tape out to pasture too soon?

A week at NABShow (National Association of Broadcasters) and two days at Tape Summit last week have given me a chance to revisit my thoughts on tape.  Here's a brief summary of how my opinion of tape has changed over the years:

Stage 1: Tape was it.  It was all I knew. Backing up to disk was crazy, as it was too expensive.  (early 90s)
Stage 2: Tape was still it, but tape drives were getting too fast.  Multiplexing or disk staging was starting to be required.  Disk was too expensive to hold backups long term.
Stage 3: The dedupe craze hit.  It was both theoretically possible, as well as financially feasible (for some) to store all backups on disk — and still have an offsite copy.
Stage 4: (Pretty recently).  I compared the pricing of today's dedupe systems to similarly-sized tape systems.  I was shocked at how expensive disk still was (4x-8x the price of tape).
Stage 5: (Today) I think we have unsuccessfully put a very good backup and archive target out to pasture and we should really reconsider that.

[Update: Because people tend to read my old articles, I'm going to update this one almost a year later to reflect my current position on tape.]

[Update2: I just wrote this blog post about my response to another article about this topic.]

First, let me state that I am not saying that we should not have disk in a backup system, or that deduped systems are over-rated. What I am saying is that tape has more to offer than we've been giving it credit for lately. Here are some factors that came into my mind while considering this:

It costs 4-8 times more to acquire a disk-based backup system than it does to acquire an automated tape system.

[Update 3/9/12: Pricing obviously changes all the time, and prices on disk have come down since this original post.  I even have some vendors that claim to be as cheap as tape on the initial purchase of one disk system vs one disk robot in some situations. ]

While I've heard this from multiple sources, let me give you a real-life example to drive home this point.  I recently priced tape libraries and dedupe disk systems for a 20 TB shop, and I was surprised to learn that disk was actually still way more than the price of tape — even after dedupe.  The average street price of the tape libraries I was considering was about $15K, and the average price of the dedupe systems was about $60K.   Since the customer was getting rid of their (very old) tape library, their choices were:

A) Buy a new tape library, copy tapes and hand them to a dude in a truck  ($15K)

B) Buy a dedupe system AND a tape library.  Copy from the dedupe system to the tape library, and then hand tapes to a dude in a truck. ($60K + $15K)

C) Buy two dedupe systems and replicate between them (no truck needed) ($120K)

Option C was 8 times more expensive than Option A and was out of the question.  While it meant they could get rid of their Iron Mountain bill, they did not believe they could ever save enough money to recoup that additional $105K.  Option B offered no cost savings, so it was difficult to justify the additional $60K.  I pointed out that Option A (if done correctly) requires a disk cache in front of their tape library, but they informed me that they were already doing that.  (Based on their throughput requirements, though, adding a disk cache wouldn't have added that much to the price.)

You can undoubtedly make an argument that a backup-to-disk system is easier to manage than a hybrid tape system, but the simple fact is that the disk system will be more expensive to purchase.

Tape actually has a better bit error rate than disk

For those unfamiliar with the concept of bit error rate (BER), the following definition from Wikipedia should be helpful:

"The bit error rate or bit error ratio (BER) is the number of bit errors divided by the total number of transferred bits during a studied time interval. … The bit error probability p^e is the expectation value of the BER. The BER can be considered as an approximate estimate of the bit error probability. This estimate is accurate for a long time interval and a high number of bit errors."

LTO-5 has a bit error rate of 1:10^17.  The TS1130 from IBM has a bit error rate of 1:10^20, & the T10000C from Oracle both have a BER of 1:10^19.  SATA disk has a BER of 1:10^14 for SATA (SAS/FC is 1:10^15 but no one is using that for backup or archive).  This will probably come as a surprise to many people.  Tape has actually gotten so good at writing data, it is more reliable at writing data than disk!

While 10^15 may look really close to 10^17, it's not.  When it's bits we're talking about, it's the difference between 113 TB and 11.1 PB!  It means you are 100 times more likely to have bad data on disk than you are on an LTO-5 tape drive, and 10,000 times more likely than if the data is stored on a T1000C or TS1130 drive!

Tape uses less power than disk

Every time I calculate power consumption for tape systems vs. disk systems, tape systems win.  The reason for this is that tapes in slots take up no power at all, tape drives use very little power while they're not doing anything, and you need far fewer tape drives than you need disk drives.  I recently did a comparison for a 20 TB shop that resulted in at least a 2X difference in power consumption, and that included enough disk to do disk staging before the tape system.  (I plan to publish this once I double/triple check my numbers, but right now I feel pretty safe in saying at least a 2X difference.)

You buy the system once; you power it all day long every day.

Longterm (5+ years) storage of data on disk is not compatible with the typical lifecycle of disk, but it is compatible with tape.

This one is something we don't talk about.  An individual tape is made to hold data much longer than an individual disk, and the lifecycle of most tapes is much longer than the lifecycle of most datasets.  You cannot say the same about disks.  Storing data on disks for more than 5 years automatically assumes that you're going to migrate data from one disk unit to another.

In addition to the media, it is also very common for tape libraries and tape drives to outlast the disk systems sitting next to them. Where most companies migrate data at the end of the depreciation cycle for disk, they tend to keep their tape libraries and drives much longer than that.  They also tend to swap out their drives in the tape libraries; the same is not true in disk units. If you find a disk system in your data center older than five years, I'd be shocked.

What's the problem then?

Let's throw out the claims I've heard:

1. Tape has bitrot

So does disk.  It's called magnetism.  It happens.  The chances of bitrot happening on tape are far less than the chances of it happening on disk. [Update: See this post for further info on this.]

2. Tape is flimsy

Tell you what.  Move disks around the way you move tapes around and see how flimsy they are.

3. 80% of tape restores fail.  [Update 3/9/12: This is a fake statistic that never existed. See my updated blog post.]

This Gartner statistic has been thrown around so much and I really don't know where Gartner got this number from, but it's out there.  [Update: This Gartner statistic never existed.] What I can tell you is that in my entire career of working with backups, I've only had one or two restores that failed due to an actual bad tape — and that's why we make copies.  But I can tell you of dozens of situations where bad disk drives caused me all sorts of headaches.

I can also tell you that most of the restore failures I've seen have been caused by human error – not tape failure.

4. Tape is too slow

Baloney.  Check your facts again.  There isn't a disk drive alive that can keep up with the speed of today's tape drives.

5. Tape is hard to make happy during backups & restores

Agreed.  This is why I believe strongly in using at least disk caching.  I would never design a system that uses just tape to do backups at this point.  I'm actually OK with all of the designs mentioned above (in the A, B, C list).  I think dedupe systems are awesome, and the idea of replicating to another one is even better.  But I also know that doing this is more expensive than the alternative. The other thing I know is that it can't possibly be cheaper to store data on disk for many, many years, and it may even be risky to do so.  (See my comments on BER.

What I'm really making an argument for is the use of tape for long term archiving, and as a less expensive way of getting data offsite.  (Less expensive than having a second dedupe system and replicating to it.)

6. Tapes go bad sitting on the shelf and you never know they're bad until you need them

That is correct.  This is why both Spectralogic and Quantum have come up with products to proactively scan your old archives to find and fix any corruption issues before you need a given tape.  If it finds something wrong, it can be fixed by copying the other copy that you have.

Conclusion

Tape can be your friend for long term archives and cheap offsite storage.  Don't dismiss it so lightly.


Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at Sullivan Strickler, which helps companies manage their legacy data

12 comments
  • Curtis, re: point 6: We’ve been very happy with the Imation Secure Scan product:

    (http://www.imation.com/en-us/Imation-Products/Tape-Cartridges–Accessories/Secure-Scan/)

    we’re at a point where we are almost at steady-state with our tapes…almost as many are coming back into use as are going offsite. We’ve had several instances where we ID’d media at the end of it’s useful life BEFORE we put new data on it. Plus the gizmo is dang easy to use and economical.

    I don’t sell them, I’m just a happy owner….

    Cheers!

  • @David

    That’s nice to hear, and that looks like a good solution for smaller sites and those that cannot use the Quantum/Spectralogic offerings. The downside, of course, is that you must put tapes in there one at a time. How long does it take to scan a tape?

  • @Curtis

    Tape scans + locking take about 3 seconds per tape. It also works on tapes that are inside their cartridge cases, which is convenient. Our operators process 50-75 tapes a day and they seem happy with it (and they’d def let me know otherwise!).

  • Great article.

    Just to note a small typo…

    “Option A was 8 times more expensive than Option A and was out of the question.”

    I believe you meant…

    Option C was 8 times more expensive than Option A and was out of the question.

  • The imation “Secure Scan” product that @David recommends does look pretty cool, but it’s not going to do much for the specific problem of unused tapes losing data as they age.

    It’s basically reading the metadata off of the MIC chip, with the added option of telling that MIC chip to password-lock that tape. (all of which can be done through the poly case…)

    That MIC metadata can certainly give some indirect indications of the validity of the tape (age, hours in service, etc) as well as indications of any HISTORICAL low-level recoverable errors that happened during use.

    But the one thing that MIC can’t give is any indication of problems that developed while the tape was sitting on the shelf.

  • First off, nice seeing you in Las Vegas at Interop. As you saw from our product line, I very much have the opposite point of view. Let me address a few of your points.
    1) bitrot. Yes, as you point out hard drives can have the occasional bitrot too. However, the media in a hard drive is hermetically sealed. We have had customers using tape in rock crushing labs or construction sites and the systems fail within 6 months. Curtis, they fail over and over and over. We put in our systems and our drive trays using standard SATA drives have held up much better

    2) Our customers DO move hard drives around the way they do tapes. I don’t recommend dropping or throwing them around, but overall hard drives stand up well. Check out the video on our site. I will admit the higher density vertically encoded drives are more sensitive to dropping than an 80GB drive. But honestly I’ve not done drop tests on high density LTO4 and 5 tapes and I bet they’d mess up too.

    3. 80% of tape restores fail. Not sure if Gartner popped this statistic out of someone’s imagination but I’ve been quoting at least 50% failure for years because based on MY OWN experience. Perhaps the difference between you and I is the tape brands and usage. With tape I always did FULL backups because I didn’t trust tapes with incrementals. Full backups work the tape systems HARD and perhaps failure becomes a self-fulfilling prophecy. Brands matter too. For example going WAY back in history I remember Colorado Memory cheapy QIC40 and 80 tapes were awful, Maynard tapes sucked. many DLT tapes sucked only a little less. DAT tapes sucked. Phillips Onstream was awful. But I do admit LTO has been a bit better.

    “I can also tell you that most of the restore failures I’ve seen have been caused by human error – not tape failure.” That’s what the tape vendors tech support would have us believe. Wasting our time with SCSI termination and running cleaning tapes on the phone. I call B.S. I’m telling you tapes failed and it WASN’T My fault. Unless you say it’s because I insisted on daily full backups and I wore the darn things out. But I insisted because they failed so often.

    4. “Tape is too slow.” OK I propose a race. I’ll use a 3TB High-Rely system hooked up via eSATA or USB3 using ShadowProtect software to backup. You can use LTO tape via SCSI or SAS and whatever software you want. I bet you we will give you a very respectable race using a daily swap system that costs less.

    5. “The other thing I know is that it can’t possibly be cheaper to store data on disk for many, many years, and it may even be risky to do so.”
    Curtis – I disagree. First off I maintain tapes FAIL after about 50 uses whereas hard drives last 2-3 years even when used daily. 1.5TB LTO 5 tapes are around $90 right now. Check NewEgg.com and you will see 1.5TB hard drives are $70. Hard drives are hermetically sealed and tapes aren’t. I can’t imagine a single reason to expect tape to last on the shelf longer than a drive assuming both are stored in moisture proof, static free environments.

    6. “Tapes go bad sitting on the shelf and you never know they’re bad until you need them” Yes and maybe you argue that hard drives do too… but you can check them a heck of a lot faster than checking a tape.

  • @Rex

    I think this is a bit like that old story of two blind guys describing an elephant they are “seeing.” They both describe it as very different based on the parts they were touching. But let me try to respond to your response.

    >”as you point out hard drives can have
    >the occasional bitrot too.”

    You concede my point. My experience is that bitrot is actually a greater problem on disk than on tape. I believe it has something to do with being constantly powered and having a magnetic device passing over the bits, but that’s just me.

    >the media in a hard drive is
    >hermetically sealed. We have had
    >customers using tape in rock crushing
    >labs or construction sites and the..

    that is not bitrot, that is misuse. But in that (albeit edge) use case I’ll concede that disk is a better choice.

    >2) Our customers DO move hard drives >around the way they do tapes.

    And I can tow a motor home with my Prius. That doesn’t make it right. They’re not designed to do that. Period.

    >3. …I’ve been quoting at least 50%
    >failure for years because based on MY
    >OWN experience. … I remember Colorado
    >Memory cheapy QIC40 and 80 tapes were
    >awful, Maynard tapes sucked. many DLT
    >tapes sucked only a little less. DAT
    >tapes sucked. Phillips Onstream was
    >awful..

    Holy, crap, dude. You need to get out more. Are you REALLY telling me tape is unreliable because of a tape drive that was out of vogue before I entered the industry 18 years ago? Seriously?

    And SCSI Terminators? Seriously? I haven’t seen one of those in forever.

    >That’s what the tape vendors tech
    >support would have us believe.

    No, this is what I have seen. Most restores that I’ve seen fail had everything to do with HOW they were done, not on what media they were stored.

    >4. “Tape is too slow.” OK I propose a
    >race. I’ll use a 3TB High-Rely system >hooked up via eSATA or USB3 using
    >ShadowProtect software to backup. You
    >can use LTO tape via SCSI or SAS and
    >whatever software you want. I bet you we
    >will give you a very respectable race >using a daily swap system that costs >less.

    The point of my point there is that tape is not the thing slowing things down. LTO-5 is faster than ANY disk device on the planet. Check the stats. And the T10000C and TS1150 drives are even faster.

    >5. “The other thing I know is that it
    >can’t possibly be cheaper to store data
    >on disk for many, many years, and it may
    >even be risky to do so.”
    >Curtis – I disagree. First off ….

    You are arguing with a different point. I am talking about long term storage on tape. You are talking about repeated writes to tape. Those are two different things.

    >6. “Tapes go bad sitting on the shelf
    >and you never know they’re bad until you
    >need them” Yes and maybe you argue that
    >hard drives do too… but you can check
    >them a heck of a lot faster than >checking a tape.

    Actually, no you can’t. Assuming the same amount of data and assuming the tape has a faster throughput rate (which is does, by far), you can verify data on tape much faster. (Which, BTW two of the three big tape library manufacturers now support.)

    Having said all of this, I’m in support of disk-based backup systems. The point of my post is that tape is the best medium for long term storage of that backup, or for long term storage of an archive.

  • @Darren

    We don’t just differ on interpretations. We don’t agree on the facts. You have multiple completely incorrect factual statements in your report, including the Gartner quote, the Yankee quote, the capacity & speed of an LTO drive.

    Your LTO quote is also biased. No standalone customer would buy an LTO-5 drive, so doing the quote based on that is crazy. At best, they would use an LTO-4, which is about half the price you quoted (http://www.buy.com/pr/product.aspx?sku=206928843&sellerid=34459876), and it comes with a cable. I also found a SAS controller about 1/6th the price you quoted. (http://www.google.com/products/catalog?q=sas+controller&oe=utf-8&rls=org.mozilla:en-US:official&client=firefox-a&um=1&ie=UTF-8&tbm=shop&cid=635180723698798233&sa=X&ei=R_5aT8H8FPT1sQLFtMzZDQ&ved=0CJEBEPMCMAM). That’s $1249 + $44. Add 5 tapes for $23 and that’s another $115 for a total of $1408. At the typical compression rate of 1.5:1 it’ll hold about 1.2 TB per tape. That is way more capacity than most typical shops are going to need and less expensive than you quoted.

    Having said that, I would never recommend a small company do that, or do what you’re recommending. Hardly anyone recommends backing up directly to tape anymore, and even fewer people would recommend backing up to removable 3.5″ disk drives. Besides the fact that they’re not designed for travel (like 2.5″ drives), you’ve still got many of the same reasons why I don’t recommend removable tape for small shops. The amount of manual intervention is just too much more most people.

  • Curtis,
    Your research on the accuracy of tape statistics is commendable and I applaud you for calling attention to the inaccurate quotes. I’m guilty of Google research and quoting from mainstream articles because I do not have access to Gartner data directly. I will remove unverified references in my post by Monday night. All I can say is that my experience restoring from tape has not been good and I’m obviously not alone. Let me address some other things:
    “No standalone customer would buy an LTO-5 drive, so doing the quote based on that is crazy”. I disagree completely and “crazy” is a little harsh. They SELL the things don’t they? Else where did my screenshot come from? Besides, I chose LTO-5 for my comparison because it is state of the art, and because our products tend to run BIGGER and I wanted to compare apples to apples as much as possible so I focused on 1.5TB vs 1.5TB. How can I be more fair? Most of our sales are 2TB and 3TB. So I had to downsize my quote. I can assure you many small shops need that much backup and more. We sell RAIDPacs at 6TB and people sometimes ask for more.
    I’ll have to accept your $54 Promise SAS controller, although I’m curious if it would work with (and be recommended by the vendor for) the tape drive. I’ll have to think about whether to change my calculation on that. Good find though.
    “At the typical compression rate of 1.5:1 it’ll hold about 1.2 TB per tape” I addressed this in my post. Why do you (and tape vendors) insist on comparing compressed sizes of tape to uncompressed disk? ShadowProtect and other imaging software automatically compress their images too (and even if they didn’t you can right click a folder or drive and tell Windows to do it). So why couldn’t I sell a 3TB and advertise it as holding about 6TB? Because it’s not the norm – and it’s not even very ethical – that’s why. So why do tape vendors get the magical benefit of quoting compressed capacities and speeds? Lets keep things apples to apples OK?
    “Hardly anyone recommends backing up directly to tape anymore”. Curtis, I feel you are exposed to larger backup vendors and larger corporate clients. Small customers can’t always afford the time or money of backing up to NAS/SAN/BDR/Dedupe Backup Appliance AND then buying a separate tape drive and doing another migration. Yes Yes I know – D2D2 Tape is the gold standard in corporate world. I issue this challenge: Call your backup buddies at Symantec who sell Backup Exec and ask how many customers still use their software to go straight to tape or even straight to a low cost local NAS (and nowhere else). I bet the answer is A LOT of them. In fact I bet MOST of them.
    “even fewer people would recommend backing up to removable 3.5″ disk drives”. Admittedly we have a niche market. But we sell both 3.5 and 2.5” removable drives and I can tell you the 3.5” outsell the 2.5” by quite a bit because of the data capacity. When modern 3.5” drives shut down they park the heads, at which point they are robust & transportable. Manufacturers SHIP the things from the factory and, even with the gorillas at UPS abusing them the whole way, most of them seem to show up in working condition.
    Editor’s note: Too bad you don’t allow me to link from your blog to my blog, the way you are doing here. Since I complemented your well written blog and link to it at the bottom of mine, the link and our subsequent comments are effectively cross linked, but your point is well taken. I’ll look into allowing comments to my blogs in WordPress.