Get rid of tape? Inconceivable!

Stephen Manley published a blog post today called “Tape is Alive? Inconceivable!”  To which I have to reply with a quote from Inigo Montoya, “You keep using that word. I do not think it means what you think it means.”  I say that because, for me, it’s very conceivable that tape continues to play the role that it does in today’s IT departments.  Yes, its role is shrinking in the backup space, but it’s far from “dead,” which is what Stephen’s blog post suggests should happen.

He makes several good points as to why tape should be dead by now.  I like and respect Stephen very much, and I’d love to have this discussion over drinks at EMC World or VMworld sometime.  I hope that he and his employer see this post as helping him to understand what people who don’t live in the echo chamber of disk think about tape.

Stephen makes a few good points about disk in his post.  The first point is that the fastest way to recover a disk system is to have a replicated copy standing by ready to go.  Change where you’re mounting your primary data and you’re up and running.  He’s right.  He’s also right about snapshots or CDP being the fastest way to recover from logical corruption, and the fastest way to do granular recovery of files or emails.

In my initial post on the LinkedIn discussion that started this whole thing, I make additional “pro-disk” points. First, I say that tape is very bad at what most of us use it for: receiving backups across a network — especially incremental backups.  I also mention that tape cannot be RAID-protected, where disk can be. I also mention that disk enables deduplication, CDP, near-CDP and replication — all superior ways to get your data offsite than handing tape to a dude in a truck.  I summarize with the statement that I believe that disk is the best place for day-to-day backups.

But…

Disk has all of the above going for it.  But it doesn’t have everything going for it, and that’s why tape isn’t dead yet — nor will it be any time soon.

I do have an issue or two with the paragraph in Stephen’s post called “Archival Recovery.”  First, there is no such thing.  It may seem like semantics, but one does not recover from archives; one retrieves from archives.  If one is using archive software to do their archives, there is no “recover” or “restore” button in the GUI.  There is only “retrieve.”  Stephen seems to be hinting at the fact that most people use their backups as archives — a fact on which he and I agree is bad.  Where we disagree is whether or not moving many-years-old backup data to disk solves anything. My opinion is that the problem is not that the customer has really old backups on tape.  The problem is that they have really old backups.  Doing a retrieval from backups is always going to be a really bad thing (regardless of the media you use) and could potentially cost your company millions of dollars in fines and billions of dollars in lost lawsuits if you’re unable to do it quickly enough.   (I’ll be making this point again later.)

Cost

Disk is the best thing for backups, but not everyone can afford the best.  Even companies that fill their data centers with deduplicated disk  and the like still tend to use tape somewhere — mainly for cost reasons.  They put the first 30-90 days on deduped disk, then they put the next six months on tape.  Why?  Because it’s cheaper.  If it wasn’t cheaper, there would be no reason that they do this.  (This is also the reason why EMC still sells tape libraries — because people still want to buy them.)

Just to compare cost, at $35 per 1.5 TB tape, storing 20 PB on LTO-5 tapes costs $22K with no compression, or $11K with 2:1 compression.  In contrast, the cheapest disk system I could find (Promise VTrak 32TB unit) would cost me over $12M to store that same amount of data.  Even if got a 20:1 dedupe ratio in software (which very few people get), it would still cost over $600K (plus the cost of the capacity-based dedupe license from my backup software company).

It’s also the cheapest way to get data offsite and keep it there.  Making another copy on tape at $.013/GB (current LTO-5 pricing) and paying ~$1/tape/month to Iron Mountain is much cheaper than buying another disk array (deduped or not) and replicating data to it.  The disk array is much more expensive than a tape, and then you need to pay for bandwidth — and you have to power the equipment providing that bandwidth and power the disks themselves.  The power alone for that equipment will cost more than the Iron Mountain bill for the same amount of data — and then you have the bill for the bandwidth itself.

Now let’s talk about long-term archives.  This is data stored for a long time that doesn’t need to be in a library.  It can go on a shelf and that’ll be just fine.  Therefore, the only cost for this data is the cost of the media and the cost of cooling/dehumidifying something that doesn’t generate heat.  I can put it on a tape and never touch it for 30 years, and it’ll be fine (Yes, I’m serious; read the rest of the post).  If I put it on disk, I’m going to need to buy a new disk every five years and copy it.  So, even if the media were the same price (which it most certainly is not), the cost to store it on disk would be six times the cost of storing it on tape.

Unlimited Bandwidth

Never underestimate the bandwidth of a truck.  ‘Nuf said.  Lousy latency, yes.  But definitely unlimited bandwidth.

Integrity of Initial Write

LTO is two orders of magnitude better at writing bits than enterprise-grade SATA disks, which is what most data protection data is stored on.  The undetectable bit error rate of enterprise SATA is 1:10^15, and LTO is 1:10^17.  That’s one undetectable error every 100 TB with SATA disk and one undetectable error every 10 PB with LTO.  (If you want more than that, you can have one error every Exabyte with the Oracle and IBM drives.)  I would also argue that if one error every 10 PB is too much, then you can make two copies — at a cost an order of magnitude less than doing it on disk.  There’s that cost argument again.

Long-term Integrity

As I have previously written, tape is also much better than disk at holding onto data for periods longer than five years.  This is due to the  physics of how disks and tapes are made and operated.  There is a formula (KuV/kt) that I explain in a previous blog post that explains how the bigger your magnetic grains are, the better, and the cooler your device is, the better  The resulting value of this formula gives you an understanding of how well the device will keep its bits in place over long periods of time, and not suffer what is commonly called “bit rot.”   This is because disks use significantly smaller magnetic grains than tape, and disks run at very high operating temperatures, where tape is stored in ambient temperatures.  The result is that disk cannot be trusted to hold onto data for more than five years without suffering bit rot.  If you’re going to store data longer than five years on disk, you must move it around.  And remember that every time you move it around, you’re subject to the lower write integrity of disk.

I know that those who are proponents of disk-based systems will say that because it’s on disk you can scan it regularly.  People who say that obviously don’t know that you can do the same thing on tape.  Any modern tape drive supports the SCSI verify command that will compare the checksums of the data stored on tape with the actual data.  And modern tape libraries have now worked this into their system, automatically verifying tapes as they have time.

Only optical (i.e. non-magnetic) formats (e.g. BluRay, UDO) do a better job of holding onto data for decades.  Unfortunately they’re really expensive. Last I checked, UDO media was 75 times more expensive than tape.

Air Gap [Update: I added this a day after writing the inital post because I forgot to add it]

One thing tape can do that replicated disk systems cannot do is create a gap of air between the protected data and the final copy of its backup.  Give the final tape copy to Iron Mountain and you create a barrier to someone destroying that backup maliciously.  One bad thing about replicated backups is that a malicious sysadmin can delete the primary system, backup system, and replicated backup system with a well-written script.  That’s not possible with an air gap.

Device Obsolescence

People that don’t like tape also like to bring up device obsolescence.  They say things like “you can’t even get a device to read the tape you wrote 10 years ago.”  They’re wrong.  Even if you completely failed to plan, there is a huge market for older tape drives and you can find any tape drive used in the last 20-30 years on eBay if you have no other choice. (I know because I just did it.)

Second, if you’re keeping tapes from twenty-year-old tape drives, you should be keeping the drives.  Duh.  And if those drives aren’t working, there are companies that will repair them for you.  No problem, easy peasy.  Device obsolescence is a myth.

Device Life

Suppose you have a misbehaving disk from many years ago.  There are no disk repair companies.  There are only data recovery companies that charge astronomical amounts of money to recover data from that drive.

Now consider what you do if you had a malfunctioning tape, which is odd, because there’s not much to malfunction.  I have been able to “repair” all of the physically malfunctioning tapes I have ever experienced (which is only a few out of the hundreds of thousands of tapes I’ve handled).  The physical structure of a modern tape spool is not that difficult to understand, take apart, and reassemble.

Now consider what happens when your old tape drive malfunctions, which is much more likely.  You know what you do?  Use a different drive!  If you don’t have another drive, you can just send the one that’s malfunctioning to a repair shop that will cost you far less than what a data recovery company will cost you.  If you’re in a hurry, buy another one off eBay and have them rush it to you.  Better yet, always have a spare drive.

Legal Issues

This isn’t really a disk-vs-tape issue, but I just had to comment on the customer that Stephen quoted in his blog post as saying, “I’m legally required to store data for 30 years, but I’m not required by law or business to ever recover it. That data is perfect for tape.” That may be a statement that amuses someone who works for a disk company, but I find the statement to be both idiotic and irresponsible.  If one is required by law to store data for 30 years, then one is required by law to be able to retrieve that data when asked for it.  This could be a request from a government agency, or an electronic discovery request in a lawsuit.  If you are unable to retrieve that data when you were required to store it, you run afoul of that agency and will be fined or worse.  If you are unable to retrieve the data for an electronic discovery request in a lawsuit, you risk receiving an adverse inference instruction by the judge that will result in you losing the lawsuit.  So whoever said that has no idea what he/she is talking about.

Think I’m exaggerating?  Just ask Morgan Stanley, who up until the mid 00’s used their backups as archives.  The SEC asked them for a bunch of emails, and their inability to retrieve those emails resulted in a $15M fine.  They also had a little over 1400 backup tapes that they needed months of time to be able to pull emails off of to satisfy an electronic discovery request from a major lawsuit from Coleman Holdings in 2005.  (They needed this time because they stored the data via backup software, not archive software.)  The judge said “archive searches are quick and inexpensive. They do not cost ‘hundred of thousands of dollars’ or ‘take several months.'”  (He obviously had never tried to retrieve emails off of backup tapes.)  He issued an adverse inference instruction to the jury that said that this was a ploy by Morgan Stanley to hide emails, and that they should take that into consideration in the verdict.  They did, and Morgan Stanley lost the case and Coleman Holdings was given a $1.57B judgment.

Summary

Why isn’t tape dead?  Because there are plenty of things that it is better at than disk.  Yes, there are plenty of things that disk is better at than tape.  But move all of today’s production, backup, and archive data to disk?  Inconceivable!

Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at Sullivan Strickler, which helps companies manage their legacy data

19 comments
  • Curtis,
    Great article on data tape and I’d like to add that LTO data tape is ideal for video surveillance long-term retention… Another practice where the value of the video has importance (and business intelligence) when you need it, you just don’t always know when you are going to need it… We have a product that integrates disk AND tape for video surveillance needs… It’s not disk VERSUS tape… Also, LTO is widely used in the media and entertainment industry for long-term video storage as well… Point being, when you have the right need, disk AND data tape work so well together. Jay Bartlett/SoleraTec

  • Overall – well written and I thank you. However, I disagree with some of the points. To name a few,

    Device Obsolescence – nobody in their right mind wants to be in the business of hunting down boneyard equipment on Ebay. It’s a major data recovery risk which doesn’t align well at all with modern SLAs and RTOs and offers very little guarantee short of a data recovery service coming through with a large invoice.

    Air Gap – Easily achieved by array based snapshots then replicated offsite.

    Integrity of Initial Write – Is unfortunately offset by the inability to subsequently read the data using an aging tape, tape drive, or a tape drive at the recovery site which didn’t perform the write in the first place. I’ve seen far too many tape drive problems during DR plan testing than I care to remember. Unless tape technology has changed since I’ve used it last, a backup set which spans tapes is written as RAID0 – no ability to proceed with a restoration process using parity if one or more of the tapes is cooked.

    We’re friends and I know you’re passionate about backup and maybe even tape but I look at both of us on the same side of an ugly battle against alarming volumes of data which must be protected. It’s not so attrictive or cost effective on the disk space side either but at this point I’d choose disk poisoning over tape strangulation.

    Jas

  • Jason,

    I don’t think we’re in disagreement on device obsolescence. That’s why I said you SHOULD be keeping the drives if you’re keeping the tapes. But if you DON’T, there are a ton of drives out there and I don’t think it’s a risk, other than a risk of it taking longer while you acquire replacement drives.

    Re: Air Gap. Replicated disks do not have an air gap. If I can “reach” them electronically, that is not an air gap. A rogue admin could very easily delete your onsite snapshots and your replicated offsite snapshots in a two commands. He can’t touch offline media sitting in a vault, though, if you require two-person signin to pick up tapes.

    Re: Integrity of Initial Write. You seem to suggest that data on an “aging” tape is more at risk than data on an aging disk. Physics says otherwise. Data will stay in place on tape for decades, disk only for 5-10 years. I address that in the long-term longevity section. AND, there are “check that the data that you thought was on tape is on tape” systems that are now integrated into Quantum & Spectralogic libraries. You CAN proactively monitor older data to make sure it hasn’t suffered bit rot. If there is any bit rot, you can fix it using your other copy.

    “Disk poisoning vs tape strangulation” Nice turn of a phrase. If tape is any more strangling than disk, the system was designed wrong. And there are a lot of poorly designed systems that have poisoned people’s view of tape. For long-term storage purposes — to me — disk is far more strangling. It forces you to migrate data at least every five years (usually every three due to depreciation cycles and lease schedules), where tape does not. (Tapes are not depreciated or leased. The tape libraries are. When a library comes off of lease or is fully depreciated, you just move its tapes to the new library. Easy peasy. With disk I am forced to do an actual data movement that places my data at risk due to the lower bit error rates for disk.) It forces you to constantly power the unit where tape does not.

    You’re right. I’m passionate about helping people do the right thing. And keeping data with a .0000000001% chance of ever being read sitting on spinning disk is simply a HUGE waste of money for a TEEENY benefit if/when that file is needed.

  • There’s 1 thing though that tape can never do where disk can.

    At any point in time I can run a check to ensure the data on my disk is intact, restorable, and uncorrupted. And I can do it over and over and over again. Every day, week, or month if I so choose.

    I can’t do that with tape.

    And that’s the kind of warm and fuzzy that’s needed, especially in the legal/compliance world, to let IT managers sleep well at night.

    You get what you pay for. And when it comes to ensuring critical data can be restored, sometimes the price for certainty can be high.

  • Why does everyone think that because something is on tape, it is impossible to check its consistency?

    What consistency check do you think you can do on disk that you can’t do on tape? One check would be to restore a file and compare it to the original file. You can do that with both.

    Another check would be to store checksums for everything stored on media, and then to regularly check that those checksums match with what you see on the media. This is built into both Quantum and Spectralogic tape libraries, but I know of no disk system that has such a thing. Yes, you could write it yourself, but with tape it’s already there.

    Regarding your “every day, every week” comment, I’d say that it’s only possible to check a sample of your data every day/week, etc. You’d have to signficantly over-engineer a system to have enough horsepower to do the kind of check you’re talking about on all data every day. In addition, with a disk system, any significant amount of checking is going to effect the performance of the overall system, because everything is on RAID. AND this checking is going to be done using the host CPU, memory, and bus as well — effecting it’s performance.

    On the type of automated checking on tape I talked about above, you just give it one or two extra tape drives, and you can do all the checking you want without impacting the performance of any of the rest of the drives or the host CPU, RAM or bus, as all the work is happening inside the tape drive doing the checking.

  • I notice in your article that you begin a paragraph with, “Disk is the best thing for backups…”

    I disagree wholeheartedly. You even go on to describe disk’s shortcomings, especially for archives.

    We have an enormous tape library filled with LTOV tapes and I would never ever think to replace that with some RAID array. (Note, however, that it is *fronted* by a large RAID device. …The best of both worlds!)

    With tape, I get a new media for storing my data on. Or I overwrite it at my leisure. I am not comfortable knowing that my disk is backed by a disk, such that whatever I screw up on my primary drive propagates to my backup media.

    But moreso, everything is an engineering tradeoff- even disk. I want to back up to something cheap- as cheap as possible. This is an incredible advantage that makes a pat statement, “Disk is best” suspect. Once my media is really really cheap, I’m much more carefree about how much of it I use. Better profligate with my media than stingy with my backup techniques!

    Interesting article nonetheless and quite thought provoking.

  • Always nice to hear a dissenting opinion, especially when it’s worded respectfully.

    The problem with tape (in all but a few rare circumstances) for operational backup is the complete inability to match the speed that today’s tape devices want to run at and the speed that backups are able to run at. The former is measured in 100s of MB/s, and the latter is measured (usually) in single MB/s. For that reason, almost all backup and recovery design of the last 10 years has used disk as the primary target.

    There is still nothing wrong with creating an occasional copy of that to tape — which I think is what you’re arguing for — as long as it’s possible. (Many systems today are so large creating a full copy at all is technically infeasible.)

  • Hi Curtis,

    For those that want detailed analysis on TCO for Long-Term Storage see Clipper Group http://www.clipper.com/research/TCG2013009.pdf

    You should also consider a mention of the Oracle StorageTek drives and libs – according to IDC #1 in larger sites and used by the largest archives in the world. Why? Besides 2-orders higher reliability @ 1:10^19 and a unique CRC write/verify capability; the STK T10K is currently @ 5TB and 252MB/s (both native). Stay tuned for next gen as that model is about 3 yrs old. LTO is great and STK offers all LTOx drives (they actually fab most of the R/W heads), but some customers demand even lower TCO along with higher performance, capacity, & reliability.

    Attended one of your Truth In IT seminars and have been a follower since!

  • I did mention the Oracle and IBM drives in the article, actually.

    I think the ClipperGroup report is flawed, but it does have some interesting data in there.

  • One thing which is not often thought of, but which Curtis alluded to under “Device Life”, is that with tape, you are saving only the media which contains the data; but with drives, you are saving the media and the mechanical device which accesses the media.

    Consequently, if you are saving your data to tape, and if you are saving it for years, you don’t have to save a mechanical drive on the shelf for years, hoping that it will work if you ever need it.

  • Thank-you Curtis. I appreciate the effort it must have taken to craft a succinct as well as compelling summary.

  • Interesting article. I think it can justly be summed up by saying disks are better for backups and tapes are better for archives (with cost throwing in a little gray middle ground).

    I completely disagree with your assessment of that one user’s comment on the legal requirements to restore archival data. As you quoted it, the user is absolutely correct. That requirement does not exist unless (or until) the company comes under legal or civil scrutiny. As long as such scrutiny never occurs, there is no legal or business requirement to retrieve archived data. Period. There *is* still the legal requirement that archived data *be able* to be retrieved, as you described, but that’s different.

    It may be a small semantic difference, but I think the point the user was trying to make is that there’s a difference between backup and archive data. Backup data typically must be access frequently and recently after the backup is made. Archive data can typically be written and promptly forgotten for years (until a lawsuit comes along) – which makes it the perfect purpose for tapes and Iron Mountain.

  • I’d say your summary is spot on. I can think of no better place to put archive data than tape. There is optical media, but it is 75 times more expensive than tape and very slow. It is good at holding onto data, though. So if money and speed are no object. 😉

    As to your comment, I’m not sure if we’re in disagreement. I believe he was implying that since he didn’t have to ever read the data, then it is fine for tape — implying that if he DID need to read it, he shouldn’t put it on tape. I completely disagree with his implication. Whether or not his words are symantically correct or not is irrelevent. I was arguing with what he was saying between the lines. 😉

  • Nice to read that more people have these problems with what to choose which data you backup and how you archive data.
    We found that with a Blu-ray disk you can keep your data save and stored.
    These guys make an interesting software solution to use bluray/tape or disk for archiving.
    http://archive-disk.com/media.html

    You can store a Blu-ray up to 50 years now, this will decrease the cost over the years and make it cheaper until a better media comes around.

  • In Some Countries it’s part of the law to keep tapes for ten years, so looking worldwide from my point of view, The end of the tape is not even close!. In the other hand, disks are great for short data retentions, but for long term is to expensive at least in the Country where i live.

    Jp

  • Just a few comments:

    As a former EMC employee I can say that EMC does not manufacture a tape device that I am aware of; they do sell tape emulation on disk and perhaps resell or partner with Tape Vendors.

    The “Air Gap” feature of Tape can be met in a Disk Solution by WORM Technology, which meets the various governmental requirements.

    For long term retention copying from the older disk technology to a new generation of disk technology is a snap compared to doing the same with tape.

    That being said Tape does offer portability and cost advantages and is far from dead.

  • I’m not saying disk won’t work. I’m simply saying that so would tape, and any attempts to say that it’s unsuitable for the task are simply misguided.

    Yes, EMC currently partners with Spectra to provide tape products to their customers.

    The Air Gap feature can technically be met with disk and WORM, and that’s an interesting idea. Do you know if such a feature allows the space used by such a feature to be returned to the pool after the WORM period expires? Because otherwise that would get quite expensive.

    I think moving a significant amount of archive data from one device to another is never a “snap,” as the laws of physics still apply. 😉 But I will agree that if one is migrating from one disk array to another, one could use something like rsync or a similar tool, where with tape you are required to use your archive software. So there is a perception that it is easier. But a large tape migration is actually quite easy if you give the archive software access to both tape drives/tape libraries. You direct it to migrate data from A to B and then walk away. It’s simply a different process, but it’s only “hard” if you’re unfamiliar with it.

  • With EMC WORM the Expired capacity can be re-used. IBM supports a version of WORM Technology also.

    In my experience a Tape to Tape Migration is slower and much more labor intensive, especially with older Tape that is prone to read errors. Also the Tape to Tape Migrations that I am familiar with are Host Based, imposing an additional load on the Host.

    In the EMC Solution a version of the Tape on Disk Replication Software is used to migrate between the older Tape on Disk Technology and the new, it however requires you to stay with the EMC Technology; vendors love that.

    I am pragmatic and I don’t hate magnetic tape, it definitely has a place. If some slicked haired Salesman tells you must “Get off Tape” without knowing your requirements of processes, throw them out 😛

    What consumer product these days is magnetic tape based ? The last I owned was an answering machine and that was years ago. I listen to Led Zeppelin IV on my Ipod, not my 8 Track these days ! 🙄

  • Great article, I really appreciate that you shed some light on a bunch of common misconceptions about tape. It cant be said enough that backup isn’t archive – in my opinion, backup is for DR, archive is for data retrieval, and each should have their own service-level objectives defined.

    I’d like to add one aspect of tape infrastructure that wasn’t covered – people.

    In most, not all, but most cases, people need to manage tape. At some point, we’re either throwing money at expanding the footprint of a tape library (more likely with an archive, less likely with backups which *should* have a shorter retention period, or we’re paying someone to shuttle tapes in and out of a library.

    People are expensive and have a much-higher failure rate than either disk or tape 🙂

    That said, unless you’re building a really long-term archive, then managing a turnkey library, where you’re not moving media around, shouldn’t be difficult.

    I’m not sure that device obsolescence is an invalid concern. I get keeping gear around, but in 20 years, will we be able to connect a 4/8GB FC drive to whatever a FC SAN looks like then, or do I need to also store FC switching and for that matter, FC cables, as well? What will server infrastructure look like then, in terms of protocol and connectivity support? Best to keep some server hardware around, as well. What about keeping the application that has cataloged or indexed all of that media? I honestly don’t know the answer, but I’m wondering if there a limits the number of pieces of media that, say CV or NBU can keep in their respective databases. Will future versions of our backup applications be able to ingest 20-yo hardware and use it?

    I’m not saying disk is the answer, but organizations should consider long and hard about planning for the capability of migrating media every, I don’t know 5, 8, 10 years as an alternative to storing gear and HOPING that you thought of everything 20 years ago. SGI, for example now has FileTek, which automates that stuff. I think that as LTFS matures, there may be some improvement that make it more-attractive for long-term retention.

    OK, I’m done babbling. Nice article!