Parity RAID is dying – Long live CDP & Near-CDP

I keep reading all kinds of articles and tweets like this one from Robin Harris that predict or announce the death of parity-based RAID.  (Not everyone agrees, of course.)  The ultimate worry is that a second disk would fail while a RAID 5 group is rebuilding, or a third disk would die while a RAID 6 group is rebuilding, causing data loss.  The bigger disk drives get, the long rebuilds take, and the bigger the chance that this would happen. This paper published at Usenix geeks out as much as you could possibly want on the subject.

At the same time all of that is happening, we keep making bigger and bigger datastores that store everything from Oracle/SQL/DB2 to hundreds of VMs stored on a single large volume. The company that had a mission-critical 300 TB database comes to mind.

All I’m saying is that when we’re talking dozens to hundreds of terabytes of data in one place, at some point traditional backup is not going to cut it.  I don’t care how fast your tape drive or favorite disk target is (deduped or not), at some point a restore of any kind is just not going to meet your RTO.

Here’s my next thought.  When we think about double or triple disk failures on a critical storage array, that sounds really bad.  But what if — in the extremely unlikely event this really bad thing happens to you — you just flip a switch and you’re now running operations from your backup system, that has a very recently updated copy of your data?  If it’s a true CDP system, you might not lose any data at all.  If it’s a near-CDP system, you might lose a few minutes or an hour.  It’s all about what you’re willing to pay for.

In summary:

  1. If you’re not thinking about CDP and near-CDP solutions, you should be.
  2. Does the idea of a CDP or near-CDP system take away a little bit of the sting of the fear of the death of RAID?

Continue reading

----- Signature and Disclaimer -----

For those of you unfamiliar with my work, I've specialized in backup & recovery for 25 years. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Architect at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

Symantec Fooled Me (Under Research)

In my post that compared the performance of various target deduplication vendors, I listed the Symantec NetBackup 5000 as having 7166 MB/s of throughput.  That was based on their data sheet (as were all numbers in that post).  What I later found out via multiple sources including this discussion that this number requires a NetBackup Media Server with the media server dedupe option enabledPureDisk clientThat is source dedupe, not target dedupe.  I’m not saying source dedupe is not valid, but I am saying that performance is measured very differently in that space and it doesn’t make sense to compare the two in the same table.  Even if I was to do that, I would have to add all the other source dedupe vendors.

Since Symantec chose not to publish target dedupe numbers for regular NetBackup backups, I have put an N/A in the performance column in my previous post.

Shame on you, Symantec for not having this caveat listed in your brochure.  You won’t fool me again.

Update (11/17): Symantec has clarified what was in that blog post (and posted a thread into the comment thread of that post) to say that this was media server dedupe, not client dedupe.  So I have crossed out what I originally said and updated the blog entry accordingly.  I’m going to republish the number, as it is equivalent to what happens with Data Domain’s Boost, where the data is deduped on the media server before getting to the appliance.  But I still think that they should publish the throughput number without dedupe.

Continue reading

----- Signature and Disclaimer -----

For those of you unfamiliar with my work, I've specialized in backup & recovery for 25 years. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Architect at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.

People who like thin provisioning are not too stupid to administer storage

In his Executive Keynote at Storage Decisions yesterday, Jon Toigo likened thin provisioning to the “Lane Assist” features of some new cars (the feature that warns you if you have drifted from your lane).  He talked about a commercial that he had seen where people said things like, “I didn’t even realize that I had drifted into another lane.”   “You’re too stupid to drive!” yelled Toigo.   He then said that storage administrators who felt they needed thin provisioning were too stupid to administer storage.  If they simply knew how to administer storage, they would not need the feature. He also stated that other popular features such as deduplication, smart storage arrays, and server virtualization have no value either.  You can hear him say many of these things yourself by listening to this Infosmack Podcast that he recorded in September.

Update (11/12): Jon said in his post and subsequent comments here that he wasn’t dissing thin provisioning per se, just thin provisioning on an array.  Well it sure seemed like he was dissing it and people that use it, but in his powerpoint you can see that he was using it as an example of the types of features that smart storage arrays build in to increase value and vendor lock-in (his words).  Having said that, I know that some people do think thin provisioning is a crock, so I think the post still has value.

Anyone who is familiar with Jon Toigo knows that he’s a contrarian; he has a “bad boy act” in his words.  It’s not my style, but it seems to work for him.  I admire that he doesn’t seem to care what anyone thinks about what he says. (At the closing keynote of a show designed primarily for lead generation, I would summarize what he said as follows: don’t buy any of the products from any of the vendors, and if you did buy one of their products, get your support from a 3rd party.  If that’s not a contrarian, I don’t know what is.) I do wonder if his behavior places him on the sidelines of the game.  How far are you going to open the kimono if you know that all he’s going to do is laugh at what he sees and talk about it on stage?  But that’s not the point of my blog post today.  I simply want to express an opposite point to one of the things he said on stage.

I do not think that people who like thin provisioning are too stupid to administer storage.

I believe strongly in buying today only what you need today, and buying tomorrow what you need tomorrow, and I believe that thin provisioning allows me to do that.  Not having thin provisioning requires you to buy tomorrow’s storage today, or to spend much more time administering it.

Yes, having thin provisioning means that I need to monitor real utilization.  I need to monitor how much additional real storage is being used every day, and I need to make sure that I buy extra capacity before I hit the wall. You will also need enough spare capacity to allow for the length of time it takes you to buy additional storage, and to deal with occasional spikes in demand that you cannot predict.  Toigo mentioned that last point as an example of why thin provisioning makes no sense, because it forces you to keep extra storage in reserve.

Hogwash.  What storage administrator doesn’t keep additional storage in reserve?  Jon would say this is wasted storage.  Alright, then, let’s talk about wasted storage.  Let’s say a DBA has a new 200 GB database and they need a new LUN for it.  How big of a LUN does he ask for?  200 GB?  I don’t think so.  He asks for a terabyte or more because he knows what a pain the storage provisioning process is and he only wants to do it once in a while.  This is one example why the average real utilization in enterprise environments is nothing short of pathetic — unless they’re using thin provisioning.  You have a 200 GB database?  Great.  How big do you think it will ever get?  I mean if your wildest dreams for this application come true, multiple them times 10 and give me a number.  10 TB?  Click.  There’s your 10 TB LUN.  Never mind the fact I don’t even have 100 TB of disk.  You’ll get the amount of disk today that you actually use, and you’ll get tomorrow the amount of disk you actually use — and you’ll never have to ask me for storage for that application ever again.

Yeah, that’s really stupid.  Why would you want to do things like that when you could waste storage every time every application needs new storage, and you could have your application and database administrators ask for additional storage every time an application grows.  All that human interaction will be a collosal waste of time, but maybe the DBAs and SAs will get to know each other better.

Toigo also likes to bring up the issue of merging companies and customer databases.  What happens when you do mass imports of data from outside sources?  Wouldn’t that cause your thin provisioned LUN to smash right through the wall of available blocks?  Yes. So don’t do that without planning for it. Maybe you should have some change control and talk about any large new applications or imports before you do such a thing. What responsible administrator doesn’t have such a process?

Maybe not all applications are appropriate for thin provisioning.  Not all apps are appropriate for server virtualization, either.  That doesn’t mean it’s not a good idea.

When I think about thin provisioning, I think about my Drobo. Some might say that it’s a bad example because it’s a very simplistic style of thin provisioning, but it is thin provisioning nonetheless.

A year or so ago, I decided to rip 450+ DVDs to disk, and decided to put them on a Drobo, which is an inexpensive RAID unit designed for home or SMB use, and it supports thin provisioning.  When I did that, I bought four 1.5 TB disks and put them in the Drobo, and it automatically turned them into a 16 TB thin-provisioned HPFS+ filesystem on my Mac.  Obviously I didn’t have 16 TB of storage, but I only needed about 3.5 TB of storage.  Four 1.5 TB disks got turned into 4.5 TB of available space with parity protection.  The Drobo’s display lights and the desktop icon would tell me when I was running out of actual capacity.

Last week I noticed that the top light is yellow. I only have about 350 GB of additional capacity left.  Guess what?  Western Digital just came out with 3 TB drives.  As soon as they show up at Fry’s, I will pull out one of the 1.5 TB disks and replace it with a 3 TB disk.  All four lights will flash red until that drive has been rebuilt, then turn all green.  Then I’ll replace another 1.5 TB drive with a second 3 TB drive.  Once I do that, the Drobo will (behind the scenes) take the 3 TB of additional capacity I just gave it and magically add it to my usable capacity in my 16 TB thin-provisioned LUN.

Then I’ll do this all over again when I use up that 1.5 TB, at which point I’ll probably be replacing two 1.5 TB drives with two 3.5 or 4 TB drives.  I bought yesterday what I needed then, and I’ll buy tomorrow what I need then — all thanks to thin provisioning.

I guess I’m too stupid to manage my own storage.

Continue reading

----- Signature and Disclaimer -----

For those of you unfamiliar with my work, I've specialized in backup & recovery for 25 years. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technical Architect at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.