Are purpose-built VMware backup apps a good idea?

When I'm talking about backing up of virtual servers & desktops, I always get asked "what do you think about Veeam/vRanger/PHD Virtual/VMPro?  Should I buy that instead of NBU/TSM/CV/BE/AS/DP/etc's VADP agent?"

The first thing I can say is that it has definitely become a trend to install a purpose-built backup app for VMware/Hyper-V.  On one hand, it's hard to argue with success.  People who have moved to such products have often found their backups much easier than they were before.  On the other hand, since most of them are moving from the agent-in-the-guest approach, anything would be better than that.  Some of them are also moving from their attempt to use Very Crappy Backup (VCB).  That's another one that is not hard to compete with. 

These purpose-built products do have some really awesome features.  For one, they often work around the limitation that VMware creates by using the VSS_BT_COPY backup type.  They do something to make sure that the transaction logs in guests get truncated.  Some of them also have some really interesting features of being able to run a guest directly from a backup, which leads to all sorts of interesting recovery possibilities.

My concern is that many of these products are missing what I would consider to be core functionality for a centralized backup product.  When I look closely at these products, they tend to be missing one or all of the features listed below.  What's worse, when these shortcomings are pointed out, some of their representatives look at you like "why would you want that?" I've been doing nothing but backups for almost 20 years, and I think any decent backup product should have all of these features:

Support for more than one platform

Very few shops are 100% virtualized.  If you can't also backup the few physical machines their environment, you force them to also run some other product to back those up.  More backup apps means more confusion and more confusion with each failed backup and restore.  It's simply math.

Central management

It's very common to require more than one backup server to handle a given environment.  Do you have some sort of centralized management that allows you to see all of your backup servers and manage them all from one place? 

Copies

Backup functionality #1 is to support the copying of the data from the backup source to the destination, and then to another destination.  Many of these purpose-built backup tools are really good at the first step, but have absolutely nothing for the second step.  They tell you that you can run another backup to another destination, which is not good for a number of reasons.  Or they tell you that you can buy a 3rd-party dedupe product that can replicate your backups for you.  I'm sorry, this should be in your product.  You should not require me to buy other people's products to do what your product should do on its own.  (The lack of this feature is why so many people buy inexpensive products like Backup Exec to back up the datastores created by these products.)

Config backup

If you have a database that stores configuration and/or history information about your product, it should be built into your product to back up that database.  Period.  Telling someone they can run a cron job just doesn't cut it.  And if how/when to run that cron job isn't even in your documentation, shame on you.

Tape support

This is at the end of the list for a reason.  I'll admit that this is the least important on the list, but I do believe that a backup product should have the ability to copy to tape for long term storage purposes.  Not everyone can store their backup on disk as long as they need to.  Again, many people use Backup Exec to handle this, but I believe it should be built into any backup product.

My position above has made me very unpopular with some of the purpose-built folks — some have been very upset with me. That doesn't change the fact that the above features should be considered table stakes for any backup application.  You could argue that tape support isn't table stakes anymore, but I disagree.  That is, it is still table stakes if you want to be considered a full-fledged backup app.

Let's see how things go.

Continue reading

Don't be a Snail: Stop Changing Your Backup Software

Sometimes I walk to work in the mornings (I live just over 2 miles from the office), and if I leave early enough I walk down this particular sidewalk that has just been sprinkled with water.  On each side of this sidewalk is grass, and watered grass here in Southern California tends to mean that snails will be there. The snails take advantage of the wet, cool sidewalk (and the fact that the sun isn’t overhead), and they decide to cross from one bunch of grass to another bunch of grass. 

On any given morning there will be several hundred snails crossing from one side to another.  200 or so will be crossing from the left to the right, and 200 or so will be crossing from right to left.  You know, ’cause the grass on the other side is, well… you know.  The funniest one I saw was a snail that had made it 95% of the way from one side to the other, and then changed his mind and turned around.

I got to thinking about backup software (as one does), and all of the people I know that are moving from product A to product B.  Then a bunch of other people that are moving from product C to product A, while others are moving from product B to product C — and they all think that this will make their lives soooooo much better.

I’ve done hundreds of backup assessments over the years.  I can only think of one or two where the gist of the recommendation was, “Your backup software sucks.  You should change it.”

I can, however, think of many, many times where the problem was “you’re not streaming your tape drives,” or “you’re manually specifying an include list and you should use the auto-selection feature,” or “you’re making too many full backups,” or “you’re using the scheduler in a way that it wasn’t designed to work,” and on and on and on.

Changing backup products is one of the riskiest things you can do to your backup environment.  The learning curve of the new backup product is almost definitely going to reduce your recoverability for a significant period of time.

What would be much better is to bring in an expert in that product for a few weeks and have him/her tell you how best to use the product you already have.  The learning curve is much easier, the cost is much lower, and the period of instability will be much shorter.

Don’t be a snail.  Learn what the grass on your side of the sidewalk really tastes like before you start crossing the sidewalk.  Remember that some snails die along the way.

 

Continue reading

IT is not in charge!

 I was helping a guy on a plane understand what "the cloud" is.  Once I did that, we begun a discussion on trust.  I shared with him my opinion that we have been trusting other vendors since we started IT.  We trust every hardware and software we have not to put backdoor stuff in our hardware or software that is designed to do things we don't know about. We trust technicians to know enough not to use bad passwords. (Of course, sometimes we're wrong.)  I don't see trusting a cloud vendor as being so terribly different.

I'm sure a bunch of you will focus on that first paragraph, and not on what this blog post is actually about. But here goes anyway.

Eventually we got to the part of the discussion where he mentioned that "our IT department would never allow that."  He explained how he has to carry three laptops (personal, corporate 1 and corporate 2) whenever he travels and how he has to dial four digits on his phone before he makes any calls.  I'm guessing that we just hit the tip of the iceberg of how his IT department is soooo security concsious that they have forgotten their primary purpose — to enable people to do work.  (BTW, this guy wasn't working on missile launch codes or anything.  I forgot what he does for a living but I remember wondering was security was that important for this particular company.)

I ranted a little bit about that to him, to which he replied, "well, they are in charge."  I asked who he meant, and he said, "IT."

I just about lost it.

If you are in IT and you think you are in charge, you are wrong.  The only thing you are in charge of is helping people get their job done.  We buy decent laptops & desktops, so they'll stay up and people can get their job done.  We make backups so when things go wrong, we can get people their work back, and let them get their job done.  The only reason we do security things is to keep our company from losing the efforts of the people that work there.

Sometimes IT people forget that we are there to serve the business.  If you enact a security policy that's so rigid that it slows down people's work, you forgot your job. If you turn on a backup system that slows down the servers, and by association the work of the people, you forgot your job. 

You are not in charge.  The business is.  I feel better now.

 

Continue reading

FCC is a LITTLE out of date WRT its backup designs

The FCC gives discounts to schools and libraries if they want to buy a tape-based backup system, but not if they want to use disk or any type of cloud-based architecture.

No, this is not me saying this is an example of how tape is better.  It’s me, an American citizen expressing frustration at how inefficient my government is — at least in this case.

For those (like me) who don’t live in this world, here’s what I’m talking about.  According to their website, “The Schools and Libraries Program of the Universal Service Fund, commonly known as “E-Rate,” is administered by the Universal Service Administrative Company (USAC) under the direction of the Federal Communications Commission (FCC), and provides discounts to assist most schools and libraries in the United States to obtain affordable telecommunications and Internet access.”

If you download the list of things that are eligible for the E-rate program, you will find that “tape backup” is eligible, but Online Backup Solutions are specifically not eligible.  There is no mention of disk-based backup devices.  Here’s the best part.  Tape backup is defined as “QIC, DAT, 8mm, DLT, AIT, and ADR.” ADR was end-of-lifed 9 years ago, QIC & AIT were EOLd 3 years ago.  Note that their is no mention of LTO, a device that was released 12 years ago and currently owns 90% of the market.  So to say that the FCC is a bit behind the times is an overstatement.

By the way, they also list floppy disks and CD-Rs as the only examples of removable storage.  No mention of DVDs or BluRays — and when was the last time you saw a floppy drive?

Thanks to Christina Weil (@c_weil) for pointing this out via Twitter.

Amazing, just amazing.

Update (3/22): This program is aimed at getting schools and libraries connected.  So I’ve been told that the network parts of this document are mostly up to date.  The only reason backup is in the document in the first place is to help ensure that the connectivity systems remain available and connected.  What I think happened is that the network vendors knew about this program and made sure their parts got updated, and the tape/storage folks have ignored it (or not know about it).

Continue reading

More misinformation about backups

I don’t care if you use disk, tape, or the cloud to back up your systems.  (In case you think I’m swayed by advertising, I have advertisers from all of those categories.)

Having said that, it bothers me when I see misinformation being used to sway you one way or the other.  This is why I wrote this article that disproved the Gartner 71% tape failure “quote,” and this article disproving the Yankee Group 42% failure “quote.”  And since he used my comment system to link to his article, I also thought I’d write this blog article dispelling the misinformation in the article he linked to.

He said it’s been a long time since people have seen tape used for backups. 

The live survey of the hundreds of attendees to last year’s Backup Central Live shows showed that 82% of them still use tape as their final destination for backups.  So much for not seeing tape in a while.

He said IT pros are still skeptical that removable drives have a legitimate place in backup

Yes, we are.  I think that 3.5″ removable disk drives are a very bad place for backups. 2.5″ drives yes, 3.5″ drives, not so much.  They’re simply not designed for excessive portability.  Adding to that is this fact:  every portable hard drive I have ever used for backing up my laptop has died long before the drive it was backing up.  Every single one.

He said cloud backup is shiny and new and that’s why people are choosing it.

No, it’s because it’s a complete and total outsourcing of backup functionality.  Backups can be onsite and offsite without ever touching a disk drive or tape drive.  AND you will be constantly notified if your backups are working or not working.  You often even get notified even if you shut off all your backups!  That’s not the case with any backup software product that I’ve ever used.  There are a lot of reasons to use cloud backup over removable disk drives or tape.  In fact, there are so many that I strongly recommend cloud backup for small to medium sized companies.

He said disk is cheaper for small companies

Yes, it is.  It is cheaper to acquire the drives as long as you never need to add capacity.  If you do, however, need to add capacity, disk costs will double.  Tape costs will not.  It’ll cost you about $.02/GB to add more capacity to a tape-based system.  (Having said that, I do not recommend backing up directly to tape; I haven’t in a while.)  Having said that, I priced a slightly different tape-based system than the one he quoted in his article, and it was approximately the same cost.

As to comparing their 10-bay disk systems against an autoloader, I don’t see how you can do that.  Backup software products simply don’t know what to do with 10 removable disk drives, but they do not what to do with autoloaders.  (I’m sure he knows a backup software product that will work his configuration, but I don’t know of one.)

He said disk is more reliable than tape

Baloney.  I’ve written about this before.  Tape has a much higher reliability rate than SATA disk — one hundred times more reliable.  I’ve already shown above that the statistics he quotes in his article are bunk.  Almost every failure I’ve ever seen with restores was the fault of anything but the media that was being used.  (I still don’t think tape should be used as the initial target for backups, but it is a very reliable place to put the second copy.)

He said LTO-5 speed is 140 MB/s, but only with compression

Sorry, Charlie.  That’s the native speed.  It’s up to 280 MB/s with compression.  With the 1.5:1 I see all the time, it’s 210 MB/s easy.  Having said that, it’s very hard to feed data to a drive that fast.  This is why I don’t recommend tape as the initial target for backups.  But if you’ve already made a copy on disk, you should have no problem streaming that tape drive.

He said single file restores are faster from disk

Yes, they are.  That extra minute it takes the tape to load and get to the single file would probably put most companies out of business.  Seriously, you’re going to make a case out of a minute of tape loading time?

He said upgrading/replacing is cheaper with disk

Are you kidding me?  I addressed this already.  If you need more capacity than the initial purchase, it’s much cheaper to expand a tape system. Most customers go years without upgrading their drives.

He said doing synthetic fulls is easier on disk

Yes, it is.  And CDP and CDP-like tech is also only possible on disk.  Disk has a lot of things going for it.

He said tape has to be replaced more often

Again, baloney.  A tape that is used once a week will last four years with the chart quoted in the article.  That is longer than most disks I’ve used.

Summary

I don’t recommend using tape as the initial target for backups, but I still think it’s a great place to put the next copy.  And if you are a smaller company, I think the best thing you can do is to use a cloud backup service that totally automates everything, and alllows for a local copy of your data.  But I think that any backup system that requires small companies to manually swap removable media just to make backups happen is a bad idea.

Continue reading

Yankee Group never said 42% of tape restores fail

I just wrote a blog post about how Gartner never said that 71% of tape restores fail.  They never said anything like it.  Another statistic that is often quoted is "The Yankee Group said that 54% of tape restores fail."  Guess what?  They never said that, either.

What they did say in a 2004 paper is that 40.7% of 362 IT executives believed that they had suffered at least one restore failure in the previous year due to tape unreliability.  That's not even close to saying htat 42% of all tape restores fail, but who need truth, right?

Also, I'd like to throw out that these were IT executives.  What this stat really means is that 40.7% of them were told that they had restores that failed in the previous years due to bad tapes.  That's not quite the same thing as it actually happening.  How many backup people even know that the reason for their failure is their own misconfiguration?  And if they did, how many of them would admit that to their boss, rather than saying "the dang tape failed again."

Now the only statistic left is the Strategic Research one, but I can't find anything on that one.

It appears that at least 66% of all tape statistics are made up. 😉

Continue reading

Gartner never said 71% of tape restores fail

How many times have you read that Gartner said 71% of tape restores fail?  Google it.  You’ll find dozens of references to this Gartner “statistic.”  It was cited again recently in an article by Highly Reliable Systems, along with a bunch of other stats about how tape sucks. I saw Dave Russell of Gartner last week and asked him about this statistic.  He said he had never heard it, but that he would look into it.  It turns out that the only way he could find it was to Google it.  He searched Gartner’s entire archive and could find no paper that ever suggested at 71% failure rate for tape restores.

He said, “I am somewhere between annoyed and pretty darn angry about what I believe are continued misquotes re. Gartner and tape failure rates.  I’ve been the lead analyst for backup and recovery technologies since 2005, and none of what’s out there have been published during my watch.”  The only report that referenced tape and the number 71% was a report David did in March of 2006.  Here is what it said:

New, and less-expensive, disk options make the use of disk for faster recovery a more viable option than backup to tape. In a poll of 252 attendees at the 2005 Gartner PlanetStorage conference, 26 percent reported that half or more of their recoveries were currently done from disk. That number jumped to 62 percent when the time frame was extended to 2007. As they look five years into the future to 2010, 71 percent expect that tape will be used mostly for archiving and disaster recovery.

I did a bunch of web searches for “Gartner 71% tape restores fail,” and found that if I search for those words prior to March of 2006, I don’t find much.  I do find an article from Jon Toigo in 2005 that says he hears IT people quoting a 10% failure rate from Gartner, but he believes that number is fictitous (which it probably was.)  I also find a whitepaper from Exabyte that refers to a 2002 article from Adam Couture of Gartner Group.  I just asked David Russell to see if he can find that article.  I also found another whitepaper from Tandberg citing similar numbers and the same paper.  Maybe that one has some basis in reality.  Most interestingly, I did find this page which claims to be the text of a Feb 2003 article from Computer Technology Review that says that “A recent study [it doesn’t cite the study] found that while tape backups are used extensively, restoring data from a tape backup system fails an astounding 70 percent of the time. The reasons for such an alarming rate of failure range significantly–and may vary from bad tapes or tape drives to the inability to find the backup tapes or careless processing by IT staff.”  (My experience has been it’s been far more careless processing by IT staff than bad tapes.)

The important thing is that prior to March of 2006, a Google search shows no references to Gartner thinking that 71% of tape restores fail.  Then David Russell wrote his report in March of 2006 that said that “71 percent expect that tape will be used mostly for archiving and disaster recovery.”  If you change your Google search to the year after his paper came out, you find a bunch of quotes to the 71%, the first of which comes from this DPM Datasheet from Microsoft — promoting DPM.  Then all of the sudden, the floodgates are open and everyone is quoting this number — no one (including Microsoft) actually giving their source, other than simply saying “Gartner said it.”  Most of them also seem to quote the Yankee Group (saying 42%) and Strategic Research (54%).  I wonder if they ever said what these articles say they said. 

Another quote I’ve seen is this: “according to Ben Matheson, group product manager for Microsoft’s “Data Protection Manager” Division,  42% of attempted recoveries from tape backups in the past year have failed.”  (BTW, please note that this is the same number as the Yankee Group number above, so maybe he was just quoting that number) I saw this in an article updated last week. According to LinkedIn, Ben Matheson hasn’t worked for Microsoft since February of 2006, so that quote can’t be correct either.  But once you’ve got a great quote, why let it go?  Wait, I may have found our Gartner quote culprit. Let’s see, Ben Matheson leaves Microsoft as its DPM product manager in February of 2006.  The new person took over shortly thereafter.  The next month a Gartner paper is written, and within two months we have the Microsoft DPM product group citing it incorrectly.  Could it have been a new gung-ho product manager misquoting Gartner?  Then everyone else starts quoting Gartner by quoting this Microsoft paper.  Next thing you know it, it’s real!  (This is just conjecture, of course. Don’t sue me, person who took over from Ben Matheson.)

So what?

We all know tape backups and restores, fail, right?  Who cares if no one at Gartner said it?  The first reason is truth.  This statistic is cited so often that it has been accepted as truth, and it isn’t.

The second reason is that you can’t debate the truth of a fake report. If it was a real report, we could check the stats behind the stat, and see how many of these “tape restore failures” were caused by human error and had nothing to do with the fact that they were using tape.   But since there never was any report, we can’t do such a thing.

Please, people.  Don’t quote third parties like that if you can’t cite the source.  It’s too easy to misquote.

Continue reading

Backup Central Live Q1 starts next week!

We've got new and exciting content for 2012, and we're starting our seminars this year with San Jose and San Diego next week.   These free seminars are first-come first serve (end-users only), and we're almost at capacity in the first two cities, so you'd better act now if you want to go.  I've also listed the rest of our backup seminars for Q1.  (Other cities will be announced soon.)

Date Event Where
Jan 24 Backup Central Live! San Jose, CA
Jan 26 Backup Central Live! San Diego, CA
Feb 7 Backup Central Live! Raleigh, NC
Feb 21 Backup Central Live! Miami, FL
Feb 23 Backup Central Live! Tampa, FL

See you there!

Continue reading

Stop SOPA/PIPA

Before I say anything about SOPA, let me say that I am not battling SOPA because I’m into illegally downloading books/music/movies.  As a provider of content (author of three books), I am strongly FOR paying for the media I use.  And don’t give me that crap about it isn’t stealing cause you were never going to buy it anyway.  You didn’t have something, now you have it, and you neither paid for it nor obtained the permission of the person who provided it.  You call it what you want; I call it stealing.

Now that THAT is out of the way….

What I’m also very much against is the government wasting time and MY money trying to stop something they will never stop.  I’m against SOPA for many of the reasons I’m against the TSA.  The TSA is security theater; SOPA is anti-piracy theater.  The only thing it will accomplish besides wasted money is some government folks getting to say that they did what they could — all while wasting millions of taxpayer dollars and the time of other companies’ IT departments.

Chime in!  Especially over at my new domain stopsopacentral.com. 😉

 

Continue reading

Contemplating File Sync/Sharing Services

I wrote a few months ago about what a difference the cloud has made for how I conduct business.  I rarely buy software for my new company anymore; I often am paying for some type of cloud-delivered service.

One of those services that I use (and love) is Dropbox.  It is an incredibly easy replacement for a file server when you need to share 10s to 100s of GB of files between mutliple users.  However, I definitely have some security concerns about it, and not just since the big snafu a few months ago.

One of my issues with dropbox is that they can access my data.  Data is encrypted in transit, but they can access my data because they have my password.  The same appears to be true of Syncplicity & Sugarsync.  Why do I think that? Because they have a "reset my password" link.  How does encryption work if they can change my password without a problem?  Compare this, for example, to wuala's answer and boxcryptor's answer to the question about a lost password.

Even with Wuala, who says they don't know my password, how do they share encrypted data with users I specify?  If all data is encrypted/decrypted locally, how does the person with whom I'm sharing files decrypt them?  I'm curious.

The last two listed are open source alternatives.  They're too limited in functionality for me, but I thought I'd throw them on there anyway.

SugarSync

Wuala

Syncplicity

Boxcryptor

iFolder

Sparkleshare

What do you think about all this?  Anyone I left out that I shouldn't have?

Continue reading