Veeam excites and frustrates me

Veeam is one of the most innovative backup and recovery tools designed specifically for VMware and Hyper-V.  They've also done a really good job of marketing this tool.  In a matter of a couple of years, they've gone from "who's Veeam?" to the mindshare leader in this space.  I'm not sure what they're actual market share is, and there are several other tools that are also making a name for themselves, but it's hard to think of a product that has more successfully captured the hearts and minds of their target market than Veeam.

They announced their vPower functionality at Tech Field Day in Seattle quite some time ago.  To summarize, this is the ability to run a VM from their backup image of that VM.  This opens up all sorts of different levels of functionality, such as instant VM recovery and automated, full testing of the viability of your backups of a given VM.

This is why I looked forward to their presentation at Tech Field Day 7.  At first, I was not disappointed. They announced support for Hyper-V.  Yay!  They also announced further refinement of their vPower functionality.  (They even gave me credit in one of the Powerpoint slides for some suggestion I made that they acted on.) They also hinted at a new version that is almost out, but wouldnt' really talk about it or show it.  We definitely were not allowed to ask questions about it.  Note to future Tech Field Day presenters: I can't think of a way to frustrate bloggers more than to tell them about a new version that you're not going to talk about, show us, or let us ask questions about.  To make that matter worse, they kept hinting about the new version throughout the presentation, but then kept telling us we couldn't ask about it.

Where the wheels fell off the truck for me was when I brought up the fact that most Veeam customers use Backup Exec to back up Veeam.  Another way to say that is that Veeam can't back itself up.  This resulted in a 20 minute conversation during which I got quite riled up, while Doug Hazelmen kept looking at me like he had no idea why I had such an issue with this.  You can watch the whole conversation here.  It's from 1:24 to 1:45.  He occasionally snickered, as if to say that the whole point of the discussion was ludicrious.  At one point he actually said the statement that they can't back themselves up was "stupid."  Yet he confirmed that the most common practice for Veeam customers was to use Backup Exec to back up Veeam.

Veeam data is stored in two places: the SQL database and the backup jobs directory.  There is no way within the product to make a special backup of the SQL catalog so that it can be easily restored without creating a catch-22 situation.  For example, one suggestion was to use one Veeam server to backup another Veeam server.  That creates a catch-22 of having to restore one server before you can restore the other server.  What if both servers are gone?   Doug hinted that losing the SQL database just isn't that big of a deal because it's just job configuration information.  You could just redo it if you lost it.  Is this really a backup company talking to me?

The second part of their data is the backup jobs history.  It has no catalog; everything that Veeam needs to know about the backups is stored with the backups.  The question is: what happens if one or more of those files gets corrupted?  What happens if some well-meaning admin looking for space deletes some jobs?  What happens if a rogue administrator deletes all of them?  As far as I could tell, Veeam has no way of recovering from this situation — which is why most Veeam customers use Backup Exec to back up Veeam.

Doug seemed to think that I was pushing for tape support.  In a way, I was.  Tape is still the least expensive way to get data offsite.  In many organizations, it's the only way to get data offsite.  They just have too much data to be able to afford a pipe big enough to replicate their backups — even if they have been deduplicated.  That issue aside, I wasn't pushing so much for tape as I was a method for creating a backup of my backup.  Files stored in filesystems get corrupted.  It just happened to me today.  For no apparent reason, a file whose modification time hadn't changed was telling me that it couldn't be copied.  It was a movie file on an iMac.  I can play the movie, but I can't copy the file.  Weird. That's what files on filesystems do — and that's why we back them up.  But the guys at Veeam just don't seem to get this, and that's why they frustrate me.

On one hand, I think the idea of a backup that can test itself in a totally automated fashion is completely awesome, and a lot of other areas of functionality are very impressive as well.  On the other hand, them not understanding the issue I do have (and therefore not addressing it) is really frustrating.  I hope we can work this out eventually, but they'll first have to stop calling what I'm saying "stupid." 😉


Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at Sullivan Strickler, which helps companies manage their legacy data

15 comments
  • Hi Curtis, I am actually with you here. What you are saying is definitely not “stupid” (I think Doug did not really mean to say that). The points you are bringing above are valid, and I am in full agreement with you.

    The only small correction I have is that Veeam backups are in fact restorable without another Veeam server. We have always provided tiny standalone tool for exactly that purpose (extract.exe) with the product. So, you can have Veeam server backup itself (assuming it is a VM), and then quickly restore the whole server should the need arise.

    Agree on tape support.

  • Hi,

    I agree with your statement that their response wasn’t a good one, but I disagree about the product design lacking. Veeam Backup is a backup to disk product, and never pretends to be otherwise.
    Adding tape to it would dilute its simplicity, and given how difficult it is to manage tape (when mature products have issues recovering and alerting when there are tape errors), I would expect that they wouldn’t do it well out of the gate.

    Most sites have some number of physical machines, and a backup to tape product. Use that product to augment Veeam Backup–do a weekly DR offsite of the Veeam Repository and backup a SQL dump of the (tiny) database.
    If you really want immediate offsite, use the Veeam Replication part of Backup and Replication.

    If I was managing the product, I would do my best to keep my eye on where I could provide the most value. And backing up to tape isn’t something that they’ll do better than another product.

    What I would tell Veeam to do is to better document how to protect the system. They are lacking in DR info for the Veeam Backup server.

    -Dave Gold
    CCS

  • @Dave

    It’s not so much about tape, although there is a need for tape for long term retention of backups. What I’m pushing for is some way to protect the datastore from logical corruption, and replication doesn’t provide that. If you replicate and you have logical corruption, the logical corruption is replicated.

  • Hi,

    Veeam doesn’t replicate data from the backup repository, it requires a separate job that reads from the source datastore and replicates that. In other words, there is no backup to disk and replicate the backup image with this product.

    Veeam is also one of the few products that does check its backup images. The first option is an auto verify at the end of the backup, and the second is to use the Surebackup feature.

    I understand the concern about logical corruption, but so long as a suitable array with background verification is in use, I don’t see a major problem here. (I’m not talking about unprotected storage, like the hard drive in my laptop)

    In terms of the SQL database getting corrupt, I would use standard SQL tools to maintain it. dbcc and a dump for SQL Express, or the gui based maintenance tools for the standard versions of SQL.
    Veeam should definitely update their documentation regarding doing that maintenance, and clarifying how to plan for DR. But given how stable SQL is, it is not the most pressing concern, IMHO.

    –Dave Gold
    CCS

  • I agree with you Curtis. You’d think they would feel a bit embarrassed telling you to use another backup product to back up Veeam.

  • Would you mind mentioning the “other tools”? I have the same concerns that you do. I have to protect 14TB of data and do not have the resources or bandwidth for offsite hosting. I still need to use tape to move data offsite for the time being. Veeam recommended I use BE for that also (I’m currently testing Veeam). Having to use two products irks me. You might also want to mention that Veeam will not backup stand-alone ESXi boxes using the free license from VMware (this is a VMware constraint apparently). My solution was the use they keys that I use in vCenter for my licensed hosts on my stand-alone hosts. I still do not connect these hosts to vCenter but Veeam sees a valid VMware license and it will backup the VMs on the standalone host.

    I left BE 3 years ago and went to CommVault. It has been a nightmare. No one seems to be able to work on it outside of NJ. Every time I try and find a local VAR who can support it I end up having them recommend a $12K vacation from NJ to our location. It is nuts. It is stable software but impossible to operate without thousands of dollars in training. I’m done with it as soon as I can find the right solution. I’m tired of the punitive licensing model.

    I’ve been testing Veeam and it is amazingly fast and easy to setup and use. I had my first backup configured in 15 minutes. And it worked the first time without error, and consistently until the trial license expired. It is amazingly useful at moving VMs between hosts if you don’t have storage vMotion and being able to preserve thin provisioning.

    I share your points of contention and I think they would have an unbeatable product once those are addressed. Always enjoy your presentations Curtis.

  • @Abdul, your blog post contains so many incorrect, misleading and simply false statements about Veeam, that I don’t know where to start. Although, what else to expect from Symantec marketing guy (something you avoided disclosing in your comment here).

    My only suggestion to you would be not to blog about something you have no idea about, and did not try hands on. Your “sources” of Veeam knowledge are clearly not trustworthy.

    Nevertheless, it’s nice to see Symantec being so obsessed with Veeam considering how young we are. Because we are only just starting, really. Watch for our version 6.0 in the coming weeks!

  • @Anton,

    Let us stay professional here, shall we?

    1. My apologies for not explicitly disclosing that I am from Symantec. I know Curtis in person, moreover the link I shared has an ‘About Me’ tab where I have disclosed this. In any case, it is a good practice to disclose my employer info for the broader audience. I do this all the time. It was quite late and I simply forgot. Thank for that reminder. I appreciate it. By the way, it looks like you had forgotten to add the disclaimer too.

    2. That blog is not sponsored by Symantec. The comments there are my own.

    3. Instead of accusing me of making incorrect, misleading and false statements; I would appreciate you professionally pin-pointing what those statements are. A blind allegation does not lead to conclusions (and any corrections if applicable). You can comment on my blog or you can comment here. Are we on the same page here?

    4. My intention is not to call your baby ugly. I understand the emotional aspect. I am an engineer who is proud to market what I believe in. I am open to discussions.

    Thank you for your time and attention.

    Warm regards,

    Rasheed

    Disclaimer: I work for Symantec. Comments are my own.

  • Thanks for the offer to comment – but it would be complete offtopic to do right here, and I don’t want to start this on your blog where the comments are moderated by you.

    Additionally, I don’t feel it would be a good use of my time – I am not a teacher to red pencil your bad homework. Many signs point that you do not know even the basics of our product (starting from core stuff such as default backup mode). You clearly have never touched the product, nor even opened the User Guide. You really should start from there first.

    You are talking about being professional, but how professional is that to dissertate about something you have absolutely no knowledge of.

    Cheers!

  • @Anton

    I am sorry that you feel that way especially when my sources are your partners as well. In any case, I prefer to stay professional in action. I am on board for not littering Curtis’ blog.

    Warm regards,

    Rasheed

    Disclaimer: I work for Symantec. Comments are my own.

  • Curtis,

    Excellent post! This is the first I’ve heard of Veeam, so I’m excited to download and run the trial.
    The lack of tape support probably means that I won’t take it beyond the trial period, but otherwise, I’m impressed (based on what I saw/heard in the Tech Field Day video).
    From convenience and usability perspectives, I understand your point about a self-backup. Those willing to take a few extra steps could use the task scheduler to dump the Microsoft SQL database for Veeam. In SQL management studio, right-click the database, and in the Tasks menu, click Backup. Set your backup options as desired, then in the Script menu, click Script Action to File. You can then batch sqlcmd against this file, and assign a scheduled task to run the batch. (here’s a nicer how to: http://www.brianmadden.com/blogs/guestbloggers/archive/2007/05/07/how-to-automate-the-backup-of-a-sql-server-2005-express-data-store.aspx) It’s probably slower than if this were neatly packaged into Veeam, but at least it’s possible.
    On a separate note, have you used BackupAssist (backupassist.com)? If so, what is your take?

    Disclaimer: I’m “guilty” of having used AT + NTBACKUP for backup. 🙂 However, I am not affiliated with any backup or storage vendor.

  • Curtis – I use Veeam and recommend it to clients in most of them that have a requirement (SLA, Compliance, … other) to archive or offsite data to tape we use BE. It’s the best solution we could come up with.

    I agree that some built in facility to “back itself up” would be nice. I think my solution below combined with the extract.exe script is probably the best that can be done at present.

    For those who utterly despise tape the other option we’ve started doing (when possible) is using removable hard drives as offsite media and ditch tape completely. In those cases we just have a post backup script that does an rsync/robocopy of the entire Veeam repository to removable disks and rotate the disks. This doesn’t apply to everyone, but does meet some folks requirements, as Iron Mountain is pretty willing to take whatever you’re willing to send them.

  • I just went through a backup evaluation and at the start of it I thought that I was going to go with Unitrends hands down. After the long process of evaluating Unitrends new UEB software, I ended up going with Veeam and have not been happier. People live and die by their backups in IT. You can find my experience here if interested. http://techninety.me/category/veeam/

    All I can say is stay far far away from Backup Exec!