Hyper-V ahead of VMware in the backup race

Hyper-V offers fully-supported, application-aware, transactionally-consistent backups of any applications that have a VSS writer.  These include Exchange, SQL Server, Oracle, SharePoint, and others.  There is no need in Hyper-V to put an agent in any VM.  Your applications will get properly backed up and they will know they’ve been backed up (thus clearing their logs) — without any agents.  VMware, on the other hand, offers no such support — even though the functionality to do so has been available for seven years. And I think it’s time we talk about it.

BTW, if you’re not that familar with Microsoft VSS and all it allows you to do, you should read this blog post I wrote last week first.

This all started with a blog post from Scott Waterhouse of EMC that said that the only way to get application backups in VMware was to use an agent-based backup like Avamar.  My immediate reaction was that Scott was off his rocker.  He had no idea what he was talking about.  (Wouldn’t be the first time I thought that, eh, Scott?)  So he and I talked via email and then he called his people and I called my people.  Along the way I had a Twitter epiphany which I blogged about.

There are three features that are very important here:

  1. Can Hyper-V/VMware get a consistent backup of a virtual volume within a VM (i.e. take a snapshot of the volume via VSS and then back up that snapshot)
  2. Can Hyper-V/VMware get a consistent backup of an application residing with a VM (i.e. Use VSS to quiesce the app, use VSS to take a snapshot of the volume(s) the app resides on via VSS and then back up that snapshot, then release the app)
  3. Can Hyper-V/VMware tell the application that is has been backed up so that it can clear its transaction logs (i.e. use VSS to tell it that it has been backed up)

Number 1 is table stakes.  If you can’t do that, forget it.  Number 2 is also important obviously.  However, without Number 3, you’d need to run a backup agent just to tell the app to clear its logs.  With that in mind, the following table explains what is possible in both products.

Feature

Hyper-V w/Windows
2003 guest

Hyper-V w/Windows
2008 guest
VMware w/Windows
2003 guest
VMware w/Windows
2008 guest
Consistent backup of a volume Yes Yes Yes Yes
Consistent backup of applications Yes Yes Yes Yes
Applications aware of backups Yes Yes Yes No

VMware obviously can back up a Windows volume via VSS.  However, its support for applications is sadly lacking. (Update 11/30/10: They addressed the Windows 2008 issue.) It can only quiesce applications if they happen to be running on a Windows 2003 guest, and it It cannot tell any of the applications they have been backed up.

What this means is that anyone wishing to get proper backups of applications in Windows must run an agent of some kind in their guests in order to make this happen.  Please note that this is not to say that you must perform guest-level backups.  While those may like sound contradictory, statements this post should help clear up the fact that they’re not.

This means that any backup tools that are using only VMware’s infrastructure are going to have the same limitations.  These include, but are not limited to:

  • Almost all mainstream commercial backup applications are using the VCB and vSphere backup APIs and nothing else. If VMware isn’t quiescing an application and telling it [the app] it’s been backed up, then neither are they.  This is one of the reasons why so many people continue to do guest-level backups of VMs to this day.
  • VMware Data Recovery (of course) only uses VMware tools and can’t properly back up applications either.
  • Vizioncore’s vRanger Pro also uses VMware’s VSS tools, according to this this forum post from last month.  I reached out to them for comment and they did not reply. [Update: They have also addressed this issue.]
  • PHD Virtual’s ESXpress didn’t appear to be using VSS at all. (Try searching for the phrase in their documentation.)  However, I did find this forum post via a google search that talks about an undocumented feature for doing this.  They also have not replied to my request for comment. [Update: They have addressed this issue.]

A few rays of hope

Another way to backup VMware has been using NetApp’s SnapManager for Virtual Infrastructure (SMVI), so I took a look at their website.  But their product page for that product says that they work “in conjunction with VMware snapshots” which… call.. the VMware VSS requester.  So I wasn’t feeling very good about that until I talked to NetApp.  They confirmed that if you want to perform application-consistent snapshots, you want to use SMVI in conjunction with the appropriate SnapManager product (e.g. SnapManager for Exchange, SQL, or Oracle).  These SnapManager products will place an agent in the guest, but the only purpose of the agent is to talk to Exchange and VSS to get it to do what it needs to do before it’s backed up.  If you do this, NetApp says the products work together to “do the right thing.”

Veeam has a similar approach (which is now mirrored by their competitors). Their website says, “Unlike other vendors featuring limited VSS support, Veeam provides the most complete implementation of VSS support, equipping you for proper restore of VSS-aware applications (e.g. Active Directory, Exchange) from the created backups.”  Hmm, this sounded interesting.  I did a google search against their site and turned up several forum posts like this one where they’re using all the right terms (which I’ll cover later in this post).  I contacted them and they clarified that they do use VMware’s VCB or vStorage API to do backups without sending data through the guest, but enhance VMware’s capabilities by placing small agents in the guests that they communicate with at the appropriate time to do the right thing.

So this is what I meant when I said earlier that putting an agent in a guest did not automatically translate into doing guest-level backups. Sometimes the agents are there just to coordinate things.

Another ray of hope is that, although they won’t say when it’s coming, my VMware contact told me that they are working on full support for VSS backups including telling the application that it’s been backed up. 

And now I’m steamed

Do you have any idea how difficult it was to drag up this dirty little secret?  It should have been as simple as reading a support matrix on VMware’s website.  There is no such matrix for the vStorage API and you can only find the VCB version of it with a lot of digging.  And even when you find  that page it appears that they do the right thing in Windows 2003 (at least).  It took a phone call to a very helpful VMware person to explain that this only means that they can quiesce the app; they do nothing to tell the app that it’s backed up.

My opinion is that this is just as important as which piece of hardware you need to buy to run VMware on, and it should be listed prominently in the VMware compatibility guides.  This might prevent someone, for example, from upgrading from a version where they do support something to a version where they don’t support it.

Summary

The net/net of this is that Scott’s original post that started this needs a few corrections.  He’s close, and he gets kudos for bringing the whole thing up.  But you can get transactionally consistent backups without doing guest-level backups.

  1. As long as you’re still running Windows 2003, you You can get application-consistent backups using VCB or the vStorage API.  It’s just that you’re going to need to figure out how to get them to truncate their logs.
  2. You can get fully-functional backups — and the apps will know they’re backed up — if you use NetApp’s SnapManager line or Veeam’s backup product (or any of the other products that now support this).  They both do put an agent in the guest, but the guest is only for coordination and does no data transfer in the VM.
  3. And, of course, you could get all this wtih pretty much all backup products if you put your apps into Hyper-V.

Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at Sullivan Strickler, which helps companies manage their legacy data

10 comments
  • I hope they’re not pulling my leg, but I finished up watching a recorded webinar from Symantec for NBU7+VMware. The speaker said that the NBU7 backup is clientless (on the vm hosts) and that it pokes VSS to quiesce the filesystem. I’m not sure if it nudges any applications that are responsive to VSS (I’d expect it to, but I’m not holding my breath). It’s also not a “crash consistent” backup.

    To your defense, the speaker didn’t mention anything about telling Windows that it was backed up. A real-world example for me is if by some circumstance, I’d want to run Exchange and Enterprise Vault in VMs. The safety backup, email removal, and shortcut creation all require that the applications know they’ve been backed up (both Exchange and EV).

  • Trust me. NBU 7 is NOT quiescing the apps. And if it’s quiescing the filesystem without quiescing the apps, then yes, it’s a crash-consistent backup.

    As the article said, VMware does not support doing that for anything but Windows 2003, so if the product didn’t build a work-around, then it’s not getting app-consistent backup. Backup Exec actually did build a workaround, but NBU did NOT. They’re trusting the VMware will eventually do the right thing.

    As to telling the apps they’re backed up, definitely not. VMware uses VSS_COPY and that does nothing when it’s done. Until they change that, the apps will not know they’ve been backed up and will not truncate their logs — even for 2003.

  • Curtis,

    thanks for dissecting the whole VSS story. Lots to learn here. Since you posted this, there has been an update to the platform / consistency-level matrix with the VMware VDR 1.2 admin guide (http://www.vmware.com/pdf/vdr_12_admin.pdf). W2K8 now has application-level consistent snapshots on ESX 4.1, subject to a bunch of constraints (no IDE disks, enough available SCSI slots on the virtual bus, no dynamic disks). Doesn’t read straight-forward nor could I find any further notes on application awareness during the backup process. Hard to believe that the VMware VSS integration has way to go when something essential like it should rather be working out of the box and/or at least be well documented.

    Sebastian

  • Surely MS can support multiple apps with log truncation because they support only their own applications (i.e. the writer has to be aware of the specific application in order to perform all required functions).

    If so, as mentioned elswhere, it’s really the backup vendor’s responsibility to add a lightweight helper agent to perform that function (like NetApp does).

    What VMware could do is to make these backup helper agents pluggable components of the VMware Tools package. That is, a vCenter server-side plugin supplied by the backup vendor includes the backup vendor’s backup helper agent, and a vCenter policy is set to install the backup vendors agent on all/some VMs as part of the tools).

    At least then the management of the backup helper agent can be done centrally.

    Otherwise VMware will forever be adding application specific VSS code the VMware tools. Or am I wrong this?

  • When we use Hyper V to take a full or incremental backups, what is the type of backup that the applications in the guest VM get? Is it the same or is it always full?

  • Curtis, just saw that you’ve updated the article WRT to VMware’s W2K8 application quiescence support. Do you have a reference to public VMware information by any chance (release notes or such)?

  • I’m on the road at the moment and am not sure where I read it, but I read it in multiple places. Basically, you’ll see support for application-consistent snapshots, which requires the functionality I’m describing.

  • Thanks Curtis. The relevant bit is contained in the “What’s New in vSphere 4.1” document. http://www.vmware.com/support/vsphere4/doc/vsp_41_new_feat.html.

    “vStorage APIs for Data Protection (VADP) Enhancements. VADP now offers VSS quiescing support for Windows Server 2008 and Windows Server 2008 R2 servers. This enables application-consistent backup and restore operations for Windows Server 2008 and Windows Server 2008 R2 applications.”

    There’s some more technical information in http://kb.vmware.com/kb/1028881 “Enabling Windows 2008 application-consistent quiescing on ESX 4.1”

  • Thanks Curtis, very good piece of information.
    Consistent backup plus transaction logs truncation is important indeed, however there is another part which I don’t think is properly handled on either platform.
    According to Exchange VSS backup API, the Exchange database must be verified with eseutil or CHKSGFILES API.
    The goal of such verification is that once a corruption occurs, backup application notifies the administrator and fails the backups, not flushing the transaction logs. So either the database will be repaired manually, or restored to the latest available state by restoring from last good backup, plus rolling forward the transaction logs.
    Without such verification, the corruption will get undetected into the backup storage, and eventually all “good” database states will expire and will be purged from the backup.

    To be honest I’m now aware of any host-based backup application which properly handles this case

  • Another issue that I am wondering about is how to create a non-VSS-aware application backup.

    As simple as.
    1. Shutdown or pause the non-vss-aware application
    2. Netbackup or any other application calls VSS
    3. VSS creates the snapshot and leaves the disk in read only mode
    4. Start or pause the non-vss-aware application
    5. Take the backup.

    How do I do that?