New solutions for using replicated backup data

There have been some interesting options that have recently presented themselves for using deduplicated, replicated backups on a regular basis.  If you’re not familiar with the challenges of using these backups without some help, I’ll explain those first, and then go into the solutions to these problems that are showing up.

Update: I’m talking about backups from a regular backup software product that are going to a VTL or IDT (intelligent disk target).  I know there are all kinds of other backups out there that use replcation.  In this blog post, I’m talking about regular ol’ backups.

Those of you that have followed the blog know that I’m a fan of using deduplication and replication to get backups offsite without shipping tapes.  However, there are challenges with using those replicated backups to do things like make copies to tape, or (Heaven forbid) perform restores.

If you want to use them only in the case of disaster, what most people do is use a standby main backup server and then figure out some way to regularly get the backup catalog/database over there.  This can be done via replication of the catalog/database itself, replication of catalog/database backups to the disk device, or even shipping catalog/database backup tapes to the remote site.  The down side to the latter two approaches is the downtime required to restore the backup catalog/database before restores can begin.  The first approach works well, but many people do not use it for some reason.

Those that are more focused on day-to-day copies to tape at the replication destination have thought of using a media server/storage node/device server in the remote site that can access the replicated backups, but there are problems with this approach.  If we’re talking a NAS system, the replicated copy is served from a different hostname, so the backup software will be not look for it there.  . (I have thought about hacking host files so that the remote backup server thinks it’s mounting the same NAS system that’s in the source site, but I’m not sure if that will work.)  If we’re talking a VTL, the matching bar codes will cause the backup software to think that one tape is in two places, so that definitely doesn’t work.

The first solution to this problem was Symantec’s OST, which I’ve already blogged about here and here. The second solution was with CommVault.  Although I have not tested it, they tell me that they can have a media agent watch a replication target directory and look for backups that show up there after replication.  Since the media agent will have access to the database that is tracking what’s in those backups, they’re saying that this would allow you to use the media agent at the remote site to copy backups to tape on a regular basis.  It appears that this will work only with NAS products, as they would still have the matching bar code problem with VTLs.

I also talked with the HP people yesterday where they updated me on what they’re doing with Data Protector. They also have a unique solution to this problem if you use Data Protector and the HP VLS (a VTL based on SEPATON).

Their solution is enabled by a new feature in DP that allows one server to import backup catalog entries from another DP server.  This allows you to move a tape (or replicate a virtual tape) to another DP server and have the other DP server ask the first DB server for the catalog/database entries of what’s on that tape. 

So their solution starts with a separate DP server in the replicated site (not a media server).  It then uses an automated script that is run occasionally via a scheduler.  The script looks for new virtual tapes that weren’t there the last time the script ran, and then it asks the source backup server for the backup catalog information for those tapes.  Once that data is imported from the source server to the destination DP server, the tapes can be used for whatever you want — copies to tape or DR restores.  It’s essentially like the best option described above (mirroring of the catalog) without having to use mirroring software.

I look forward to other solutions to this problem.

Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at Sullivan Strickler, which helps companies manage their legacy data

11 comments
  • Avamar has done this type of thing for at least a couple of years. It’s very cool. We have our remote Avamar servers replicate to our main site and the main site is replicated to one of the remote sites. From the Avamar console (or a script) you can select the replicated systems and restore stuff, pretty much the same way as for a local system.

    We use this to put product data PDFs out to our web site. A script in a cron job restores all the PDFs from the remote plants and then rsyncs them to the web server.

  • I know that the source dedupe systems (Avamar, Puredisk, Asigra, Evault, etc) can do this. That’s not what I’m talking about.

    What I’m talking about in this post is doing things like this with target dedupe systems. Because many people need target dedupe due to the performance characteristics of source dedupe. (Source dedupe is fine for remote sites and smaller datacenters, but not larger datacenters.)

  • Hello Curtis,

    It appears that the release of Native IP replication in IBM ProtecTIER has escaped your attention. This feature has been available since early September. It allows customers to replicate virtual cartridges based on user definable policies to a secondary site. The mechanism is designed to mimic the ‘real world’ as much as possible.

    Replicated cartridges are sent to a virtual shelf at the destination site, where they can be imported for physical tape copy or used in DR scenario’s. Visibility of the replica can be controlled, and moving virtual cartridges from site to site is accomplished through virtual library mailslots. This allows full integration in any backup software. Both single domain and multi-domain backup environments can be used with this approach.

  • I know that ProtecTIER has native replication. I also know that it had a method that worked prior to that. So does every other dedupe system on the planet. (Well, except Netapp’s VTL, that is.)

    Getting the tapes there is only half the battle. Those tapes need to be usable by the backup app. The IBM solution (and most others) only address the first half of the problem. However, if you’re going to use the replicated backups to regularly create tape copies offsite, well, you’ve got a problem. And many people have talked about having an onsite disk copy, an offsite disk copy, and an offsite tape copy. The products listed in this article have made that possible to varying degrees.

    In addition, the backup app doesn’t know about your replicated copy. It can’t report that it’s there, not there, etc. If a customer is using something like Aptare, Bocada, or EMC DPA to report on their backups, your replicated copy will not be reported on. If your product supported OST and it was a NetBackup customer, it would be. Or if your product supported a NAS connection and it was a CommVauult customer, it would be. The next thing ProtecTier needs to support is OST — and please make sure you do it via Fibre Channel.

  • With ProtecTIER you create a local export copy of your backup sets, just like you would with physical tape. This does mean extra processing on the backup server to create the export sets, but requires no additional storage space as all data for the copied sets is already in the repository and gets deduped real-time inline. The export tapes will have their own barcodes, their own location etc.

    The backup app can place these copied volumes in the virtual mailslot which automatically moves the logical location of the tape to the remote site. There they can be inserted in the virtual mailslot of the remote library which will prompt most backup software to move them to slots and scan them. The backup app knows about these copies, and Aptare et al will be able to report about them.

    In a single domain model, both local and remote library will be under control of the same backup server. It will know about the replicated tapes (it created them as export set) and it will not be ‘surprised’ when these show up in another library under its own control. This is identical to the physical world where a courier would pick up the box with tapes, deliver them to the remote site and an operator puts them in the remote library.

    I agree that in a dual domain model the backup server needs to scan the newly arrived sets, as they will appear foreign to the secondary backup server. It is then up to the capabilities of the backup software to create additional copies to tape.

    Can’t comment on the additional features we should or should not implement, I’m just a simple engineer ๐Ÿ˜‰

  • But that’s the same 2/3 that any vendor that has dedupe and replication gets. Because, while the backup sofware will know about the copies, it won’t know if they GOT there because it’s not controlling the replication. So you’ll need some script to figure out that the tapes have finished replicating before you can start the copy. And if the replication fails, the backup software (and Aptare/Bocada/EBA/VBR) will have no idea.

    I know, I’m a pain.

    BTW, I don’t think that simple is what the S in SE stands for .;-)

  • Don’t you know I’m talking about regular backups? Well, if I confused even you, then I must do an update to remove confusion.

  • You’re talking about the traditional backup paradigms that have been in place for years. Now, add deduplication and replication – makes sense, I agree that’s a great combination for taking advantage of technology.

    The however in my comment is this, if a customer is running on NetApp – why not take advantage of that technology for protecting their data through deduplication, replication. Heck even if they are running on othEr Massively Complex disk they can front end that with V-Series and take full advantage of all the simplicity AND features found in the NetApp storage such as thin provisioning, replication, deduplication, etc.

    Then if they want to drive it to tape – they can use a backup server to do so via NDMP. Sure reduces the amount of $$ spent on backup licenses and more.

    Chapa

  • Hvaing read the article (hoping I caught your idea); I’d like to comment the approach you qouted from CommVault. Isn’t it so that the media agent would as well be depending on the ability to connect to the catalog in the primary site then? I am not Commvault specialist though, but me the approach mentioned for Commvault sounds like what almost every more or less enterprise backup product can do: to rescan a given media files directory, and identify that it suddenyl finds carridges here which have presently been stored in the primary sites media directories…. in my opinion, this is a re-wrap of admins doing inventory of a tape library who’s door has been opened and closed by what used to be called a backup operator (I wonder if that role exists still in this way? I wouldnt know). I wouldnt know how much that would deliver independence from the problem of having only one place where catalog data is consistently stored at the time of backup. The issue is that as long as there is no clustered database for catalog data (trying to not give impression that a clustered database would be the solution), there is no redundancy.
    I’d suggest to try a different brain experiment, since I have recently started to look into storage virtualisation products since I started working for a vendor in this field recently after having spent many years on the backup software vendors side of the world. Why not virtualize the backup server in a clustered server virtualisation, letting the primary sites backup server incarnation run from and store its catalog on a set of virtual LUNs that are provided by a clustered storage virtualisation product which does replication (e.g. based on periodic snapshots, or if you wish, block based CDP) to the DR site, and let the backup server write its backup data to a VTL product which does basically the same, a cartridge file delta replication to the same DR site as well. To safe bandwidths, I’d suggest to de-duplicate the stored cartridges on the primary site before replicating the data, but I have to say the schedule would then be a thing to look at very precisely because this would have to be done before replicating the VTL cartridge data and all within the time frame which is given by the customers protection expectations. At the end of this time frame X (to be designed according to the business’ needs – obviously depending on what you want to spend), there would be a consistens replication of the catalog and OS LUN’s of the backup server in the storage virtualisation cluster’s secondary node, plus all the corresponding VTL cartridges that are needed for this catalog to make sense, in the VTL cluster’s secondary node at the DR site. Firing up the backup server at the DR site could be done on top of a temporary R/W view (e.g. based on the latest snapshot) of the replicated LUN as to avoid premanent (and conflicting) changes to the replication target LUN or replication pairing itself, for testing purposes. In the bad case where the primary site cannot be used, the secondary site would have to inherit the role of being primary, and backup production could continue here. The virtualisation of the backupsoftware allows to go live with the secondary site version of the backup server with no change, since it is launched from a LUN which contains the exact copy of the backup server’s LUN as it was used in the primary site, before. Did I forget something? Ermm licenses would maybe have to be covered by virtualising the license server itself as well, or maybe spanning a multi node grid of tiny e.g. linux boxes running the license service.

    On the other hand, I believe that if there is a need to launch (and keep running) the backup server on the secondary site permanently (which automatically excludes any option to snycroneously mirror or even replicate based on schedule, whatever device the primary backup server stores its catalog on), and we have no time window to go in maintenance mode and do magic behind the scenes to copy over catalog data or sync volumes etc, the only way I can imagine is to replicate the VTL cartridges in what could be called a “backup catalog aware” mode, which means that all data of the two-be-replicated data would have to be read by the backup server, transferred over to the DR site, and inserted there into local VTL cartridges as well.

  • I have thought about hacking host files so that the remote backup server thinks it’s mounting the same NAS system that’s in the source site, but I’m not sure if that will work.

    Yep, this absolutely works. Vendors such as Data Domain actually recommend this approach in the scenario you describe (for those customers not using OST at least). Before OST existed, it was the defacto recommended approach. It’s not the most robust of approaches but it certainly does work.