NetBackup Open Storage Option (OST) vs sliced bread

Maybe you were like me when you first read about NetBackup’s Open Storage Option (OST): you were underwhelmed.  You also may have been waiting for vendors to jump on it.  (It was announced two years ago.)  Things have definitely changed since then.  In fact, at a recent very large customer that was considering purchasing a dedupe target, they chose only to look at vendors that supported NetBackup’s OST.  Click Read More to see why this option is so important for NetBackup customers, and why other backup software vendors better come up with something like it real-quick-like.

Like others (especially VTL vendors) I did initially see Symantec’s Open Storage Option as an attempt by Symantec to kill the VTL market and replace all of them with regular disk behind PureDisk servers. I remember saying to one of their product managers (in a not-so-friendly manner), “VTLs are not going away, and the sooner you stop ignoring that fact, the better!”

Then I found out that OST wasn’t anti-VTL. It’s one of the most common misconceptions about OST (promulgated on Scott Waterhouse’s The Backup Blog). The truth is that OST doesn’t care how the target storage device stores its backups, which allows each disk vendor to store them how they want to store them. If they want to store them on a filesystem, great. If they want to store them on virtual tapes, fine. Just be able to accept a backup by name and store it and retrieve it by name. I liken the API to RMAN. RMAN gives you a blob of data and calls it “database_string_date_timestamp_etc.” and asks you to store it. It doesn’t care HOW or where you store it, just that you do, and that you give it back when they ask for it. That’s exactly how OST is.

The Open Storage Option (which, for some reason is referred to by Symantec folk as OST.  I’ve been told it’s Open STorage.  Alright…) is an API put forth by Symantec for writing to disk. When NBU wants to send a job to disk, it tells the storage device via the API that it has a backup for it to store, and it gives that backup a name. The storage device then accepts the backup and stores it in a manner of its choosing. After that point, NBU can do one of three things: expire it, restore from it, or copy/duplicate it. If the storage device is told by NBU (via the API) to expire the image, then it is supposed to delete it from storage. When NBU wants to restore from an image, it tells the storage device via the API what image it wants to restore from. The storage device then delivers the image to NBU for restore. (Again, both of these are very similar to the way RMAN talks to backup apps.)

True coolness comes when NBU wants to duplicate a backup. OST supports what it calls “optimized duplication.” In order for this to work, you need to have registered two OST-capable storage devices from the same vendor. NBU then tells device A to perform an “optimized duplication” to device B. The two devices then communicate and figure out which new data segments have to be replicated from device A to device B in order for device B to be able to say it has this image. (The number of segments replicated will vary greatly based on deduplication factors and how unique this image is in comparison to previous images already stored on the target device.) When the replication/optimized duplication is done, it reports that to NBU via the API, and all of this shows up in the Activity Monitor. Obviously, if there are problems with the copy, the storage device can report that via the API as well, and that will be reflected in the Activity Monitor. When the job is completed successfully, NBU will now know about both copies of the image. You can connect a media server to the replicated device B and can use those images just like you would any other images, which includes copying those images to tape.

This capability is 100% in line with what I’ve been describing for years as the Nirvana of dedupe. Backup to one device, replicate to another, have the backup app know about the replicated backup, then copy the replicated backup to tape. Now I have onsite backups, offsite backups, and a long-term-retention copy on tape – and I never have to move tapes around. No more guys in trucks! No more chances that one of them will go in Starbucks while the truck is stolen, or drop your tape along the way – no more chance your company will have to go on CNN explaining why you lost your tape. Your only tapes are in a locked tape library inside a locked cage inside a locked data center inside a secure building – and they never move. Suuuwheet.

So that’s my feeling about OST. I think it’s the greatest thing since sliced bread. Other vendors need something like it, but I’ll do another post about what I think they should do.

Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at Sullivan Strickler, which helps companies manage their legacy data

4 comments
  • I’ve configured OST (I just thought the “T” was silent) for NBU with Data Domain and thought it was pretty slick. I finally got to use Storage Lifecycle Policies. Instead of Storage Units, we used SLPs, choosing the local Data Domain appliance for the backup and the remote Data Domain for the replication target.

    For Data Domain, it appears that OST will only work for the filesystem side, not the virtual tape side. On the Data Domain itself, you actually create a separate space for OST and then create Disk Pools in NBU, so the OST space is not accessed via the fibre connections. At least, not at this point.

  • Curtis,
    Glad you find OST to be preety cool. OST = OpenSTorage. The “T” comes from storage.

    The decision about where to use the API and on what media server that plug-in will work is dependent on each vendor. For example, FalconStor supports an OST plug-in for their “FalconStor VTL” with dedupe on Solaris. They’ll add support for Window shortly. Data Domain supports a few more platforms (Windows, Solaris, RedHat). You get the picture and can always find the latest details on the NBU HCL.

    – Peter

  • Very good article indeed.

    My questions:

    How does A know that B has X data? I assume fingerprints are checked by NBU and not by Data Domain (as in my case)

  • @Chirag. A doesn’t need to know that B has the the same data. NB master server needs to know. There are 2 copies of the catalog file, primary and secondary. Once opt duplication has completed, the secondary catalog image is created on the master server. To use the secondary catalog image for restore, you set copy 2 to primary, this tells NB to use the data on “B”. hope that helps.