Thirteen years ago, two companies accomplished the impossible and created NDMP. It’s become such a standard way to back up NAS that you may have forgotten just how revolutionary it was when it came out. I’m going to remind you of its history and say that history needs to repeat itself with dedupe & virtualizatioin. Click Read More to see what in the world I’m talking aobut.
Thirteen years ago, NetApp & PDC (the makers of BudTool, remember them?) realized they had a common problem. NAS filers were this weird thing that no one knew how to back up and PDC was getting a lot of requests to back up filers. They created the network data management protocol, or NDMP. There was FTP for file transfer, an SMTP for mail transport, and then there was NDMP for backing up filers.
First, let me commend NetApp for what they didn’t do. They didn’t create a NetApp-only solution or a PDC-only solution. They could have, but they didn’t. (It’s on my list of reasons why I’ve always liked that company.) They actually got competitors to sit around a table and talk about what they needed to be in (and more importantly outside of) a specification for a backup protocol for filers. One big concession was that the backup format (e.g. tar, cpio, dump) was outside of the spec. Each vendor could use their own backup format, but they would all be controlled in the same way. While this created a problem for those wanting to restore NetApps to Celerras, it gave each vendor just enough control to agree to a standard way to back up filers.
Now NDMP is the standard way to back up filers and no one disputes that. Other than NFS/CIFS-mounting them to your backup server, it’s pretty much the only way to back up filers. Other methods have tried and failed. (Remember NetWorker’s ClientPak for NetApp?) NDMP is the standard.
Now we have two very different technologies that have similar problems: target dedupe & server virtualization. First let’s talk about dedupe.
Once you’ve backed up to a dedupe target, how do you copy its backups to another one offsite? How can you do that in a way so the backup software can control it, know about it, and report on it? What about copying to tape? If you have replicated from one dedupe box to another, how do you copy that replicated backup to tape at the replicated destination? Good luck on both these accounts.
I’ve already posted previously about my thoughts about NetBackup’s Open Storage Option (OST). The short version is that I’m fan of what OST does, but I’m not a fan of how Symantec did it. When they asked me years ago about the OST idea and their NDMP “direct-to-tape” mechanism (which is a completely different way to solve part of the problem, and is incompatible with OST), I told them I liked the idea, but I begged them to bring more than one ISV (independent software vendor) to the table. I knew all the OEMs would participate, of course, because they were Symantec and they owned (and still own) the lion’s share of the backup software market. So we knew that Data Domain, EMC, Quantum, Falconstor, SEPATON, and others would sign up to partner with them on this. What I wanted, though, was at least one other backup software product to be in there so that they wouldn’t create a Symantec-only solution. Well, we know what happened. Open STorage (OST) is anything but open, and Symantec put NetBackup-specific stuff into the NDMP direct-to-tape feature as well.
This leaves me torn. On one hand, I think that this functionality is so important that users should vote with their dollars and show the other ISVs that they agree by converting to NetBackup just to get this functionality. On the other hand, I don’t want to reward Symantec for doing what I ultimately feel was a selfish act that was completely opposite of what NetApp did thirteen years ago. By the way, I’ve also heard from dedupe OEMs and other backup software products that some of them are working on their own product-specific way to solve this problem. Great…. Thanks for starting a trend, Symantec.
Someone needs to step up to the plate and stop this madness. We need at least one major ISV and at least one emerging ISV (or another major ISV, of course) and more than one OEM to get together and work this out. Stop thinking of it as a competitive advantage and think of it as a common problem that you can all work out together much more easily — and then move on to more important things like making your systems faster and more reliable.
- Will Symantec make Open Storage actually open? I’d be fine with that, assuming other backup products could actually use that API.
- Can NDMP be further extended to meet the needs of the dedupe community? It’s already been extended beyond its original design to include management and cataloging of snapshots, and creation of tapes by VTLs. Is this so far off?
- Will anyone from CA/CommVault/EMC//HP/IBM/Arkeia/Atempo/Bakbone, or any combination of the above sit down at the same table?
- Will Data Domain/EMC/Exagrid/Falconstor/HP/IBM/NETAPP/Overland/Quantum/NEC/SEPATON/Sun, or any combination of the above sit down at the same table?
- Do any of you have any ideas? I’d be happy to listen to them, offline if necessary. Drop me an email.
Let’s not forget server virtualization. While VMware, HyperV, VirtualIron, and Xen server are all great, backing them up stinks. It combines the most I/O intensive application in the datacenter (backup) with the system least able to handle intensive I/O loads (virtual servers). VMware has created VMware Consolidate Backup, or VCB, which makes things better (albeit more complicate) for VMware customers, but what about the rest of these products? I don’t have as much voice in the virtualization community as I do in the backup community, but it seems to me that the different virtualization products are very analogous to the early filer products, and that someone (probably VMware) should step up to the plate and create an NDMP for VMware. Hey, they can even use NDMP for all I care. Just give us a better way to back up our virtual servers that isn’t tied to a specific product. Can the NetApp of the server virtualization world (that would be you, VMware), be so magnanimous as to start such an initiative, or will they only see it as a way to help out their competitors? Or will they see it the way NetApp saw it, as a way to further the market and therefore make everybody’s pockets heavier?If I can facilitate any of the above happening, I’d be very glad to do so. If you can’t wait for The BD Event in June, I’d be happy to broker a meeting at SNW or Storage Decisions. Just let me know.
Let’s get history to repeat itself, shall we?
----- Signature and Disclaimer -----
Written by W. Curtis Preston (@wcpreston). For those of you unfamiliar with my work, I've specialized in backup & recovery since 1993. I've written the O'Reilly books on backup and have worked with a number of native and commercial tools. I am now Chief Technologist at Druva, the leading provider of cloud-based data protection and data management tools for endpoints, infrastructure, and cloud applications. These posts reflect my own opinion and are not necessarily the opinion of my employer.