Copyright (C) 2007 Alain Georgette / Copyright (C) 2006 Frantisek Hliva. All rights reserved.

2007-07-31 02:18:55

I think NBU6.5 will use source de-duplication, as it will have puredisk technology built in. I wonder how much more resources this will require from master and/or media servers.

it is also the only way to do efficient remote backups, but nobody ever talks about (remote) restores. prepping a server at the main site and then shipping it somewhere at the other side of the globe is not always a good/prefered way to do it.

for me the restore part is still one piece of the puzzle that needs to be solved.

2007-07-31 11:01:37

You can already buy the Puredisk product now, and yes, it is source de-dupe. NBU 6.5 will offer greater integration between the two products.

_I_ talk about remote restores! Check out my next blog entry: "De-duplication & remote restores."

66.92.234.xxx

You seem to suggest, rather explicitly, that dedupes are functioning on the file level. Not only am I pretty sure that's not true (including not of Puredisk, at least, that's not what they said at the EDPF in Minnesota in 2005), but I think it'd be pretty boneheaded to hash files rather than blocks.

Was that just for ease of understanding for the reader?

2007-07-31 17:00:43

I just re-read the article, and I could see where you think I'm saying that de-dupe is file-level, but that's not what I'm saying. I just gave a duplicated file as an example. I'll re-edit the blog entry and give another example.

BTW, There is file-level de-dupe and sub-file-level de-dupe. File-level de-dupe is also called CAS, or content-addressable storage, and yes -- they do the hash at the file level.

But what we're talking about here is sub-file-level de-dupe. This catches not only duplicated files, but also duplicated pieces of files (i.e. blocks) that have already been seen. So when you back up a spreadsheet every day because it gets updated every day, each day you should back up only the new blocks in that spreadsheet.

2007-08-02 09:23:46

So, which way will be the most cost effective route to take for data de-dupe? By using NBU 6.0 & Purchasing the Pure Disk Option OR by upgrading to NBU 6.5 with the Pure Disk already built in?

I have been looking at other options than NBU for de-dupe and there are some good ones and bad one in comparison to Pure Disk.

2007-08-02 10:15:07

As far as I can tell, Puredisk will not be built into the base product. It'll be an option just like it is now. It's just that it will be more integrated with the base product.

Nice try, though.

2007-08-24 10:17:07

In a recent podcast (sorry, I forget which one), Curtis said "inline or delayed de-dupe doesn't matter when selecting a backup appliance" (my words and estimation of Curtis' intent). Although he went on at length to discuss some effects, I thought he left out a few important considerations.

First, some or all delayed de-dupe systems seem to require scheduling of a de-dupe batch process where the data or data location may not have full functionality during the de-dupe process.

Second, the de-dupe process is extremely resource (especially CPU) hungry. With the process running, the appliance may not perform reasonably with other functions during the process. of course this will change for the better over time. Inline de-dupe appliances are built to withstand high-resource consumption during backup ... there really are no significant back-end processes.

Third, if the deduped data is sent "off site", such as with a proprietary system sending the de-dupe information to a similar box, ... the movement off site is necessarily delayed until de-dupe can be accomplished. No such delay is required if inline de-dup.

Fourth, if de-dupe is done "inline", the process of getting data from its source to off site is simpler. Simple is good.

For these reasons, I see the vendors with "inline" de-dupe as having a significant advantage ... one that shouldn't be waived off as unimportant.

Of course, it's possible that I didn't realize my hearing aide batteries needed changing at the time I listened to Curtis' podcast.

... and my generalizations may well be worse than I think Curtis' was. cheers, wayne

2007-08-24 14:51:30

I did say that none of the features that people typically debate matter (inline vs post process, MD5 vs SHA-1 vs custom, reverse vs forward referencing, etc). What matters is:

1. How big is it? (i.e. how much disk do you give me and what de-dupe ratio do I get with my data?)

2. How fast is it? (i.e. how fast are backups, restores, and the overall de-dupe process?)

3. How much does it cost?

All of your arguments against post-process above are aiming at #2. My opinion is neither in-line or post-process de-dupe system can claim any kind of victory. Both have advantages and disadvantages that have to be tested out with your data and your servers. Then, when all that testing is done, you get to compare how big, fast, and expensive the systems are. THAT's all that matters.

Having said that, I'd like to comment on some of your statements, as I think they represent common misunderstandings about the process. Instead of doing it here, I'm going to do it in another blog post.

Login Form

Search Backup Central

Disclaimer

Copyright (C) 2007 Alain Georgette / Copyright (C) 2006 Frantisek Hliva. All rights reserved.

Sponsored Links