The opinions contained within this website, it's blog(s), forums, and Wikis, are those of the original poster and do not represent the position of my (or any other) employer.
The truth about the dedupe vendors
Written by W. Curtis Preston
Thursday, 31 July 2008
What do you want to know about your favorite dedupe vendor? What do you think you know that you don't? What do you have no idea about? Which vendors' claims are exaggerated?
This isn't going to be easy -- trust me. This is the largest growing segment of the storage industry, and the amount of FUD and exaggerated claims the vendors are throwing out is crazy! I spend a lot of time trying to looking at all of this, and I'm constantly trying to figure out the difference between what they're saying to me and what they're not saying to me.
If you're an end user considering the purchase of a dedupe solution, I think you'll find this series extremely useful. Remember I do have a day job. Imagine how helpful it would be to have me or some of my coworkers on your side when you're trying to pick your next backup product or maintain your current one. (Hint, hint.) Feel free to click the Contact Curtis link to find out more.
If you're a vendor and you read something that you believe is incorrect, you've got two choices. The first is to hit the Contact Curtis button in the menu. We can have a private conversation that way, which is often a lot easier than a public one. If I have anything in an article that's factually wrong -- I'll change it immediately. If we can't come to an agreement, then you're welcome to try the second option, which is clicking the Comment button. I won't censor your comments, but I might make my own comment in response. ;) Please don't use the unofficial, non-existent third option which is to contact my employer and try to pressure them to get me to change what I wrote. My relationship with them is too good for that to work, and that method only gets everybody's hackles up.
Do you want this kind of information? Consider this a teaser trailer. Watch this space.
Which vendors say they have block level data de-duplication?
W. Curtis Preston
- None of them
|2008-08-01 13:59:08
I believe what you're asking is which vendors do sub-file level deduplication. We don't say "block-level dedupe," as it suggests that they're looking at the data block-by-block. They're typically looking at data chunk-by-chunk, where a chunk is something smaller than a file but bigger than a block.
Troy Schuler
- re:
|2008-08-02 19:48:44
sabujp wrote:
Which vendors say they have block level data de-duplication?
A better question is to ask who (and which product) does fixed length and who does variable length segment deduplication.
W. Curtis Preston
- Again...
|2008-08-04 12:45:24
That question will only apply to those who do segment-based deduplication. A few of them do something completely different.
Jaspreet
- Fixed block Vs valiable block
|2008-11-13 10:05:10
tschuler wrote:
A better question is to ask who (and which product) does fixed length and who does variable length segment deduplication.
This is valid for only source de-dup but totally agree. Variable block length de-dup -
1. Has a better possibility of fining duplicate content between two different files.
2. takes care of shifing of blocks within a file which is very common in large files.
W. Curtis Preston
- Where do you get your information?
|2008-11-13 12:29:33
It absolutely is valid for target dedupe as well.
Jaspreet
- re: Where do you get your information?
|2008-11-23 09:58:41
cpreston wrote:
It absolutely is valid for target dedupe as well.
Curtis, in target de-duplication the disk or the appliance gets just plain blocks, it does not have any idea about files i.e. can not make out if the blocks are from same file or other.
Since the relevance is lost the variable chunk size technique cannot be applied.
With the source based de-dup the agent has a good idea about the file structure and can do a lot of tricks.
Agents like Puredisk (current implementation) are a bit dump and do a bad job here, but avamar should be good.
I used to work for Veritas, but now work for a startup.
