Backup Central

Hi,
What would likely be the de-dupe ratio if tsm clients do archive processing daily (file level, no tdps) with encryption enabled?

Thanks.

As with all questions like this, the answer is "it depends".
It depends on the make-up of your data (# of DB full dumps, % of DB
dumps to filesystem data, % of change on the client, etc)
It depends on the vendor of DeDupe you are using.

FWIW, I am about to replace a 100TB of LTO tape with a DataDomain 560
dedupe box starting next week. Once the migration from tape to disk is
complete, I will be reporting what I saw in my environment. The DD folks
are saying that the worst case scenario will be a 7X reduction (i.e.
70TB of data squeezed into a 10TB DataDomain appliance). We shall see.

Ben


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of
lamont
Sent: Wednesday, January 23, 2008 1:29 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+----------------------------------------------------------------------
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------


The Blue Cross of Idaho Email Firewall Server made the following annotations:
------------------------------------------------------------------------------
*Confidentiality Notice:

This E-Mail is intended only for the use of the individual
or entity to which it is addressed and may contain
information that is privileged, confidential and exempt
from disclosure under applicable law. If you have received
this communication in error, please do not distribute, and
delete the original message.

Thank you for your compliance.

You may contact us at:
Blue Cross of Idaho
3000 E. Pine Ave.
Meridian, Idaho 83642
1.208.345.4550

==============================================================================

Encryption might have a DRAMATIC effect, completely eliminating the
benefits of either deduplication or compression. I predict 1:1. i.e. NO
savings for dedupliaction, with TSM client encryption.

This is why encryption at the tape drive is a very popular option with
LTO4. You can both encrypt and compress at the same time.

Roger Deschner University of Illinois at Chicago rogerd < at > uic.edu
Academic Computing & Communications Center


On Wed, 23 Jan 2008, Ben Bullock wrote:

As with all questions like this, the answer is "it depends".
It depends on the make-up of your data (# of DB full dumps, % of DB
dumps to filesystem data, % of change on the client, etc)
It depends on the vendor of DeDupe you are using.

FWIW, I am about to replace a 100TB of LTO tape with a DataDomain 560
dedupe box starting next week. Once the migration from tape to disk is
complete, I will be reporting what I saw in my environment. The DD folks
are saying that the worst case scenario will be a 7X reduction (i.e.
70TB of data squeezed into a 10TB DataDomain appliance). We shall see.

Ben


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of
lamont
Sent: Wednesday, January 23, 2008 1:29 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+----------------------------------------------------------------------
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------


The Blue Cross of Idaho Email Firewall Server made the following annotations:
------------------------------------------------------------------------------
*Confidentiality Notice:

This E-Mail is intended only for the use of the individual
or entity to which it is addressed and may contain
information that is privileged, confidential and exempt
from disclosure under applicable law. If you have received
this communication in error, please do not distribute, and
delete the original message.

Thank you for your compliance.

You may contact us at:
Blue Cross of Idaho
3000 E. Pine Ave.
Meridian, Idaho 83642
1.208.345.4550

==============================================================================

Oooh, what a great question!
I'd guess if client encryption is on and working, the dedup ratio should be
about 1:1; because the data should never encrypt the same way twice.


On 1/23/08, lamont <tsm-forum < at > backupcentral.com> wrote:

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+----------------------------------------------------------------------
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------

I agree about client encryption wrecking dedup ratios.

FWIW however, if you turn on both COMPRESSION and ENCRYPTION on the client,
the client is also smart enough to compress first, then encrypt, so you get
the compression benefits.

However, that of course takes a lot of cycles on the client, and can really
slow down restores. Outboard compression/encryption in the hardware is
definitely superior.


On 1/23/08, Roger Deschner <rogerd < at > uic.edu> wrote:

Encryption might have a DRAMATIC effect, completely eliminating the
benefits of either deduplication or compression. I predict 1:1. i.e. NO
savings for dedupliaction, with TSM client encryption.

This is why encryption at the tape drive is a very popular option with
LTO4. You can both encrypt and compress at the same time.

Roger Deschner University of Illinois at Chicago rogerd < at > uic.edu
Academic Computing & Communications Center


On Wed, 23 Jan 2008, Ben Bullock wrote:

As with all questions like this, the answer is "it depends".
It depends on the make-up of your data (# of DB full dumps, % of DB
dumps to filesystem data, % of change on the client, etc)
It depends on the vendor of DeDupe you are using.

FWIW, I am about to replace a 100TB of LTO tape with a DataDomain 560
dedupe box starting next week. Once the migration from tape to disk is
complete, I will be reporting what I saw in my environment. The DD folks
are saying that the worst case scenario will be a 7X reduction (i.e.
70TB of data squeezed into a 10TB DataDomain appliance). We shall see.

Ben


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of
lamont
Sent: Wednesday, January 23, 2008 1:29 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+----------------------------------------------------------------------
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------


The Blue Cross of Idaho Email Firewall Server made the following
annotations:

------------------------------------------------------------------------------
*Confidentiality Notice:

This E-Mail is intended only for the use of the individual
or entity to which it is addressed and may contain
information that is privileged, confidential and exempt
from disclosure under applicable law. If you have received
this communication in error, please do not distribute, and
delete the original message.

Thank you for your compliance.

You may contact us at:
Blue Cross of Idaho
3000 E. Pine Ave.
Meridian, Idaho 83642
1.208.345.4550


==============================================================================

True as well as any files that are already "Compressed" We have SQL DB's
doing Flat File Dumps to Disk with compression and we see 1.7:1 Ick.....
Also TDP RMAN backups can use Files per set function which if set to
more than 1 RMAN will "multiplex" each file set differently so you see
different data every time. We have our RMAN set to files per set =1
then the DBA's run multiple channels so we see 20:1 of course our DBA's
do fulls daily ....

We've even forced Compress = No in a Server Side Client option set,
which only applies to File System backups, the compression statement
does not apply to the TDP's as far as I know.

Also do what you can to have Like Data go to the same dedupe devices
(assuming you have more than one). Example Oracle Prod / Non-Prod with
their associated OS's go to the Same Dedupe stgpoool, Exchange etc...

Data DeDupe can be cool, but if you do not pay attention your data types
you can ruin a good thing.

I Cant wait to see how the newer Dedupe engines that are coming out that
perform the DeDupe process "Out Of Band" compares to the Inband DeDupe
methodology. Of course the Inbound Devices dedupes as data comes in
which can affect Backup Performance, (just add more widgets) but it will
be interesting to see how "Out of Band" dedupe methodology will perform
if you "get behind" (i.e. Days one Backup Data is still being DeDuped
while your are taking in Day 2's Backup data, then you add in Backup
Stgpool, Reclamation etc that will force the dedupe engine to re-dupe /
re-factor the data everytime the data is read.....

There's been many Dedupe Threads in this user list, you could almost
write a VTL - DeDupe Best Practice Guide.



-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of
Wanda Prather
Sent: Wednesday, January 23, 2008 9:42 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: Re: [ADSM-L] Data Deduplication

Oooh, what a great question!
I'd guess if client encryption is on and working, the dedup ratio should
be about 1:1; because the data should never encrypt the same way twice.


On 1/23/08, lamont <tsm-forum < at > backupcentral.com> wrote:

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+---------------------------------------------------------------------
+-
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+---------------------------------------------------------------------
+-



This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity to
which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

Hmm, I was going to say I'd expect almost none, because the eencryption
wouldn't generate the same data each time through.

But maybe It depends on encyption scheme, on how keys are managed (I would
expect the same data to encrypt the same way if the same keys are used -
although I am no cryptologist), on the level at which the data is
'collated' - changed block, whole files, etc.. etc.. and how the de-dupe
algorithm of choice does it's thing.


Matt.



Internet
rogerd < at > UIC.EDU
To
ADSM-L
Sent by: ADSM-L < at > VM.MARIST.EDU cc

23/01/2008 15:47 Subject
Re: [ADSM-L] Data Deduplication

Please respond to
ADSM-L < at > VM.MARIST.EDU










As with all questions like this, the answer is "it depends".
It depends on the make-up of your data (# of DB full dumps, % of DB
dumps to filesystem data, % of change on the client, etc)
It depends on the vendor of DeDupe you are using.

FWIW, I am about to replace a 100TB of LTO tape with a DataDomain 560
dedupe box starting next week. Once the migration from tape to disk is
complete, I will be reporting what I saw in my environment. The DD folks
are saying that the worst case scenario will be a 7X reduction (i.e.
70TB of data squeezed into a 10TB DataDomain appliance). We shall see.

Ben


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of
lamont
Sent: Wednesday, January 23, 2008 1:29 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+----------------------------------------------------------------------
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------


The Blue Cross of Idaho Email Firewall Server made the following
annotations:
------------------------------------------------------------------------------

*Confidentiality Notice:

This E-Mail is intended only for the use of the individual
or entity to which it is addressed and may contain
information that is privileged, confidential and exempt
from disclosure under applicable law. If you have received
this communication in error, please do not distribute, and
delete the original message.

Thank you for your compliance.

You may contact us at:
Blue Cross of Idaho
3000 E. Pine Ave.
Meridian, Idaho 83642
1.208.345.4550

==============================================================================




This message and any attachments (the "message") is
intended solely for the addressees and is confidential.
If you receive this message in error, please delete it and
immediately notify the sender. Any use not in accord with
its purpose, any dissemination or disclosure, either whole
or partial, is prohibited except formal approval. The internet
can not guarantee the integrity of this message.
BNP PARIBAS (and its subsidiaries) shall (will) not
therefore be liable for the message if modified.
Do not print this message unless it is necessary,
consider the environment.

---------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le
"message") sont etablis a l'intention exclusive de ses
destinataires et sont confidentiels. Si vous recevez ce
message par erreur, merci de le detruire et d'en avertir
immediatement l'expediteur. Toute utilisation de ce
message non conforme a sa destination, toute diffusion
ou toute publication, totale ou partielle, est interdite, sauf
autorisation expresse. L'internet ne permettant pas
d'assurer l'integrite de ce message, BNP PARIBAS (et ses
filiales) decline(nt) toute responsabilite au titre de ce
message, dans l'hypothese ou il aurait ete modifie.
N'imprimez ce message que si necessaire,
pensez a l'environnement.

The other posters are correct. You will get 1:1. Dedupe works by
finding patterns. There are no patterns in encrypted data.

One question would be why would you do that? Most people are encrypting
data as it leaves their site. The best way to do that is hardware
encryption (tape drive or SAN-based). Do that on the other side of your
dedupe box and before it goes to tape -- not at the client -- and you'll
have no issues with dedupe.

---
W. Curtis Preston
Backup Blog < at > www.backupcentral.com
VP Data Protection, GlassHouse Technologies

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of
lamont
Sent: Wednesday, January 23, 2008 12:29 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+----------------------------------------------------------------------
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------

Hi Curtis,
Unfortunately, this was already the case when I came, client encryption is the only option and the tapes are needed to be sent to offsite.
I think we need to consider this - enabling/disabling client encryption and see how - in the test case on the upcoming POC with a de-dupe vendor.

Thanks.

The other posters are correct. You will get 1:1. Dedupe works by
finding patterns. There are no patterns in encrypted data.

One question would be why would you do that? Most people are encrypting
data as it leaves their site. The best way to do that is hardware
encryption (tape drive or SAN-based). Do that on the other side of your
dedupe box and before it goes to tape -- not at the client -- and you'll
have no issues with dedupe.

---
W. Curtis Preston
Backup Blog < at > www.backupcentral.com
VP Data Protection, GlassHouse Technologies

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of
lamont
Sent: Wednesday, January 23, 2008 12:29 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+----------------------------------------------------------------------
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------

Is it possible to change from Client Encrypt to Tape Device Encrypt?
(i.e. LTO4 / 3592 etc) The you're encrypting your offsite but your
onsite is now getting better "Factoring" compression ratios.

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of
lamont
Sent: Wednesday, January 23, 2008 11:18 PM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi Curtis,
Unfortunately, this was already the case when I came, client encryption
is the only option and the tapes are needed to be sent to offsite.
I think we need to consider this - enabling/disabling client encryption
and see how - in the test case on the upcoming POC with a de-dupe
vendor.

Thanks.


cpreston wrote:
The other posters are correct. You will get 1:1. Dedupe works by
finding patterns. There are no patterns in encrypted data.

One question would be why would you do that? Most people are
encrypting data as it leaves their site. The best way to do that is
hardware encryption (tape drive or SAN-based). Do that on the other
side of your dedupe box and before it goes to tape -- not at the
client -- and you'll have no issues with dedupe.

---
W. Curtis Preston
Backup Blog < at > www.backupcentral.com
VP Data Protection, GlassHouse Technologies

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf
Of lamont
Sent: Wednesday, January 23, 2008 12:29 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+---------------------------------------------------------------------
+-
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+---------------------------------------------------------------------
+-


+----------------------------------------------------------------------
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity to
which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

Yes and no.

All the data backed up by a client in encrypted format stays encrypted, it
can only be decrypted by the original client (or a client with the original
encryption key). If you turn on encryption on the drives that's OK, if
client-encrypted data gets sent there via MOVE DATA, reclaim, or BACKUP
STGPOOL, it will work fine. The drives apply their own encryption
algorithm, but it's transparent to everybody. The drives won't be able to
compress the client-encrypted data, but you're no worse off than you are
now.

But if you turn on tape encryption, you can turn off client encryption.
Then the drives will compress first, then encrypt, so you get
good compression ratios for the data. If you send your onsite data to a
de-dup VTL and your TSM copy tapes to encrypting drives, you will get the
benefits of dedup in the VTL and the benefits of compression on the
drives. As older data expires, your overall compression ratio will get
better over time.





On 1/24/08, Hart, Charles A <charles_hart < at > uhc.com> wrote:

Is it possible to change from Client Encrypt to Tape Device Encrypt?
(i.e. LTO4 / 3592 etc) The you're encrypting your offsite but your
onsite is now getting better "Factoring" compression ratios.

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of
lamont
Sent: Wednesday, January 23, 2008 11:18 PM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi Curtis,
Unfortunately, this was already the case when I came, client encryption
is the only option and the tapes are needed to be sent to offsite.
I think we need to consider this - enabling/disabling client encryption
and see how - in the test case on the upcoming POC with a de-dupe
vendor.

Thanks.


cpreston wrote:
The other posters are correct. You will get 1:1. Dedupe works by
finding patterns. There are no patterns in encrypted data.

One question would be why would you do that? Most people are
encrypting data as it leaves their site. The best way to do that is
hardware encryption (tape drive or SAN-based). Do that on the other
side of your dedupe box and before it goes to tape -- not at the
client -- and you'll have no issues with dedupe.

---
W. Curtis Preston
Backup Blog < at > www.backupcentral.com
VP Data Protection, GlassHouse Technologies

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf
Of lamont
Sent: Wednesday, January 23, 2008 12:29 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] Data Deduplication

Hi,
What would likely be the de-dupe ratio if tsm clients do archive
processing daily (file level, no tdps) with encryption enabled?

Thanks.

+---------------------------------------------------------------------
+-
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+---------------------------------------------------------------------
+-


+----------------------------------------------------------------------
|This was sent by alancb < at > gmail.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity to
which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.