Backup Central

Thank you all for the information thus far.

Can any of you provide real world backup times for full tree traversal backups to TSM using NDMP? I understand my mileage may vary depending on the data and it's size as well as the hardware configuration but there must be a fairly standard throughput rate for file server data.

Lastly I'm not sure I understand why an incremental backup cannot be run on the Isilon. Please explain.

Thanks in advance,
Jeff

I don't have any personal experience with Isilon, so these answers are based on other experience with NDMP and filer devices:

Can any of you provide real world backup times for full tree traversal backups to TSM using NDMP? I understand my mileage may vary depending on the data and it's size as well as the hardware configuration but there must be a fairly standard throughput rate for file server data.

NDMP is a very old backup protocol without much smarts. It does not traverse the file tree, it dumps the entire share as a single object. The amount of time increases with the size of the share, and is affected by whether you are doing it via TCP/IP or direct to tape via fibre, the disk in the device, etc. The biggest problem is that you can only do fulls and differentials. That means you are pushing backups of the same unchanged data over and over again, and if you want to have 6 months coverage of your backup data, you have to keep 6 months of these enormous full dumps.

Lastly I'm not sure I understand why an incremental backup cannot be run on the Isilon. Please explain.

NAS devices are closed operating systems, so you can't install a TSM client on them. If you are accessing the data via CIFS shares, you can run an incremental backup by putting the TSM client on a Windows machine and backing up the data via the UNC name or mounted drive letter. (Or do the equivalent using a UNIX client for NFS shares.) The problem with that, it means that you are scanning the file tree via the CIFS share, which is much slower than scanning the file tree on a local hard drive. Then you are pulling the identified changed data across the network to the Windows machine running the TSM client, then across the network again to the TSM server. If your file tree is millions of files, it's dirt slow. But you get a true incremental TSM backup that way, and the data can be managed at the file level with normal TSM management classes/retention rules. Much better than keeping TB of NDMP dumps, but the question is how long it takes. KEEP YOUR SHARES SMALL if you want to do it that way. I have customers where we've solved the problem by running multiple proxy servers, each responsible for 1-2 shares, so that all the shares can get backed up daily. But it's a lot more work for you, than using the -snapdiff API, which is designed by Netapp to address the problem (and works well).

On 13/02/12 05:48, Prather, Wanda wrote:
I don't have any personal experience with Isilon, so these answers are based on other experience with NDMP and filer devices:

Can any of you provide real world backup times for full tree traversal backups to TSM using NDMP? I understand my mileage may vary depending on the data and it's size as well as the hardware configuration but there must be a fairly standard throughput rate for file server data.

NDMP is a very old backup protocol without much smarts. It does not traverse the file tree, it dumps the entire share as a single object. The amount of time increases with the size of the share, and is affected by whether you are doing it via TCP/IP or direct to tape via fibre, the disk in the device, etc. The biggest problem is that you can only do fulls and differentials. That means you are pushing backups of the same unchanged data over and over again, and if you want to have 6 months coverage of your backup data, you have to keep 6 months of these enormous full dumps.

Lastly I'm not sure I understand why an incremental backup cannot be run on the Isilon. Please explain.

NAS devices are closed operating systems, so you can't install a TSM client on them. If you are accessing the data via CIFS shares, you can run an incremental backup by putting the TSM client on a Windows machine and backing up the data via the UNC name or mounted drive letter. (Or do the equivalent using a UNIX client for NFS shares.) The problem with that, it means that you are scanning the file tree via the CIFS share, which is much slower than scanning the file tree on a local hard drive. Then you are pulling the identified changed data across the network to the Windows machine running the TSM client, then across the network again to the TSM server. If your file tree is millions of files, it's dirt slow. But you get a true incremental TSM backup that way, and the data can be managed at the file level with normal TSM management classes/retention rules. Much better than keeping TB of NDMP dumps, but the question is how long it takes. KEEP YOUR SHARES SMALL if


you want to do it that way. I have customers where we've solved the problem by running multiple proxy servers, each responsible for 1-2 shares, so that all the shares can get backed up daily. But it's a lot more work for you, than using the -snapdiff API, which is designed by Netapp to address the problem (and works well).
To be clear the NDMP protocol can do incrementals (depending on the
storage and it's implementation of NDMP). It is TSM that does not
support incremental NDMP.

Grant

At 01:48 PM 2/12/2012, Prather, Wanda wrote:
NAS devices are closed operating systems, so you can't install a TSM client on them.

Depends on what you define as a "NAS device". I believe (not positive) that SoNAS has an integrated TSM client in it. Some windows-based fileservers can use an on-board windows TSM client (and thus take advantage of JBB).

We don't have a lot of first-hand experience with NAS here, other than a netapp gateway (IBM N6210), but I've been reviewing a bunch of the technologies, with emphasis on how to protect them. I see the following major categorizations of how to protect (large) file servers effectively. Please feel free to comment on this, as I'm looking to refine my view.

I should say that our "protection goal" would be to have a copy of data at our medical college campus, which is ~200+ miles away and accessible via a 10Gb WAN.

1. Backup using NDMP: breaks down as data size increases, because of need for periodic full backups and lack of incremental forever. However, probably fast recovery times from tape. Most NAS products support some version of NDMP.

2. Incremental file-level backup based on Journal Scanning, saving a walk of the filesystem. Examples include Windows-based fileservers, and (I think) SONAS.

3. Incremental backup based on Snapshot differencing, again saving a walk of the filesystem. This would be NetApp, but not with MultiStore vFilers - a problem for us.

4. Asynchronous replication to a second NAS. Fast RTO, but also expensive.

Options 1-3 use TSM.
Options 2&3 run faster and are more scalable than option 1, but likely have longer RTO.
Option 4 is probably most expensive.

..Paul



--
Paul Zarnowski Ph: 607-255-4757
Manager, Storage Services Fx: 607-255-8521
719 Rhodes Hall, Ithaca, NY 14853-3801 Em: psz1 < at > cornell.edu

On 02/13/2012 11:46 AM, Paul Zarnowski wrote:


[...] I see the following major categorizations of how to protect
(large) file servers effectively. Please feel free to comment on
this, as I'm looking to refine my view.

I should say that our "protection goal" would be to have a copy of
data at our medical college campus, which is ~200+ miles away and
accessible via a 10Gb WAN.

1. Backup using NDMP: breaks down as data size increases, because of
need for periodic full backups and lack of incremental forever.
However, probably fast recovery times from tape. Most NAS products
support some version of NDMP.

2. Incremental file-level backup based on Journal Scanning, saving a
walk of the filesystem. Examples include Windows-based
fileservers, and (I think) SONAS.

3. Incremental backup based on Snapshot differencing, again saving a
walk of the filesystem. This would be NetApp, but not with
MultiStore vFilers - a problem for us.

4. Asynchronous replication to a second NAS. Fast RTO, but also
expensive.

Options 1-3 use TSM.
Options 2&3 run faster and are more scalable than option 1, but likely have longer RTO.
Option 4 is probably most expensive.


I'm going to assume

0. Conventional TSM backup via a proxy node. Requires keeping shares
quite small (no more than a few TB), if you want anything approaching
daily incrementals. Bottleneck appears to be metadata scan, walk of
tree.

I mention that because Isilon (and others?) are starting to offer
options that let you keep metadata on SSD, which may change the
maximum size for reasonable conventional incrementals. We're
expecting to do a POC of Isilon before too long, I intend to examine
that.


-----


As for comment, I want to amplify what you said about RTO: Your option
4 is in a completely different universe from the 0-3. For all of 0-3,
the DR critical path includes an acquisition cycle. I'm not sure our
heirarchy could get a PO out in a week if their hair were on
fire... yours may be better.

Option 4 has nearly immediate recovery, in degraded performance. So,
it's more expensive, but it moves you to a totally different recovery
mode.

Our thinking abount isilon backups includes a remote "Near-line" unit,
that has relatively large, slow, cheap drives. It would be a
replication target for several other units, giving it good efficiency
of scale.



- Allen S. Rout

Allen,

Thanks for your response.

I did not include your option 0, because I was thinking "large file servers".

Your point about option 0 being potentially more viable if metadata is kept on SSD is interesting. Does anyone have any first-hand experience with this that they can share?

Option 4 is admittedly expensive and in a different RTO class. I included it because in some situations, it may indeed be the only viable option for offsite protection. And as you say, if the replica can be on slower, cheaper disk, that brings the cost down some.

..Paul

At 09:26 AM 2/14/2012, Allen S. Rout wrote:
On 02/13/2012 11:46 AM, Paul Zarnowski wrote:


[...] I see the following major categorizations of how to protect
(large) file servers effectively. Please feel free to comment on
this, as I'm looking to refine my view.

I should say that our "protection goal" would be to have a copy of
data at our medical college campus, which is ~200+ miles away and
accessible via a 10Gb WAN.

1. Backup using NDMP: breaks down as data size increases, because of
need for periodic full backups and lack of incremental forever.
However, probably fast recovery times from tape. Most NAS products
support some version of NDMP.

2. Incremental file-level backup based on Journal Scanning, saving a
walk of the filesystem. Examples include Windows-based
fileservers, and (I think) SONAS.

3. Incremental backup based on Snapshot differencing, again saving a
walk of the filesystem. This would be NetApp, but not with
MultiStore vFilers - a problem for us.

4. Asynchronous replication to a second NAS. Fast RTO, but also
expensive.

Options 1-3 use TSM.
Options 2&3 run faster and are more scalable than option 1, but likely have longer RTO.
Option 4 is probably most expensive.


I'm going to assume

0. Conventional TSM backup via a proxy node. Requires keeping shares
quite small (no more than a few TB), if you want anything approaching
daily incrementals. Bottleneck appears to be metadata scan, walk of
tree.

I mention that because Isilon (and others?) are starting to offer
options that let you keep metadata on SSD, which may change the
maximum size for reasonable conventional incrementals. We're
expecting to do a POC of Isilon before too long, I intend to examine
that.


-----


As for comment, I want to amplify what you said about RTO: Your option
4 is in a completely different universe from the 0-3. For all of 0-3,
the DR critical path includes an acquisition cycle. I'm not sure our
heirarchy could get a PO out in a week if their hair were on
fire... yours may be better.

Option 4 has nearly immediate recovery, in degraded performance. So,
it's more expensive, but it moves you to a totally different recovery
mode.

Our thinking abount isilon backups includes a remote "Near-line" unit,
that has relatively large, slow, cheap drives. It would be a
replication target for several other units, giving it good efficiency
of scale.



- Allen S. Rout


--
Paul Zarnowski Ph: 607-255-4757
Manager, Storage Services Fx: 607-255-8521
719 Rhodes Hall, Ithaca, NY 14853-3801 Em: psz1 < at > cornell.edu