My backup/restore runs slowly. Where do I start troubleshooting?

This Wiki is brought to you by Backup Central, where you can find the Mr. Backup Blog, Forums, and a mailing list for each forum!
Backup FAQs	Service Providers	Backup Software	Backup Hardware	Backup Book Wiki	Free Stuff	Miscellaneous

A. Eric C. Schwartz wrote (17 Sept 1999): Slow backups, below are some things to look at and consider: What's the speed of network link between the client and Networker server? faster is better when you're doing Networker backups, are network is currently at 100mb How many hops between the client and the Networker server? Too many hops slows down the data transfers. Is the data going through routers? Are the routers overloaded? Not good if they are.

How "clean" is your network data path? To many errors will slow down the data transfer rates. Not sure, maybe do some large file transfers between the client and server and see what kind of throughput you get. analyze the results.

What's CPU processor/memory utilization and processor queue length of the client and Networker server system when backups are running? If either or both are overloaded the amount of data being sent to the tape drive is reduced What type of DLT drives are you using. native speeds are for the DLT7000 5mbs, DLT8000 6mbs and DLT4000 1.5 mbs if your data's compressible, you'll see higher transfer rates.

What's the client data like, is it fragmented? probably not, but you could check...

Albert Moser added in response to the same question:

Getting backup throughput of 200 - 300 Kb/s on a 100 Mb LAN typically means you have a network problem. When I see these slow numbers on a Networker client, I check the NIC settings on the client and the settings of the switch port of this particular client.

If they don't match, you will get lots of collisions and therefore terrible network throughput. Additionally I make sure NOT to autonegotiate between the NIC and switch. I prefer fixed settings for best network performance. I've also seen switches that do not work very well with Full duplex, even they should. Half Duplex on these switches gave me better through put.

When this question was asked again, Marc G. Fournier (23 Mar 2000) wrote: Way back, some of you might recall that I too got a DLT7000 and had atrocious backup speeds compared to what I was hoping to get with it. The problem, in retrospect, was very stupid:

I was using the wrong device At least under Solaris, each /dev/rmt/* device has a different meaning, and one of them deals with how compression is handled. On my machine, I needed to use /dev/rmt/0ubn in order for the OS to send the proper codes to the

DLT7000 to do its "high capacity/compression" mode ... Keith A. Clay shared with the list (22 June 2000) a very methodical plan of investigation he undertook when his backups ran too slow, & the results he found: After great advice from the list, here is what I have tried:

1. ran uasm -s -n -v /my_partition -- took 1 hour and 40 minutes to run on 18gigs
2. ran save with the '-i' to ignore directive files -- backup time was no better
3. ran save with '-i' and 'E' for estimate -- had to kill, was running more than a day
4. ran find piped to cpio -- took 8 hours to make copies. Then ran save on the directories with the cpio archives. Took 45 minutes to backup 18 gig.

After isolating the problem to the partition itself, we found that in updating the mailstore, there were thousands (if not tens of thousands) of links that were unresolvable (this is solaris 7). We are right now assuming that these links are causing the increased save time. Later this evening, we are going to take the server down and remove this links. I am not sure if this will work, but it seems the logical place to begin.

Also Mike Myer's contribution (29 Mar 2000), although Solaris specific, suggests some useful tricks: To eliminate your tape drive and cabling you should check your raw tape drive throughput. Under Unix you can do something like the following:

dd if=/dev/zero of=/dev/rmt/0cbn bs=128k

where bs is set to the value you have tuned the blocking size of your device. Then use the "iostat -x 5" command (the -x is necessary under Solaris to get "extended" stats (eg. stats on ALL devices) -- I'm not sure what the exact command is on other platforms). If you see throughput below the native throughput of your tape device you have a problem. You should see something closer to twice the native throughput because of compression. I have a suspicion that it doesn't go higher than about twice because of the little CPU that's doing the compression on the drive (because a stream of zeros should be HIGHLY compressible).

Another trick to mention that's specific to NetWorker -- if you do a "mt -f /dev/rmt/?cbn fsf 2" before doing your testing on a tape it will protect the NetWorker label on the tape so you don't have to either find a "real" scratch tape or relabel when you're done.

From experience I would start with changing the cable out (assuming you have your st.conf file tuned properly under Solaris and you're using the right tape device)