Backup Central

Hi all, long time lurker first post.

So my backup time window is being pushed to the upper limits. I would normally shrug this off and look at reducing data backed up or look into commissioning another backup server, however it led me to notice backup speeds. There is a fairly wide range - from 4000 MB/min to 100 MB/min. The majority sit just below the 1000MB/min mark.
The quick jobs are my Hyper-V servers and Exchange mailboxes.

All our servers backup to disk (a SCSII connected RAID array of SATA drives) first then replicate to tape in BE 2010 (all latest updates and latest agents on servers). All servers have Gigabit network cards and switch ports - have looked and not found any duplex mismatches or anything of the like on the network (not me but the CCIE guy in the office - he assures me all is well on the network).

I am concerned about the slow servers and would ideally like to have everything somewhere around the 3000 MB/min mark going to disk.

My question to you, the biggest backup enthusiasts I can find, is how would you go about troubleshooting?

Also - defragging the slow servers had no impact. The slowness does not seem to be centered around any particular time of day or night. Nor does the number of running jobs especially impact the slow servers speed.

Oh Right,

The majority of our servers are Windows Server 2003R2 and 2008. We are primaraly windows based. The slowest systems seem have very little in common. One is a database/application server, one is a domain controller, another just monitors websites.
I'm stumped.

We just switched from 12.5 to 2010; the 12.5 was running on 2003R2 (32bit) and the 2010 is running on 2008R2 so we were hoping to get some better performance, but still have various speed issues. The best I can deduce is that the servers have different data types and there are rules/exceptions for different servers which may be affecting the throughput. On the 12.5 we had a 2GB NIC team and on the 2010 we have a 4GB NIC team (Link aggr) but some of the servers, particularly our file server, have not really gone up or down in throughput. However, our file server has quite a few exception rules which I am taking a look at to see if we can reduce that. Also we have a good deal of non-english (Japanese) code character and not really sure if that is affecting anything. I have used Advanced Open file option on our Notes (v8.5) server and that sped up things quite a bit (300MB/Min to ~2500MB/Min)

Hmm, Interesting.

I had always steered clear of AOFO, being told by my predecessor that it was flakey and slowed things down. Might be worth looking into.
Thanks for the reply.

Gruk

I was also told the same thing - avoid AOFO at all costs........... But so far with 2010 I don't see it killing my backups, and in the case of Lotus Notes Domino server, actually sped the backup throughput to levels we've never seen before. I'm not saying AOFO does that with all backups, as some of the testing I've done shows no difference at all. That's where I get to head scratching: why some and not others??

To add onto everything, the research I did also did not support using AOFO for Notes Domino, in fact, Symantec has a KB dedicated to the "PROPER" way to backup a Notes server and it only includes using the Notes agent....... imagine that

AOFO when using VSS and not VSP is a good thing. This is for all Win2k3 and higher servers. It's a necessity to use it for Exch/SQL/AD/etc.

When backing up to disk, the benefit in BE is that you can send multiple jobs to disk simultaneously. Leverage what disk can do well and that is random reads and writes. So you should look to send more, but smaller jobs to your B2D, to shrink your backup windows.

For your Windows 2003 hosts, more than likely you've lost a bit of performance due to how your disks are configured and partitioned. you need to align the partitions to improve IOP's and performance. This goes for both exch/SQL and any Windows volume. To do it to a boot volume takes some extra steps, but basically any D: or greater drive, it should be done to. I typically see some 10-30% more performance in IOP's doing that on Win2k/Win2k3 OS's. Win2k8 doesnt need to do this.

For your BE media server, format the volume that your B2D resides using the largest possible allocation unit size. 4 or 8k if possible. That makes a tremendous difference in write speeds of the BKF files.

Again, create smaller selection lists. So if you had 20 servers, you should have maybe 4 different jobs with 5 servers in each, all set to send to disk simultaneously. You also always should separate out database backups from filesystem backups, and use VSS on those.

Oh wow, thanks for the response, looks quite promising and has given me a lot to play with.Would you suggest diskpart.exe for aligning disks or am I getting confused.
I'll have to do some manual reading to see what I can get out of AOFO and how best to implement it.

Time to test! I'll let you all know how things tun out.

So here's what I have done so far:

I seperated all my jobs into slow and fast and configured BE so 10 slow jobs run at the same time, and two fast run at the same time. I previously had six jobs going to disk at once. Since making these changes, all of my B2D jobs are now completed by about 7am - an improvement of about 8 - 9 hours! This leaves the tape drive all day to read from the disk uncontested.

I still have piles of slow jobs going to disk, I have just made it so they all run at once so they don't slow create too much of a bottleneck.

I do notice a variety of speeds with duplicating to tape however. Is this due to types of files being duplicated? Is there a way to tune this?

Cheers!

Gruk

You can tune the tape device, but it highly depends on the tape technology being used. I am only familiar with LTO3 and greater tuning methodologies, and those typically can have changes made to its device properties to improve speeds.

As an example, read this article:
https://www-secure.symantec.com/connect/articles/tuning-my-lto4-tape-drive

One more thing to add, It's a good idea to regularly defragment your B2D locations. While Windows defrag is okay, the 3rd party products are much better... This will have a dramatic effect on speeds over time as B2D will heavily fragment disks as data is expired and overwritten.

Mitch, thank you. I hadn't checked in a long while because it was never an issue in the past, but fragmentation on the B2D drives was around 97%. I have since formatted these drives and changed my setup to reduce fragmentation and my speeds are looking way better.

Previously I had hit good speeds for about a week following a format and noticed that speeds to tape were degrading rapidly. Now with less fragmentation my to tape speeds are approaching the theoretical maximum.

You guys are a handy bunch to have around <3 thanks!

Peetew - the network is sweet, everything can connect directly to the backup server on one of it's multiple interfaces. Thanks!