Backup Central

My backup systems are Solaris, I have the "luxury" of vxfs filesystems
for my staging & database areas.

I do however back up Windows file servers, Are there any guidelines to
NTFS volumes that people would recommend ?

I thinking along the lines:

Defragmenting,
Number of streams,
LUN Virtulization tech,
Volume Sizes,
Maintaining free space,
Snapshot methods,
impact of ohh sooo many small files

Performance improvements with Advanced client / Flashbackup,
SAN Media server,
(For the adventurous) SAN client ?

For example, i currently have pain with about a dozen windows clients,
from what i can tell

we do not do defragmentaion
their LUNS live on HP EVA's sharing spindles with hosts
Free Space is minimum (~7%)
Volumes are only ~500GB
We backup with Multiple streams (Exceeds weekend (and daily)
backup window if we don't (Windows are large)


Currently backing up the windows dataservers is a pain point for me, I
am interested in hearing peoples learnings / Golden rules when it comes
to backing up large (over 500GB) NTFS Volumes.

Adam Mellor
Senior Unix Support Analyst
CF IT TECHNOLOGY SERVICES
Woodside Energy Ltd.


________________________________

From: Ed Wilts [mailto:ewilts < at > ewilts.org]
Sent: Thursday, 14 February 2008 1:17 PM
To: Mellor, Adam A.
Cc: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Defrag DSU?


On Feb 13, 2008 6:22 PM, Mellor, Adam A. <Adam.Mellor < at > woodside.com.au>
wrote:


Although I am not currently defragmenting my current DSU
volumes, I
previously had ~4TB in a single DSU under NBU 5.1 . This volume
was
running vxfs


vxfs says it all, you lucky guy. NTFS just sucks... try a 4TB DSSU on
Windows and see how much fun you have.

I do like your idea of dropping the threshold to a low value to empty it
out more frequently though.


.../Ed

--
Ed Wilts, Mounds View, MN, USA
mailto:ewilts < at > ewilts.org

NOTICE: This email and any attachments are confidential.
They may contain legally privileged information or
copyright material. You must not read, copy, use or
disclose them without authorisation. If you are not an
intended recipient, please contact us at once by return
email and then delete both messages and all attachments.


_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

On Fri, Feb 15, 2008 at 7:35 AM, Mellor, Adam A. <Adam.Mellor < at > woodside.com.au ([email]Adam.Mellor < at > woodside.com.au[/email])> wrote:
I do however back up Windows file servers, Are there any guidelines to
NTFS volumes that people would recommend ?

I thinking along the lines:

Defragmenting,
Number of streams,
LUN Virtulization tech,
Volume Sizes,
Maintaining free space,
Snapshot methods,
impact of ohh sooo many small files

Performance improvements with Advanced client / Flashbackup,
SAN Media server,
(For the adventurous) SAN client ?

For example, i currently have pain with about a dozen windows clients,
from what i can tell

we do not do defragmentaion
That's not a backup issue - the Windows admins should be doing that on a regular basis anyway. You can't worry about it. The worse the fragmentation, the worse the load on the client.


their LUNS live on HP EVA's sharing spindles with hosts
Same here. Not an issue for us. In fact, some of our DSSUs were on the same EVA as the hosts they're backing up. We've since moved most of our DSSUs off to a SATABeast.


Free Space is minimum (~7%)
Not an issue for backups unless you're using FlashBackups - there is a space requirement for the snapshots.


Volumes are only ~500GB
That's that not that bad unless you have lots of little files.


We backup with Multiple streams (Exceeds weekend (and daily) backup window if we don't (Windows are large)
So do we with a few exceptions - large FlashBackups on 32-bit Windows systems has a tendency to tip the box over so those single-stream. Because they're clusters and using virtual servers, NetBackup gives us no good way of single-streaming them so we use a dedicated DSSU with a concurrent job limit of 1 to act as the throttle point.


Currently backing up the windows dataservers is a pain point for me, I
am interested in hearing peoples learnings / Golden rules when it comes
to backing up large (over 500GB) NTFS Volumes.

If you have a large number of files, use FlashBackup. For us, it cuts the time in half. 64-bit Windows is also appears much faster at feeding the backup stream than 32-bit Windows. TOE cards seem to help but only on file servers - on SQL servers, there are rumored bugs in the driver that cause all sorts of grief (lots and lots of little transactions which is not typical for file/backup servers).

Here's an example of a recent largeish Windows backup picked at random: 525GB backed up (as reported by NetBackup), 21M files, using FlashBackup on a 64-bit Windows box. Elapsed time 5 hours and 17 minutes with a reported speed of about 29MB/sec. That was writing to DSSU on a Solaris media server (different EVA than the source volume).



We don't present any Windows volumes (other than DSSUs) greater than 1TB for two reasons: 1) the backup time gets too long, and 2) if you get into a chkdsk situation (don't forget, NTFS is non-journaling), you may need to take the volume offline for a long time...

We don't use SAN Media Servers but hope to test the SAN Client sometime...it's on the TODO list.

.../Ed

--
Ed Wilts, Mounds View, MN, USA
mailto:ewilts < at > ewilts.org ([email]ewilts < at > ewilts.org[/email])

Adam,



This is a good thread and I am going to add my comments, but am interested
to hear what others have to say.



First, we have the same issue with large, slow, Windows clients. Typically
we employ the break it up into separate datastreams/policies approach for
these. This works, but is a real headache to get the right configuration to
fit into the window.



I've also tried flashbackup, which works better if you have the bandwidth,
but ended up taking about the same amount of time as several Windows
policies. The problem in this situation was the large amount of free space
that gets backed up. I opted to go back to the several policies approach,
for now.



I am very interested in checking out the SAN client, but it currently has
limitations of disk only targets and certain HBAs. The situation I am in
does not really allow us to turn the clients into SAN Media Servers, but I
have done this in the past and I did not like it. As you probably know,
drive issues or NBU maintenance may require rebooting your Media Server,
which is a bad thing when it's a production application server. So you then
get into the no backups or take a downtime lose-lose choice.



Defragging can help, and there are several of the defrag vendors out there
that have data to support this. I don't believe that the included Windows
'diskeeper lite' version will help much, but I don't have any personal
experience to support this, only what I've heard from the list and others.



If your data is pretty static, synthetic backups may be an option (but I
don't know if I would do it!). I'm also interested in what PureDisk can do
for these clients.



I plan on testing this in a lab in the next few weeks and I'll share my
findings when I get to that point.



-Rusty



________________________________



From: veritas-bu-bounces < at > mailman.eng.auburn.edu
[mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of "Mellor,
Adam A." <Adam.Mellor < at > woodside.com.au>

Sent: Friday, February 15, 2008 7:36 AM

To: <VERITAS-BU < at > mailman.eng.auburn.edu>

Cc: Ed Wilts <ewilts < at > ewilts.org>

Subject: [Veritas-bu] Speaking of NTFS:





My backup systems are Solaris, I have the "luxury" of vxfs filesystems

for my staging & database areas.



I do however back up Windows file servers, Are there any guidelines to

NTFS volumes that people would recommend ?



I thinking along the lines:



Defragmenting,

Number of streams,

LUN Virtulization tech,

Volume Sizes,

Maintaining free space,

Snapshot methods,

impact of ohh sooo many small files



Performance improvements with Advanced client / Flashbackup,

SAN Media server,

(For the adventurous) SAN client ?



For example, i currently have pain with about a dozen windows clients,

from what i can tell



we do not do defragmentaion

their LUNS live on HP EVA's sharing spindles with hosts

Free Space is minimum (~7%)

Volumes are only ~500GB

We backup with Multiple streams (Exceeds weekend (and daily)

backup window if we don't (Windows are large)





Currently backing up the windows dataservers is a pain point for me, I

am interested in hearing peoples learnings / Golden rules when it comes

to backing up large (over 500GB) NTFS Volumes.



Adam Mellor

Senior Unix Support Analyst

CF IT TECHNOLOGY SERVICES

Woodside Energy Ltd.





________________________________



From: Ed Wilts [mailto:ewilts < at > ewilts.org]

Sent: Thursday, 14 February 2008 1:17 PM

To: Mellor, Adam A.

Cc: VERITAS-BU < at > mailman.eng.auburn.edu

Subject: Re: [Veritas-bu] Defrag DSU?





On Feb 13, 2008 6:22 PM, Mellor, Adam A.

wrote:





Although I am not currently defragmenting my current DSU

volumes, I

previously had ~4TB in a single DSU under NBU 5.1 . This volume

was

running vxfs





vxfs says it all, you lucky guy. NTFS just sucks... try a 4TB DSSU on

Windows and see how much fun you have.



I do like your idea of dropping the threshold to a low value to empty it

out more frequently though.





.../Ed



--

Ed Wilts, Mounds View, MN, USA

mailto:ewilts < at > ewilts.org



NOTICE: This email and any attachments are confidential.

They may contain legally privileged information or

copyright material. You must not read, copy, use or

disclose them without authorisation. If you are not an

intended recipient, please contact us at once by return

email and then delete both messages and all attachments.





_______________________________________________

Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu

http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu








_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Adam, all,

If your NTFS volume is over 80% full, the performance starts to
degrade. I've tested this and verified that it does happen. I didn't do
enough testing with controls to truly characterize performance, but it
can be demonstrated. At 85% full, you will notice a significant
performance decrease. From 85% to 90% the performance will drop in half!
It seems to be geometric once you hit 85%.

Defragmenting will help the NTFS filesystem performance. Be aware,
that the NTFS defrag likes to have 25% freespace. If you get up to 85%
full, the defrag may not even run. You can now set up scheduled NTFS
defrags with Win2003 - it wasn't possible without a 3rd party product
until Win2003.

Don't let the Windows guys use disk compression. Backup performance
will go straight to h***. And, guess what happens if you do a large
restore on a volume that has compression turned on? That's really fun.

Many, many small files will kill performance. So will directory
depth. Once I had a 500 GB NTFS filesystem that was taking 3 days to
backup! And, incrementals would actually take longer. I laid out the
steps we needed to run through to get it backed up. First of all, it was
over 90% full. I told them they needed to use 75% full as their goal,
including growth. When we migrated the data, we defragged it too. If I
recall correctly we could then run a backup in about 18 hours or so.
Then I set up Flashbackup using VSS. After all was said and done the
Flashbackup would run in about 3 - 4 hours. I considered 3 days to 3
hours a fairly decent performance increase. It really operates very
similar to a Flashbackup of VxFS, if you've ever done that. And if you
do defrag with Flashbackup, only defrag prior to the full backup.

If you turn on multi-streaming with Windows and do All Local Drives,
it creates one stream per drive - C:, D:, etc... If your drives are
separate disks, separate luns, that's ok. However, say the local disk
space is coming off of a locally attached SCSI array where the disks are
setup in RAID 0+1 or RAID 5, then the RAID disk is split up to create
different disks for the server. All multi-streaming will do for you in
that case is increase disk contention. If your disk is coming off of a
large array, like a DMX, Clariion, EVA or such, this is not as much an
issue, although it can be if your various luns are coming off of the
same set of spindles.

Large Windows file servers rarely get good disk I/O performance. It
has been steadily improving, but I have usually seen the network I/O
exceed the disk I/O. DB servers are the exceptions to this. Large SQL or
Oracle servers can usually generate a much faster I/O stream, everything
else begin equal.

SAN media servers? High cost that _may_ give you a performance
increase. Make sure you can read from your disk faster than your network
throughput. With tuning, a decent Windows server should be able to send
out in excess of 60 MB/s over GigE. Make sure you can read from your
disk(s) that fast before you spend the money on the SAN backup solution.

Bryan


My backup systems are Solaris, I have the "luxury" of vxfs filesystems
for my staging & database areas.

I do however back up Windows file servers, Are there any guidelines to
NTFS volumes that people would recommend ?

I thinking along the lines:

Defragmenting,
Number of streams,
LUN Virtulization tech,
Volume Sizes,
Maintaining free space,
Snapshot methods,
impact of ohh sooo many small files

Performance improvements with Advanced client / Flashbackup,
SAN Media server,
(For the adventurous) SAN client ?

For example, i currently have pain with about a dozen windows clients,
from what i can tell

we do not do defragmentaion
their LUNS live on HP EVA's sharing spindles with hosts
Free Space is minimum (~7%)
Volumes are only ~500GB
We backup with Multiple streams (Exceeds weekend (and daily)
backup window if we don't (Windows are large)


Currently backing up the windows dataservers is a pain point for me, I
am interested in hearing peoples learnings / Golden rules when it comes
to backing up large (over 500GB) NTFS Volumes.

Adam Mellor
Senior Unix Support Analyst
CF IT TECHNOLOGY SERVICES
Woodside Energy Ltd.


________________________________

From: Ed Wilts [mailto:ewilts < at > ewilts.org]
Sent: Thursday, 14 February 2008 1:17 PM
To: Mellor, Adam A.
Cc: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Defrag DSU?


On Feb 13, 2008 6:22 PM, Mellor, Adam A. <Adam.Mellor < at > woodside.com.au>
wrote:


Although I am not currently defragmenting my current DSU
volumes, I
previously had ~4TB in a single DSU under NBU 5.1 . This volume
was
running vxfs


vxfs says it all, you lucky guy. NTFS just sucks... try a 4TB DSSU on
Windows and see how much fun you have.

I do like your idea of dropping the threshold to a low value to empty it
out more frequently though.


.../Ed

_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Hi,
Im sure others will reply, but my thought....

Defragmentation will happen on most file systems. Programs like
Diskeeper will do the trick, but ensure its done out of hours. To be
honest, I have not seen that much of a performance increase in backup
speeds after a defrag. I think alot depends on whether you are using
volume compression to compress the Data and how the clients are
configured for backups. also, the type of Data on the volumes. The built
in defrag sucks in my view, so consider another alternative rather than
the "Win2k / Win2k3" version.

Hosts connected to a HP EVA are all going to be sharing spindles so
I would rule that out. Again, it comes down to the end client, how its
configured, whats being backed up.

We use SAN Media Servers for our large systems with volumes. It works
well.

What I would tend to do is look at your multiplexing and see if you can
alter to make use of more drives if poss. Also, verify how the backup
policy is configured for streams. For example if you have a Win2k3
System with 5 Drives, consider streaming the drives seperately or maybe
2 volumes into a single stream.

If these are LAN based clients, then you are not going to get good
throughput, especially if the volumes are fast. As a rule of thumb if it
takes approx 30 hours to backup a 500GB volume, a SAN Media Server could
(depending on the files) backup the system in say 12 - 14 hours. Again,
alot does tend to rely on compression, type of data, file size ect.

One other option to consider is Flash Backup. I am looking into this,
but not sure at this stage if it will really help my situation.

Faster disks (like the HP EVA) is a bonus. Turning off any extra I/O
like Anti-Virus during a backup WILL also help.

Just my views
Thanks, Simon



-----Original Message-----
From: veritas-bu-bounces < at > mailman.eng.auburn.edu
[mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of Mellor,
Adam A.
Sent: Friday, February 15, 2008 1:36 PM
To: VERITAS-BU < at > mailman.eng.auburn.edu
Cc: Ed Wilts
Subject: [Veritas-bu] Speaking of NTFS:

My backup systems are Solaris, I have the "luxury" of vxfs filesystems
for my staging & database areas.

I do however back up Windows file servers, Are there any guidelines to
NTFS volumes that people would recommend ?

I thinking along the lines:

Defragmenting,
Number of streams,
LUN Virtulization tech,
Volume Sizes,
Maintaining free space,
Snapshot methods,
impact of ohh sooo many small files

Performance improvements with Advanced client / Flashbackup,
SAN Media server,
(For the adventurous) SAN client ?

For example, i currently have pain with about a dozen windows clients,
from what i can tell

we do not do defragmentaion
their LUNS live on HP EVA's sharing spindles with hosts
Free Space is minimum (~7%)
Volumes are only ~500GB
We backup with Multiple streams (Exceeds weekend (and daily)
backup window if we don't (Windows are large)


Currently backing up the windows dataservers is a pain point for me, I
am interested in hearing peoples learnings / Golden rules when it comes
to backing up large (over 500GB) NTFS Volumes.

Adam Mellor
Senior Unix Support Analyst
CF IT TECHNOLOGY SERVICES
Woodside Energy Ltd.


________________________________

From: Ed Wilts [mailto:ewilts < at > ewilts.org]
Sent: Thursday, 14 February 2008 1:17 PM
To: Mellor, Adam A.
Cc: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Defrag DSU?


On Feb 13, 2008 6:22 PM, Mellor, Adam A. <Adam.Mellor < at > woodside.com.au>
wrote:


Although I am not currently defragmenting my current DSU
volumes, I
previously had ~4TB in a single DSU under NBU 5.1 . This volume
was
running vxfs


vxfs says it all, you lucky guy. NTFS just sucks... try a 4TB DSSU on
Windows and see how much fun you have.

I do like your idea of dropping the threshold to a low value to empty it
out more frequently though.


.../Ed

--
Ed Wilts, Mounds View, MN, USA
mailto:ewilts < at > ewilts.org

NOTICE: This email and any attachments are confidential.
They may contain legally privileged information or
copyright material. You must not read, copy, use or
disclose them without authorisation. If you are not an
intended recipient, please contact us at once by return
email and then delete both messages and all attachments.


_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

This email (including any attachments) may contain confidential and/or
privileged information or information otherwise protected from disclosure.
If you are not the intended recipient, please notify the sender
immediately, do not copy this message or any attachments and do not use it
for any purpose or disclose its content to any person, but delete this
message and any attachments from your system. Astrium disclaims any and all
liability if this email transmission was virus corrupted, altered or
falsified.
---------------------------------------------------------------------
Astrium Limited, Registered in England and Wales No. 2449259
REGISTERED OFFICE:-
Gunnels Wood Road, Stevenage, Hertfordshire, SG1 2AS, England

_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu