Backup Central

I seem to be having some issues with amanda getting data off my drives in a reasonable time.
using dd reading off my drives and writing to tape I get around 120mbs to the LTO4 drive.
When using amanda however this number fluctuates quite a bit. At times I have seen it write to tape
at around 110mbs but it's usually around 40 or so.

I have tried several different configuration options and none seem to really improve the situation.

I am using amanda to do local backups.
I have a bunch of large (+2TB) netapp dumps on a 24 disk 16T raid5 array connected to a 3Ware.
I have also configured a 14 disk netapp shelf with 15k scsi drives striped with software raid connected via a 2GB fcal to use as
a scratch disk to break up dumps or to use as a holding disk.


Can anyone recommend amanda config options that would work with my setup?
So far it seems that best results are achieved without using the holding disk.
I have tried setting the tape_splitsize to 4Gb and fallback_splitsize to 8Gb with no
real change in performance. I have 12GB of memory.


Heres the relevant bits I have so far:

netusage 100 mbps
device_output_buffer_size 1 Gb
tapedev "tape:/dev/nst0"
device_property "BLOCK_SIZE" "1024k"

holdingdisk hd1 {
comment "main holding disk"
directory "/scratch/"
use -100 Mb
chunksize 80Gb
}

define tapetype LTO4 {
comment "Dell LTO4 800Gb - Compression Off"
length 802816 mbytes
filemark 0 kbytes
speed 120 mbps
blocksize 1024 kbytes
}

define dumptype user-tar-span {
#holdingdisk no
root-tar
tape_splitsize 80 Gb
split_diskbuffer "/scratch/"
#fallback_splitsize 8 Gb
priority medium
}


If I need to provide anymore info let me know.

Thanks,
William

Hi William,

Could you tell us please:
- What tape library are you using? There are various LTO-4 tapetype definitions depending on your tape library.
- What is the size of the holding disk "hd1"? How much free space is available on it?

I'd try increasing the tape_splitsize to 200Gb (assuming you have this amount of free space available on the /scratch filesystem):

define dumptype user-tar-span {
#holdingdisk no
root-tar
tape_splitsize 200 Gb
split_diskbuffer "/scratch/"
#fallback_splitsize 8 Gb
priority medium
}

Best,
Valeriu

On Mon, Feb 22, 2010 at 10:11:01AM -0800, William Taylor wrote:
I seem to be having some issues with amanda getting data off my drives in a reasonable time.
using dd reading off my drives and writing to tape I get around 120mbs to the LTO4 drive.
When using amanda however this number fluctuates quite a bit. At times I have seen it write to tape
at around 110mbs but it's usually around 40 or so.

I have tried several different configuration options and none seem to really improve the situation.

I am using amanda to do local backups.
I have a bunch of large (+2TB) netapp dumps on a 24 disk 16T raid5 array connected to a 3Ware.
I have also configured a 14 disk netapp shelf with 15k scsi drives striped with software raid connected via a 2GB fcal to use as
a scratch disk to break up dumps or to use as a holding disk.


Can anyone recommend amanda config options that would work with my setup?
So far it seems that best results are achieved without using the holding disk.
I have tried setting the tape_splitsize to 4Gb and fallback_splitsize to 8Gb with no
real change in performance. I have 12GB of memory.


Heres the relevant bits I have so far:

netusage 100 mbps
device_output_buffer_size 1 Gb
tapedev "tape:/dev/nst0"
device_property "BLOCK_SIZE" "1024k"

holdingdisk hd1 {
comment "main holding disk"
directory "/scratch/"
use -100 Mb
chunksize 80Gb
}

define tapetype LTO4 {
comment "Dell LTO4 800Gb - Compression Off"
length 802816 mbytes
filemark 0 kbytes
speed 120 mbps
blocksize 1024 kbytes
}

define dumptype user-tar-span {
#holdingdisk no
root-tar
tape_splitsize 80 Gb
split_diskbuffer "/scratch/"
#fallback_splitsize 8 Gb
priority medium
}


If I need to provide anymore info let me know.

Thanks,
William

William,

Please don't forget to copy your replies to the 'amanda-users' list.

Valeriu

On Mon, Feb 22, 2010 at 12:17:25PM -0800, William Taylor wrote:
It's a Dell PowerVault TL4000 connected via SAS
The stripped holding or split_diskbuffer disk is 1.9T

Il try to increase the split size and see what it does. It looked like to me before that
it would write the split file to disk but I never say reads coming of the disk. Like it was writing
it plus streaming it to tape at the same time. Is this normal?

On Feb 22, 2010, at 10:47 AM, Valeriu Mutu wrote:

Hi William,

Could you tell us please:
- What tape library are you using? There are various LTO-4 tapetype definitions depending on your tape library.
- What is the size of the holding disk "hd1"? How much free space is available on it?

I'd try increasing the tape_splitsize to 200Gb (assuming you have this amount of free space available on the /scratch filesystem):

define dumptype user-tar-span {
#holdingdisk no
root-tar
tape_splitsize 200 Gb
split_diskbuffer "/scratch/"
#fallback_splitsize 8 Gb
priority medium
}

Best,
Valeriu

On Mon, Feb 22, 2010 at 10:11:01AM -0800, William Taylor wrote:
I seem to be having some issues with amanda getting data off my drives in a reasonable time.
using dd reading off my drives and writing to tape I get around 120mbs to the LTO4 drive.
When using amanda however this number fluctuates quite a bit. At times I have seen it write to tape
at around 110mbs but it's usually around 40 or so.

I have tried several different configuration options and none seem to really improve the situation.

I am using amanda to do local backups.
I have a bunch of large (+2TB) netapp dumps on a 24 disk 16T raid5 array connected to a 3Ware.
I have also configured a 14 disk netapp shelf with 15k scsi drives striped with software raid connected via a 2GB fcal to use as
a scratch disk to break up dumps or to use as a holding disk.


Can anyone recommend amanda config options that would work with my setup?
So far it seems that best results are achieved without using the holding disk.
I have tried setting the tape_splitsize to 4Gb and fallback_splitsize to 8Gb with no
real change in performance. I have 12GB of memory.


Heres the relevant bits I have so far:

netusage 100 mbps
device_output_buffer_size 1 Gb
tapedev "tape:/dev/nst0"
device_property "BLOCK_SIZE" "1024k"

holdingdisk hd1 {
comment "main holding disk"
directory "/scratch/"
use -100 Mb
chunksize 80Gb
}

define tapetype LTO4 {
comment "Dell LTO4 800Gb - Compression Off"
length 802816 mbytes
filemark 0 kbytes
speed 120 mbps
blocksize 1024 kbytes
}

define dumptype user-tar-span {
#holdingdisk no
root-tar
tape_splitsize 80 Gb
split_diskbuffer "/scratch/"
#fallback_splitsize 8 Gb
priority medium
}


If I need to provide anymore info let me know.

Thanks,
William

* William Taylor <williamt < at > corp.sonic.net> [20100222 13:12]:
I seem to be having some issues with amanda getting data off my drives in a reasonable time.
using dd reading off my drives and writing to tape I get around 120mbs to the LTO4 drive.
When using amanda however this number fluctuates quite a bit. At times I have seen it write to tape
at around 110mbs but it's usually around 40 or so.

OK, I'll bite.

Hardware: SuperMicro X8DTN, 2xQuadCore Xeon 2.27GHz, 12Gb Ram
Overland ArcVault48, 2 HP LTO4 drives, LSI Dual port
U320 scsi HBA.
OS is Debian/Lenny with kernel 2.6.31 64bit.

I'm running amanda-2.6.1p2 and I routinely get +100MBps from a
holddisk to LTO4. An HP LTO4 will stream when speed to tape is
40-120MBps. To get around and above 100MBps you really need to tune
your holddisk/hardware and kernel IO. I use a 4-spindles striped
logical volume, Seagate 1TB Sata disks, with XFS on top of it, tuned
again to get max output. The disks are on a inboard backplane inside
the server. Note that I use hardware compression on the LTO4 drives
but these drives are smart so I don't think it matters very much.

I use tape_splitsize=10GB but I don't think this is relevant in terms
of performance. blocksize=2048MB but again my tests never showed
significant differences in the range [512MB-2048MB]

My advice: you won't get anywhere until you tune your holddisk.
This is the major bottleneck! The holddisk is your friend!

LTO4 drives are *fast* and you need to carefully tune your hardware IO
stack to get them close to max performance. That is, once you have
validated your tape drive and scsi path to them, HBA, cabling, etc...

hth,
jf


I have tried several different configuration options and none seem to really improve the situation.

I am using amanda to do local backups.
I have a bunch of large (+2TB) netapp dumps on a 24 disk 16T raid5 array connected to a 3Ware.
I have also configured a 14 disk netapp shelf with 15k scsi drives striped with software raid connected via a 2GB fcal to use as
a scratch disk to break up dumps or to use as a holding disk.


Can anyone recommend amanda config options that would work with my setup?
So far it seems that best results are achieved without using the holding disk.
I have tried setting the tape_splitsize to 4Gb and fallback_splitsize to 8Gb with no
real change in performance. I have 12GB of memory.


Heres the relevant bits I have so far:

netusage 100 mbps
device_output_buffer_size 1 Gb
tapedev "tape:/dev/nst0"
device_property "BLOCK_SIZE" "1024k"

holdingdisk hd1 {
comment "main holding disk"
directory "/scratch/"
use -100 Mb
chunksize 80Gb
}

define tapetype LTO4 {
comment "Dell LTO4 800Gb - Compression Off"
length 802816 mbytes
filemark 0 kbytes
speed 120 mbps
blocksize 1024 kbytes
}

define dumptype user-tar-span {
#holdingdisk no
root-tar
tape_splitsize 80 Gb
split_diskbuffer "/scratch/"
#fallback_splitsize 8 Gb
priority medium
}


If I need to provide anymore info let me know.

Thanks,
William

On Feb 22, 2010, at 1:33 PM, Jean-Francois Malouin wrote:

I use tape_splitsize=10GB but I don't think this is relevant in terms
of performance. blocksize=2048MB but again my tests never showed
significant differences in the range [512MB-2048MB]

Are you using split_diskbuffer or fallback_splitsize?
If fallback_splitsize what size do you use 10GB?

My backups are already local on disk (/dev/sdb). Does it make sense to define
a holding disk (/dev/md0 striped 14 disks) and have amanda moving them in chunks to
it ?

* William Taylor <williamt < at > corp.sonic.net> [20100222 17:56]:

On Feb 22, 2010, at 1:33 PM, Jean-Francois Malouin wrote:

I use tape_splitsize=10GB but I don't think this is relevant in terms
of performance. blocksize=2048MB but again my tests never showed
significant differences in the range [512MB-2048MB]

Are you using split_diskbuffer or fallback_splitsize?
If fallback_splitsize what size do you use 10GB?

Same but read remark:

tape_splitsize 10Gb
fallback_splitsize 10Gb

but my experience is going in port-write mode just doesn't cut it.
I typically get ~25-30MBps to the tape, ie shoe shining.
Never bothered to know why since I always have a holddisk available.


My backups are already local on disk (/dev/sdb). Does it make sense to define
a holding disk (/dev/md0 striped 14 disks) and have amanda moving them in chunks to
it ?

There will be some contention for sure if both are on the same
controller/IO path. Why don't you test it? Sucking up 100MBps sustain
or more from a raid will be hard to achieve unless you throw a lot of
$$$ to it which is why you want a well tuned striped staging area
between your data and tape drive.

You can then also starts playing with the flush-threshold* variables
and taperflush to stuff your holddisk up to a certain level before
writing to tape. Very cool new addition to amanda.
Just a suggestion though.

regards,
jf

On Mon, Feb 22, 2010 at 5:21 PM, Jean-Francois Malouin
<malin < at > bic.mni.mcgill.ca> wrote:
but my experience is going in port-write mode just doesn't cut it.
I typically get ~25-30MBps to the tape, ie shoe shining.
Never bothered to know why since I always have a holddisk available.

Usually this is because the dumping application can't provide data
quickly enough. It can also be due to slow disk being used for the
split buffer.

As an aside, sometime in the release *after* 3.1 we're planning to
remove the need for a disk buffer, since most tape drives can now give
an indication that they are near the end of the tape, allowing Amanda
to finish every part cleanly.

Dustin

--
Open Source Storage Engineer
http://www.zmanda.com