SearchFAQMemberlist Log in
Reply to topic Page 1 of 3
Goto page 1, 2, 3  Next
tuning lto-4
Author Message
Post tuning lto-4 
Hi --

Getting about 41.6/MBs and hoping for closer to the max (120MB). I
tried maximum file sizes of 5, 8, 12GB -- 12GB the best the others
where about 35/MBs. Any advise welcomed...should I look at max/min
block sizes?
most of the data is big, genetics data -- filesizes avg in the 500/MB
to 3-4/GB -- looking at a growth from 4TB to 15TB in the next 2 years.

run results and bacula-sd.conf and bacula-dir.conf below...

thanks
-- gary

Run:
===

Build OS: x86_64-redhat-linux-gnu redhat
JobId: 5
Job: Prodbackup.2011-11-29_19.32.42_05
Backup Level: Full
Client: "bacula-fd" 5.0.3 (04Aug10)
x86_64-redhat-linux-gnu,red


hat,
FileSet: "FileSetProd" 2011-11-29 19:32:42
Pool: "FullProd" (From Job FullPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "LTO-4" (From Job resource)
Scheduled time: 29-Nov-2011 19:32:26
Start time: 29-Nov-2011 19:32:45
End time: 29-Nov-2011 21:15:53
Elapsed time: 1 hour 43 mins 8 secs
Priority: 10
FD Files Written: 35,588
SD Files Written: 35,588
FD Bytes Written: 257,543,090,368 (257.5 GB)
SD Bytes Written: 257,548,502,159 (257.5 GB)
Rate: 41619.8 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: no
Volume name(s): f03
Volume Session Id: 1
Volume Session Time: 1322622337
Last Volume Bytes: 257,740,342,272 (257.7 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Termination: Backup OK

bacula-sd.conf:
==========

Autochanger {
Name = Autochanger
Device = LTO-4
Changer Command = "/usr/libexec/bacula/mtx-changer %c %o %S %a %d"
Changer Device = /dev/changer
}
Device {
Name = LTO-4
Media Type = LTO-4
Archive Device = /dev/nst0
AutomaticMount = yes; # when device opened, read it
AlwaysOpen = yes;
RemovableMedia = yes;
RandomAccess = no;
Maximum File Size = 12GB
Autochanger = yes
Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
Alert Command = "sh -c 'smartctl -H -l error %c'"
}
bacula-dir.conf:
==========

Job {
Name = "Prodbackup"
Client = bacula-fd
FileSet = "FileSetProd"
Schedule = "ScheduleProd"
Write Bootstrap = "/var/spool/bacula/%c.bsr"
Full Backup Pool = FullProd
Incremental Backup Pool = IncrProd
Differential Backup Pool = DiffProd
Storage = LTO-4
Type = Backup
Level = Incremental
Pool = IncrProd
Priority = 10
Messages = Standard
}
FileSet {
Name = "FileSetProd"
Include {
Options {
WildFile = "*.OLD"
WildFile = "*.o"
WildFile = "*.bak"
exclude = yes
}
File = /my/home/xxxxxxx
}
Exclude {
File = /my/home/tmp
}
}
Schedule {
Name = "ScheduleProd"
Run = Full 1st sun at 16:05
Run = Pool {
Name = FullProd
Label Format = "FullProd"
Pool Type = Backup
Recycle = yes # Bacula can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 10 years
Maximum Volume Jobs = 1
}

Pool {
Name = DiffProd
Label Format = "DiffProd"
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 40 days
Maximum Volume Jobs = 1
}

Pool {
Name = IncrProd
Label Format = "IncrProd"
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 10 days
Maximum Volume Jobs = 1
}
Differential 2nd-5th sun at 16:05
Run = Incremental mon-sat at 16:05
}

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
gary artim wrote:
Hi --

Getting about 41.6/MBs and hoping for closer to the max (120MB). I
tried maximum file sizes of 5, 8, 12GB -- 12GB the best the others
where about 35/MBs. Any advise welcomed...should I look at max/min
block sizes?

Don't adjust min size.

Bacula's max block size is ~2Mb (default 65535 bytes) and setting this
should give a substantial speed boost (it did for me). Going higher than
bacula's maximum will result in it failing throught to default, so
double check the block size on a newly labelled tape before committing
to any value.

WARNING: If you adjust this value, mark all current tapes as USED before
restarting bacula-sd. Changing block size on an open tape is likely to
lead to problems getting data off it.

LTO drives tend to have 16Mb maximum buffers. Perhaps Bacula's max block
size needs increasing? (Kern?)

You should also enable spooling to _very_ fast disk before hitting the
tape. LTO3 upwards will easily run faster than any spinning media and
trivially outrun even a raid array if that has to do any seeking.

I use a raid0 set of 5 of Intel E25 60Gb drives and have seen
throughputs approaching 700Mb/s out to 3 tape drives whilst other jobs
are spooling (I've got 7 LTO5 drives carrying 6 pools but have never
seen more than 3 writing simultaneously). These days I'd be more
inclined to use one of the PCIe SSD cards as they're even faster, with
less overhead.

Device {
Name = LTO-4
Media Type = LTO-4
Archive Device = /dev/nst0
AutomaticMount = yes; # when device opened, read it
AlwaysOpen = yes;
RemovableMedia = yes;
RandomAccess = no;
Maximum File Size = 12GB

Maximum Network Buffer Size = 65536
Maximum block size = 2M

Spool Directory = /var/bacula/spool/LTO4
Maximum Spool Size = 280G
Maximum Job Spool Size = 150G

Autochanger = yes
Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
Alert Command = "sh -c 'smartctl -H -l error %c'"
}

Enlarging network buffers is possible but it must be be the same
everywhere and should be thoroughly tested first as it can as easily
cause complete breakage as speedups - especially if backups are taking
place across a routed network instead of just on your LAN.


AB



------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
Thanks much, I'll try today the block size change first. Then try the
spooling. Dont have any unused disk, but may have to try on a shared
drive.
The "maximum file size" should be okay? g.

On Wed, Nov 30, 2011 at 8:45 AM, Alan Brown <ajb2 < at > mssl.ucl.ac.uk> wrote:
gary artim wrote:

Hi --

Getting about 41.6/MBs and hoping for closer to the max (120MB). I
tried maximum file sizes of 5, 8, 12GB -- 12GB the best the others
where about 35/MBs. Any advise welcomed...should I look at max/min
block sizes?


Don't adjust min size.

Bacula's max block size is ~2Mb (default 65535 bytes) and setting this
should give a substantial speed boost (it did for me). Going higher than
bacula's maximum will result in it failing throught to default, so double
check the block size on a newly labelled tape before committing to any
value.

WARNING: If you adjust this value, mark all current tapes as USED before
restarting bacula-sd. Changing block size on an open tape is likely to lead
to problems getting data off it.

LTO drives tend to have 16Mb maximum buffers. Perhaps Bacula's max block
size needs increasing? (Kern?)

You should also enable spooling to _very_ fast disk before hitting the tape.
LTO3 upwards will easily run faster than any spinning media and trivially
outrun even a raid array if that has to do any seeking.

I use a raid0 set of 5 of Intel E25 60Gb drives and have seen throughputs
approaching 700Mb/s out to 3 tape drives whilst other jobs are spooling
(I've got 7 LTO5 drives carrying 6 pools but have never seen more than 3
writing simultaneously). These days I'd be more inclined to use one of the
PCIe SSD cards as they're even faster, with less overhead.


Device {
 Name = LTO-4
 Media Type = LTO-4
 Archive Device = /dev/nst0
 AutomaticMount = yes;               # when device opened, read it
 AlwaysOpen = yes;
 RemovableMedia = yes;
 RandomAccess = no;
 Maximum File Size = 12GB


 Maximum Network Buffer Size = 65536
 Maximum block size = 2M

 Spool Directory = /var/bacula/spool/LTO4
 Maximum Spool Size     = 280G
 Maximum Job Spool Size = 150G


 Autochanger = yes
 Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
 Alert Command = "sh -c 'smartctl -H -l error %c'"
}


Enlarging network buffers is possible but it must be be the same everywhere
and should be thoroughly tested first as it can as easily cause complete
breakage as speedups - especially if backups are taking place across a
routed network instead of just on your LAN.


AB



------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
block size change didnt make much difference, but also running with an
rsync running against the backup volume (raid 5). Adding spool and
will run with both blocksize change and spool configuration. -- gary

On Wed, Nov 30, 2011 at 10:43 AM, gary artim <gartim < at > gmail.com> wrote:
Thanks much, I'll try today the block size change first. Then try the
spooling. Dont have any unused disk, but may have to try on a shared
drive.
The "maximum file size" should be okay? g.

On Wed, Nov 30, 2011 at 8:45 AM, Alan Brown <ajb2 < at > mssl.ucl.ac.uk> wrote:
gary artim wrote:

Hi --

Getting about 41.6/MBs and hoping for closer to the max (120MB). I
tried maximum file sizes of 5, 8, 12GB -- 12GB the best the others
where about 35/MBs. Any advise welcomed...should I look at max/min
block sizes?


Don't adjust min size.

Bacula's max block size is ~2Mb (default 65535 bytes) and setting this
should give a substantial speed boost (it did for me). Going higher than
bacula's maximum will result in it failing throught to default, so double
check the block size on a newly labelled tape before committing to any
value.

WARNING: If you adjust this value, mark all current tapes as USED before
restarting bacula-sd. Changing block size on an open tape is likely to lead
to problems getting data off it.

LTO drives tend to have 16Mb maximum buffers. Perhaps Bacula's max block
size needs increasing? (Kern?)

You should also enable spooling to _very_ fast disk before hitting the tape.
LTO3 upwards will easily run faster than any spinning media and trivially
outrun even a raid array if that has to do any seeking.

I use a raid0 set of 5 of Intel E25 60Gb drives and have seen throughputs
approaching 700Mb/s out to 3 tape drives whilst other jobs are spooling
(I've got 7 LTO5 drives carrying 6 pools but have never seen more than 3
writing simultaneously). These days I'd be more inclined to use one of the
PCIe SSD cards as they're even faster, with less overhead.


Device {
 Name = LTO-4
 Media Type = LTO-4
 Archive Device = /dev/nst0
 AutomaticMount = yes;               # when device opened, read it
 AlwaysOpen = yes;
 RemovableMedia = yes;
 RandomAccess = no;
 Maximum File Size = 12GB


 Maximum Network Buffer Size = 65536
 Maximum block size = 2M

 Spool Directory = /var/bacula/spool/LTO4
 Maximum Spool Size     = 280G
 Maximum Job Spool Size = 150G


 Autochanger = yes
 Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
 Alert Command = "sh -c 'smartctl -H -l error %c'"
}


Enlarging network buffers is possible but it must be be the same everywhere
and should be thoroughly tested first as it can as easily cause complete
breakage as speedups - especially if backups are taking place across a
routed network instead of just on your LAN.


AB



------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
On 30/11/11 19.43, gary artim wrote:
Thanks much, I'll try today the block size change first. Then try the
spooling. Dont have any unused disk, but may have to try on a shared
drive.
The "maximum file size" should be okay? g.

Choosing a max file size is mainly a tradeoff between write performance
(as the drive will stop and restart at the end of each file to write an
EOF mark) and restore performance (as the drive can only seek to a file
mark and then sequentially read through the file until the relevant data
bocks are found).

I usually set maximum file size so that there are 2-3 filemarks per tape
wrap (3GB for LTO3, 5GB for LTO4), but if you don't plan to do regular
restores, or if you always restore the whole contents of a volume, 12GB
is fine.

Anyway, with the figures you're citing your problem is *not* maximum
file size.

Try to assess tape performance alone with btape test (which has a
"speed" command); you can try different block sizes and configuration
and see which one gives the best results.

Doing so will give you a clear indication on whether your bottleneck is
in tape or disk throughput.

andrea

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
thank much! will try testing with btape. btw, I ran with 20GB maximum
file size/2MB max block (see bacula-sd.conf below) and got these
results, 20MB/s increase, ran 20 minutes faster, got 50MBs -- now if I
can just double the speed I could backup 15TB in about 45/hrs. I don't
have that much data yet, but I'm hovering at 2TB and looking to expand
sharply over time. I'm not doing any networking, it just straight from
a raid 5 to a autochanger/lto-4. gary

Build OS: x86_64-redhat-linux-gnu redhat
JobId: 6
Job: Prodbackup.2011-11-30_18.49.24_06
Backup Level: Full
Client: "bacula-fd" 5.0.3 (04Aug10)
x86_64-redhat-linux-gnu,redhat,
FileSet: "FileSetProd" 2011-11-30 15:23:58
Pool: "FullProd" (From Job FullPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "LTO-4" (From Job resource)
Scheduled time: 30-Nov-2011 18:49:15
Start time: 30-Nov-2011 18:49:26
End time: 30-Nov-2011 20:14:56
Elapsed time: 1 hour 25 mins 30 secs
Priority: 10
FD Files Written: 35,588
SD Files Written: 35,588
FD Bytes Written: 257,543,092,723 (257.5 GB)
SD Bytes Written: 257,548,504,514 (257.5 GB)
Rate: 50203.3 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: no
Volume name(s): f2
Volume Session Id: 2
Volume Session Time: 1322707293
Last Volume Bytes: 257,600,822,272 (257.6 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Termination: Backup OK

bacula-sd.conf:
Device {
Name = LTO-4
Media Type = LTO-4
Archive Device = /dev/nst0
AutomaticMount = yes; # when device opened, read it
AlwaysOpen = yes;
RemovableMedia = yes;
RandomAccess = no;
#Maximum File Size = 12GB
Maximum File Size = 20GB
#Maximum Network Buffer Size = 65536
Maximum block size = 2M
#Spool Directory = /db/bacula/spool/LTO4
#Maximum Spool Size = 200G
#Maximum Job Spool Size = 150G
Autochanger = yes
Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
Alert Command = "sh -c 'smartctl -H -l error %c'"
}



On Wed, Nov 30, 2011 at 11:48 PM, Andrea Conti <alyf < at > alyf.net> wrote:
On 30/11/11 19.43, gary artim wrote:
Thanks much, I'll try today the block size change first. Then try the
spooling. Dont have any unused disk, but may have to try on a shared
drive.
The "maximum file size" should be okay? g.

Choosing a max file size is mainly a tradeoff between write performance
(as the drive will stop and restart at the end of each file to write an
EOF mark) and restore performance (as the drive can only seek to a file
mark and then sequentially read through the file until the relevant data
bocks are found).

I usually set maximum file size so that there are 2-3 filemarks per tape
wrap (3GB for LTO3, 5GB for LTO4), but if you don't plan to do regular
restores, or if you always restore the whole contents of a volume, 12GB
is fine.

Anyway, with the figures you're citing your problem is *not* maximum
file size.

Try to assess tape performance alone with btape test (which has a
"speed" command); you can try different block sizes and configuration
and see which one gives the best results.

Doing so will give you a clear indication on whether your bottleneck is
in tape or disk throughput.

andrea

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
gary artim wrote:
thank much! will try testing with btape.

Please let us know the results

btw, I ran with 20GB maximum
file size/2MB max block (see bacula-sd.conf below) and got these
results, 20MB/s increase, ran 20 minutes faster, got 50MBs --

You should be seeing 120Mb/s or thereabouts.

If you're spooling/despooling then you'll see lower overall speeds of
course. What counts is the despooling speed.

How much ram have you got and what are you using to connect the LTO4
drives up?





------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
You guys/gals are great, very responsive! I did try
spooling/despooling and my run times shot up. I was using a simple
7200 drive though, no ssd or raid...I assume the performance gain
happens when your networks multi machines...wearing multiple hats so
will report back on btape next week, unless I get some time. gary

On Thu, Dec 1, 2011 at 8:00 AM, Alan Brown <ajb2 < at > mssl.ucl.ac.uk> wrote:
gary artim wrote:

thank much! will try testing with btape.

Please let us know the results

btw, I ran with 20GB maximum
file size/2MB max block (see bacula-sd.conf below) and got these
results, 20MB/s increase, ran 20 minutes faster, got 50MBs --

You should be seeing 120Mb/s or thereabouts.

If you're spooling/despooling then you'll see lower overall speeds of
course. What counts is the despooling speed.

How much ram have you got and what are you using to connect the LTO4 drives
up?






------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
I believe (its been a while since I have needed to change my
configuration) that my LTO-3 drive does not do hardware compression on
blocks over 512K. I am using 256K blocks right now, and I did not see
any improvement above that. I am using spooling on a pair of striped
hard disks, and despooling happens at 65-80MB/s


On 12/1/2011 10:50 AM, gary artim wrote:
thank much! will try testing with btape. btw, I ran with 20GB maximum
file size/2MB max block (see bacula-sd.conf below) and got these
results, 20MB/s increase, ran 20 minutes faster, got 50MBs -- now if I
can just double the speed I could backup 15TB in about 45/hrs. I don't
have that much data yet, but I'm hovering at 2TB and looking to expand
sharply over time. I'm not doing any networking, it just straight from
a raid 5 to a autochanger/lto-4. gary

Build OS: x86_64-redhat-linux-gnu redhat
JobId: 6
Job: Prodbackup.2011-11-30_18.49.24_06
Backup Level: Full
Client: "bacula-fd" 5.0.3 (04Aug10)
x86_64-redhat-linux-gnu,redhat,
FileSet: "FileSetProd" 2011-11-30 15:23:58
Pool: "FullProd" (From Job FullPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "LTO-4" (From Job resource)
Scheduled time: 30-Nov-2011 18:49:15
Start time: 30-Nov-2011 18:49:26
End time: 30-Nov-2011 20:14:56
Elapsed time: 1 hour 25 mins 30 secs
Priority: 10
FD Files Written: 35,588
SD Files Written: 35,588
FD Bytes Written: 257,543,092,723 (257.5 GB)
SD Bytes Written: 257,548,504,514 (257.5 GB)
Rate: 50203.3 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: no
Volume name(s): f2
Volume Session Id: 2
Volume Session Time: 1322707293
Last Volume Bytes: 257,600,822,272 (257.6 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Termination: Backup OK

bacula-sd.conf:
Device {
Name = LTO-4
Media Type = LTO-4
Archive Device = /dev/nst0
AutomaticMount = yes; # when device opened, read it
AlwaysOpen = yes;
RemovableMedia = yes;
RandomAccess = no;
#Maximum File Size = 12GB
Maximum File Size = 20GB
#Maximum Network Buffer Size = 65536
Maximum block size = 2M
#Spool Directory = /db/bacula/spool/LTO4
#Maximum Spool Size = 200G
#Maximum Job Spool Size = 150G
Autochanger = yes
Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
Alert Command = "sh -c 'smartctl -H -l error %c'"
}



On Wed, Nov 30, 2011 at 11:48 PM, Andrea Conti<alyf < at > alyf.net> wrote:
On 30/11/11 19.43, gary artim wrote:
Thanks much, I'll try today the block size change first. Then try the
spooling. Dont have any unused disk, but may have to try on a shared
drive.
The "maximum file size" should be okay? g.
Choosing a max file size is mainly a tradeoff between write performance
(as the drive will stop and restart at the end of each file to write an
EOF mark) and restore performance (as the drive can only seek to a file
mark and then sequentially read through the file until the relevant data
bocks are found).

I usually set maximum file size so that there are 2-3 filemarks per tape
wrap (3GB for LTO3, 5GB for LTO4), but if you don't plan to do regular
restores, or if you always restore the whole contents of a volume, 12GB
is fine.

Anyway, with the figures you're citing your problem is *not* maximum
file size.

Try to assess tape performance alone with btape test (which has a
"speed" command); you can try different block sizes and configuration
and see which one gives the best results.

Doing so will give you a clear indication on whether your bottleneck is
in tape or disk throughput.

andrea

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
gary artim wrote:
You guys/gals are great, very responsive! I did try
spooling/despooling and my run times shot up.

They will - you're copying everything twice (disk to disk to tape), but
this is the only way to achieve fast despooling speeds - if you don't do
this then your LTO drive will start to "shoe shine" and speeds drop off
rapidly when it happens.

The trick is to run multiple jobs at once - you have to spool to achieve
this anyway or extracting will be a nightmare.

Spooling is a net gain when you're running incrementals.

Spooling MUST happen on a fast dedicated drive. You're best off dropping
in a fast SSD such as a 64/128Gb OCZ vertex3 or similar to handle it.

I was using a simple
7200 drive though, no ssd or raid...I assume the performance gain
happens when your networks multi machines...wearing multiple hats so
will report back on btape next week, unless I get some time. gary

Even on a single host, if the heads are thrashing then spooling will
save time overall. The big advantage is being able to run multiple jobs
so that several are spooling data at the same time one is despooling.

On Thu, Dec 1, 2011 at 8:00 AM, Alan Brown <ajb2 < at > mssl.ucl.ac.uk> wrote:
gary artim wrote:
thank much! will try testing with btape.
Please let us know the results

btw, I ran with 20GB maximum
file size/2MB max block (see bacula-sd.conf below) and got these
results, 20MB/s increase, ran 20 minutes faster, got 50MBs --
You should be seeing 120Mb/s or thereabouts.

If you're spooling/despooling then you'll see lower overall speeds of
course. What counts is the despooling speed.

How much ram have you got and what are you using to connect the LTO4 drives
up?










------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
In the message dated: Thu, 01 Dec 2011 16:27:33 GMT,
The pithy ruminations from Alan Brown on
<Re: [Bacula-users] tuning lto-4> were:
=> gary artim wrote:
=> > You guys/gals are great, very responsive! I did try
=> > spooling/despooling and my run times shot up.
=>
=> They will - you're copying everything twice (disk to disk to tape), but
=> this is the only way to achieve fast despooling speeds - if you don't do
=> this then your LTO drive will start to "shoe shine" and speeds drop off
=> rapidly when it happens.

And you increase wear & tear on the drive and media.

=>
=> The trick is to run multiple jobs at once - you have to spool to achieve
=> this anyway or extracting will be a nightmare.
=>
=> Spooling is a net gain when you're running incrementals.
=>

Not necessarily. Spooling is a gain if you are measuring the speed
of writing to tape. Spooling may be a net loss for end-to-end (client
machine-->spool server-->tape drive) speed.

For backups clients where the total volume being backed up is less than
the spool size, then there's a very good chance of a performance gain. As
soon as a job requires multiple rounds of spooling and de-spooling,
there's a good chance of a performance loss because bacula stops reading
from the client machine (stops spooling that job) as soon as despooling
begins. Of course, spooling allows you to run multiple jobs in parallel, a
clear win over running them in series.


See:

[1] http://copilotco.com/mail-archives/bacula-devel.2007/msg02642.html
[2] http://www.bacula.org/git/cgit.cgi/bacula/plain/bacula/projects?h=Branch-5.1

[3] http://www.mail-archive.com/bacula-users < at > lists.sourceforge.net/msg49366.html


=> Spooling MUST happen on a fast dedicated drive. You're best off dropping
=> in a fast SSD such as a 64/128Gb OCZ vertex3 or similar to handle it.

Hmm...for LTO4 (large spool files are good), you might want more space
than that, particularly if you have multiple clients (multiple spool
files). A more cost-effective option might be several fast drives (10K
or 15K SAS or SCSI) in RAID-0. It doesn't take very many drives in RAID0 to
have an aggregate drive throughput that is greater than the bus interface.

=>
=> > I was using a simple
=> > 7200 drive though, no ssd or raid...I assume the performance gain

Yeah, the sustained read speed from a 7.2k RPM drive is lower than the
possible write speed to an LTO-4 drive:

http://www.seagate.com/www/en-us/support/before_you_buy/speed_considerations

=> > happens when your networks multi machines...wearing multiple hats so
=> > will report back on btape next week, unless I get some time. gary
=>
=> Even on a single host, if the heads are thrashing then spooling will
=> save time overall. The big advantage is being able to run multiple jobs
=> so that several are spooling data at the same time one is despooling.

Absolutely. Spooling is a big win for multiple jobs, and for reducing
wear&tear on the tape drive. It may or may not give a performance increase for
any single backup job.

Mark


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
btape getting 89 MBs, so maybe my disk and sql updating is effecting
the speed? note drive has a 16384 blocksize, ran tapeinfo on the
drive...gary

[root < at > genepi1 bacula]# btape -c /etc/bacula/bacula-sd.conf /dev/nst0
Tape block granularity is 1024 bytes.
btape: butil.c:284 Using device: "/dev/nst0" for writing.
01-Dec 12:29 btape JobId 0: 3301 Issuing autochanger "loaded? drive 0" command.
01-Dec 12:29 btape JobId 0: 3302 Autochanger "loaded? drive 0", result
is Slot 12.
btape: btape.c:476 open device "LTO-4" (/dev/nst0): OK
*speed file_size=3 skip_raw
btape: btape.c:1082 Test with zero data and bacula block structure.
btape: btape.c:960 Begin writing 3 files of 3.221 GB with blocks of
2097152 bytes.
+++4
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=3.221 GB. Write rate = 89.47 MB/s
+++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=3.221 GB. Write rate = 89.47 MB/s
+++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=3.221 GB. Write rate = 89.47 MB/s
btape: btape.c:384 Total Volume bytes=9.663 GB. Total Write rate = 89.47 MB/s

btape: btape.c:1094 Test with random data, should give the minimum throughput.
btape: btape.c:960 Begin writing 3 files of 3.221 GB with blocks of
2097152 bytes.
+++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=3.221 GB. Write rate = 16.02 MB/s
+++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=3.221 GB. Write rate = 33.90 MB/s
+++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=3.221 GB. Write rate = 44.12 MB/s
btape: btape.c:384 Total Volume bytes=9.663 GB. Total Write rate = 26.18 MB/s


[root < at > genepi1 bacula]# tapeinfo
Usage: tapeinfo -f <generic-device>
[root < at > genepi1 bacula]# tapeinfo -f /dev/changer
Product Type: Medium Changer
Vendor ID: 'OVERLAND'
Product ID: 'NEO Series '
Revision: '0504'
Attached Changer API: No
SerialNumber: '2B81000045'
SCSI ID: 1
SCSI LUN: 1
Ready: yes
[root < at > genepi1 bacula]# tapeinfo -f /dev/nst0
Product Type: Tape Drive
Vendor ID: 'HP '
Product ID: 'Ultrium 4-SCSI '
Revision: 'B12H'
Attached Changer API: No
SerialNumber: 'HU17450M8L'
MinBlock: 1
MaxBlock: 16777215
SCSI ID: 1
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x46
BlockSize: 16384
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: 799204
Partition 0 Size in Kbytes: 799204
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 0



On Thu, Dec 1, 2011 at 10:49 AM, <mark.bergman < at > uphs.upenn.edu> wrote:
In the message dated: Thu, 01 Dec 2011 16:27:33 GMT,
The pithy ruminations from Alan Brown on
<Re: [Bacula-users] tuning lto-4> were:
=> gary artim wrote:
=> > You guys/gals are great, very responsive! I did try
=> > spooling/despooling and my run times shot up.
=>
=> They will - you're copying everything twice (disk to disk to tape), but
=> this is the only way to achieve fast despooling speeds - if you don't do
=> this then your LTO drive will start to "shoe shine" and speeds drop off
=> rapidly when it happens.

And you increase wear & tear on the drive and media.

=>
=> The trick is to run multiple jobs at once - you have to spool to achieve
=> this anyway or extracting will be a nightmare.
=>
=> Spooling is a net gain when you're running incrementals.
=>

Not necessarily. Spooling is a gain if you are measuring the speed
of writing to tape. Spooling may be a net loss for end-to-end (client
machine-->spool server-->tape drive) speed.

For backups clients where the total volume being backed up is less than
the spool size, then there's a very good chance of a performance gain. As
soon as a job requires multiple rounds of spooling and de-spooling,
there's a good chance of a performance loss because bacula stops reading
from the client machine (stops spooling that job) as soon as despooling
begins. Of course, spooling allows you to run multiple jobs in parallel, a
clear win over running them in series.


See:

       [1] http://copilotco.com/mail-archives/bacula-devel.2007/msg02642.html
       [2] http://www.bacula.org/git/cgit.cgi/bacula/plain/bacula/projects?h=Branch-5.1

       [3] http://www.mail-archive.com/bacula-users < at > lists.sourceforge.net/msg49366.html


=> Spooling MUST happen on a fast dedicated drive. You're best off dropping
=> in a fast SSD such as a 64/128Gb OCZ vertex3 or similar to handle it.

Hmm...for LTO4 (large spool files are good), you might want more space
than that, particularly if you have multiple clients (multiple spool
files). A more cost-effective option might be several fast drives (10K
or 15K SAS or SCSI) in RAID-0. It doesn't take very many drives in RAID0 to
have an aggregate drive throughput that is greater than the bus interface.

=>
=> > I was using a simple
=> > 7200 drive though, no ssd or raid...I assume the performance gain

Yeah, the sustained read speed from a 7.2k RPM drive is lower than the
possible write speed to an LTO-4 drive:

       http://www.seagate.com/www/en-us/support/before_you_buy/speed_considerations

=> > happens when your networks multi machines...wearing multiple hats so
=> > will report back on btape next week, unless I get some time. gary
=>
=> Even on a single host, if the heads are thrashing then spooling will
=> save time overall. The big advantage is being able to run multiple jobs
=> so that several are spooling data at the same time one is despooling.

Absolutely. Spooling is a big win for multiple jobs, and for reducing
wear&tear on the tape drive. It may or may not give a performance increase for
any single backup job.

Mark



------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
got close to 120 MBs, using 64kb buffer and 20gb maximum file size
using btape...now test with real data...gary

=======================================================
blocksize set with mt and in bacula-sd.conf to == 65536
=======================================================

[root < at > genepi1 bacula]# btape -c /etc/bacula/bacula-sd.conf /dev/nst0
Tape block granularity is 1024 bytes.
btape: butil.c:284 Using device: "/dev/nst0" for writing.
01-Dec 13:36 btape JobId 0: 3301 Issuing autochanger "loaded? drive 0" command.
01-Dec 13:36 btape JobId 0: 3302 Autochanger "loaded? drive 0", result
is Slot 12.
btape: btape.c:476 open device "LTO-4" (/dev/nst0): OK
*speed file_size=20 nb_file=10 skip_raw
btape: btape.c:1082 Test with zero data and bacula block structure.
btape: btape.c:960 Begin writing 10 files of 21.47 GB with blocks of
65536 bytes.
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 111.2 MB/s
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 111.8 MB/s
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 111.8 MB/s
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 111.8 MB/s
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 111.8 MB/s
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 111.8 MB/s




=======================================================
blocksize set with mt and in bacula-sd.conf to == 32768
=======================================================
[root < at > genepi1 bacula]# btape -c /etc/bacula/bacula-sd.conf /dev/nst0
Tape block granularity is 1024 bytes.
btape: butil.c:284 Using device: "/dev/nst0" for writing.
01-Dec 13:12 btape JobId 0: 3301 Issuing autochanger "loaded? drive 0" command.
01-Dec 13:12 btape JobId 0: 3302 Autochanger "loaded? drive 0", result
is Slot 12.
btape: btape.c:476 open device "LTO-4" (/dev/nst0): OK
*speed file_size=20 nb_file=10 skip_raw
btape: btape.c:1082 Test with zero data and bacula block structure.
btape: btape.c:960 Begin writing 10 files of 21.47 GB with blocks of
32768 bytes.
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 94.60 MB/s
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 95.44 MB/s
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 95.44 MB/s
+.....
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 95.02 MB/s

On Wed, Nov 30, 2011 at 8:02 AM, gary artim <gartim < at > gmail.com> wrote:
Hi --

Getting about 41.6/MBs and hoping for closer to the max (120MB). I
tried maximum file sizes of 5, 8, 12GB -- 12GB the best the others
where about 35/MBs. Any advise welcomed...should I look at max/min
block sizes?
most of the data is big, genetics data -- filesizes avg in the 500/MB
to 3-4/GB -- looking at a growth from 4TB to 15TB in the next 2 years.

run results and bacula-sd.conf and bacula-dir.conf below...

thanks
-- gary

Run:
===

 Build OS:               x86_64-redhat-linux-gnu redhat
 JobId:                  5
 Job:                    Prodbackup.2011-11-29_19.32.42_05
 Backup Level:           Full
 Client:                 "bacula-fd" 5.0.3 (04Aug10)
x86_64-redhat-linux-gnu,red


hat,
 FileSet:                "FileSetProd" 2011-11-29 19:32:42
 Pool:                   "FullProd" (From Job FullPool override)
 Catalog:                "MyCatalog" (From Client resource)
 Storage:                "LTO-4" (From Job resource)
 Scheduled time:         29-Nov-2011 19:32:26
 Start time:             29-Nov-2011 19:32:45
 End time:               29-Nov-2011 21:15:53
 Elapsed time:           1 hour 43 mins 8 secs
 Priority:               10
 FD Files Written:       35,588
 SD Files Written:       35,588
 FD Bytes Written:       257,543,090,368 (257.5 GB)
 SD Bytes Written:       257,548,502,159 (257.5 GB)
 Rate:                   41619.8 KB/s
 Software Compression:   None
 VSS:                    no
 Encryption:             no
 Accurate:               no
 Volume name(s):         f03
 Volume Session Id:      1
 Volume Session Time:    1322622337
 Last Volume Bytes:      257,740,342,272 (257.7 GB)
 Non-fatal FD errors:    0
 SD Errors:              0
 FD termination status:  OK
 SD termination status:  OK
 Termination:            Backup OK

bacula-sd.conf:
==========

Autochanger {
 Name = Autochanger
 Device = LTO-4
 Changer Command = "/usr/libexec/bacula/mtx-changer %c %o %S %a %d"
 Changer Device = /dev/changer
}
Device {
 Name = LTO-4
 Media Type = LTO-4
 Archive Device = /dev/nst0
 AutomaticMount = yes;               # when device opened, read it
 AlwaysOpen = yes;
 RemovableMedia = yes;
 RandomAccess = no;
 Maximum File Size = 12GB
 Autochanger = yes
 Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
 Alert Command = "sh -c 'smartctl -H -l error %c'"
}
bacula-dir.conf:
==========

Job {
       Name = "Prodbackup"
       Client = bacula-fd
       FileSet = "FileSetProd"
       Schedule = "ScheduleProd"
       Write Bootstrap = "/var/spool/bacula/%c.bsr"
       Full Backup Pool = FullProd
       Incremental Backup Pool = IncrProd
       Differential Backup Pool = DiffProd
       Storage =  LTO-4
       Type = Backup
       Level = Incremental
       Pool = IncrProd
       Priority = 10
       Messages = Standard
}
FileSet {
       Name = "FileSetProd"
       Include {
               Options {
                       WildFile = "*.OLD"
                       WildFile = "*.o"
                       WildFile = "*.bak"
                       exclude = yes
               }
               File = /my/home/xxxxxxx
       }
       Exclude {
               File = /my/home/tmp
       }
}
Schedule {
       Name = "ScheduleProd"
       Run = Full 1st sun at 16:05
       Run = Pool {
       Name = FullProd
       Label Format = "FullProd"
       Pool Type = Backup
       Recycle = yes           # Bacula can automatically recycle Volumes
       AutoPrune = yes         # Prune expired volumes
       Volume Retention = 10 years
       Maximum Volume Jobs = 1
}

Pool {
       Name = DiffProd
       Label Format = "DiffProd"
       Pool Type = Backup
       Recycle = yes
       AutoPrune = yes
       Volume Retention =  40 days
       Maximum Volume Jobs = 1
}

Pool {
       Name = IncrProd
       Label Format = "IncrProd"
       Pool Type = Backup
       Recycle = yes
       AutoPrune = yes
       Volume Retention =  10 days
       Maximum Volume Jobs = 1
}
Differential 2nd-5th sun at 16:05
       Run = Incremental mon-sat at 16:05
}


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
Hello,

blocksize set with mt and in bacula-sd.conf

Unless you are setting "minimum block size" (which you really should
not), Bacula uses the tape drive in variable block size mode, with block
sizes up to the value given in "maximum block size".

Setting a fixed block size with mt (and reading it back with tapeinfo)
is irrelevant.

btape: btape.c:1082 Test with zero data and bacula block structure.
btape: btape.c:960 Begin writing 10 files of 21.47 GB with blocks of
65536 bytes.

btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 111.8 MB/s

A 64kb max block size is *way* too small for LTO4. You'll need at least
256kB, possibly 512kB or more to achieve full throughput.

When diagnosing throughput problems, the tests with raw block structure
are more representative of the actual performance of the tape drive,
although the difference will not be that much.

What are you getting in the random data tests? With ~112MB/s for zeroes,
you're still being severely limited by something (most likely block
size). LTO-4 is rated for 120MB/s *to tape*, so you should be aiming for
110MB/s with random data and 250-300MB/s with zeroes (the latter being
dependent on the compression engine maximum bandwidth).

If you can't achieve that with any maximum block size, your hardware is
probably inadequate for the task.

andrea

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Post tuning lto-4 
180 MBs, 256MB min/max blocksize.

[root < at > genepi1 bacula]# tapeinfo -f /dev/nst0
Product Type: Tape Drive
Vendor ID: 'HP '
Product ID: 'Ultrium 4-SCSI '
Revision: 'B12H'
Attached Changer API: No
SerialNumber: 'HU17450M8L'
MinBlock: 1
MaxBlock: 16777215
SCSI ID: 1
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x46
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
Block Position: 471909
Partition 0 Remaining Kbytes: 799204
Partition 0 Size in Kbytes: 799204
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 0
[root < at > genepi1 bacula]# btape -c /etc/bacula/bacula-sd.conf /dev/nst0
Tape block granularity is 1024 bytes.
btape: butil.c:284 Using device: "/dev/nst0" for writing.
02-Dec 13:07 btape JobId 0: 3301 Issuing autochanger "loaded? drive 0" command.
02-Dec 13:07 btape JobId 0: 3302 Autochanger "loaded? drive 0", result
is Slot 12.
btape: btape.c:476 open device "LTO-4" (/dev/nst0): OK
*speed file_size=20 nb_file=10 skip_raw
btape: btape.c:1082 Test with zero data and bacula block structure.
btape: btape.c:960 Begin writing 10 files of 21.47 GB with blocks of
524288 bytes.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 177.4 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 180.4 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 178.9 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 180.4 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 178.9 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 180.4 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 180.4 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 178.9 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 180.4 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 178.9 MB/s
btape: btape.c:384 Total Volume bytes=214.7 GB. Total Write rate = 179.5 MB/s

btape: btape.c:1094 Test with random data, should give the minimum throughput.
btape: btape.c:960 Begin writing 10 files of 21.47 GB with blocks of
524288 bytes.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 26.47 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
btape: btape.c:608 Wrote 1 EOF to "LTO-4" (/dev/nst0)
btape: btape.c:410 Volume bytes=21.47 GB. Write rate = 26.51 MB/s
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


On Wed, Nov 30, 2011 at 8:02 AM, gary artim <gartim < at > gmail.com> wrote:
Hi --

Getting about 41.6/MBs and hoping for closer to the max (120MB). I
tried maximum file sizes of 5, 8, 12GB -- 12GB the best the others
where about 35/MBs. Any advise welcomed...should I look at max/min
block sizes?
most of the data is big, genetics data -- filesizes avg in the 500/MB
to 3-4/GB -- looking at a growth from 4TB to 15TB in the next 2 years.

run results and bacula-sd.conf and bacula-dir.conf below...

thanks
-- gary

Run:
===

 Build OS:               x86_64-redhat-linux-gnu redhat
 JobId:                  5
 Job:                    Prodbackup.2011-11-29_19.32.42_05
 Backup Level:           Full
 Client:                 "bacula-fd" 5.0.3 (04Aug10)
x86_64-redhat-linux-gnu,red


hat,
 FileSet:                "FileSetProd" 2011-11-29 19:32:42
 Pool:                   "FullProd" (From Job FullPool override)
 Catalog:                "MyCatalog" (From Client resource)
 Storage:                "LTO-4" (From Job resource)
 Scheduled time:         29-Nov-2011 19:32:26
 Start time:             29-Nov-2011 19:32:45
 End time:               29-Nov-2011 21:15:53
 Elapsed time:           1 hour 43 mins 8 secs
 Priority:               10
 FD Files Written:       35,588
 SD Files Written:       35,588
 FD Bytes Written:       257,543,090,368 (257.5 GB)
 SD Bytes Written:       257,548,502,159 (257.5 GB)
 Rate:                   41619.8 KB/s
 Software Compression:   None
 VSS:                    no
 Encryption:             no
 Accurate:               no
 Volume name(s):         f03
 Volume Session Id:      1
 Volume Session Time:    1322622337
 Last Volume Bytes:      257,740,342,272 (257.7 GB)
 Non-fatal FD errors:    0
 SD Errors:              0
 FD termination status:  OK
 SD termination status:  OK
 Termination:            Backup OK

bacula-sd.conf:
==========

Autochanger {
 Name = Autochanger
 Device = LTO-4
 Changer Command = "/usr/libexec/bacula/mtx-changer %c %o %S %a %d"
 Changer Device = /dev/changer
}
Device {
 Name = LTO-4
 Media Type = LTO-4
 Archive Device = /dev/nst0
 AutomaticMount = yes;               # when device opened, read it
 AlwaysOpen = yes;
 RemovableMedia = yes;
 RandomAccess = no;
 Maximum File Size = 12GB
 Autochanger = yes
 Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
 Alert Command = "sh -c 'smartctl -H -l error %c'"
}
bacula-dir.conf:
==========

Job {
       Name = "Prodbackup"
       Client = bacula-fd
       FileSet = "FileSetProd"
       Schedule = "ScheduleProd"
       Write Bootstrap = "/var/spool/bacula/%c.bsr"
       Full Backup Pool = FullProd
       Incremental Backup Pool = IncrProd
       Differential Backup Pool = DiffProd
       Storage =  LTO-4
       Type = Backup
       Level = Incremental
       Pool = IncrProd
       Priority = 10
       Messages = Standard
}
FileSet {
       Name = "FileSetProd"
       Include {
               Options {
                       WildFile = "*.OLD"
                       WildFile = "*.o"
                       WildFile = "*.bak"
                       exclude = yes
               }
               File = /my/home/xxxxxxx
       }
       Exclude {
               File = /my/home/tmp
       }
}
Schedule {
       Name = "ScheduleProd"
       Run = Full 1st sun at 16:05
       Run = Pool {
       Name = FullProd
       Label Format = "FullProd"
       Pool Type = Backup
       Recycle = yes           # Bacula can automatically recycle Volumes
       AutoPrune = yes         # Prune expired volumes
       Volume Retention = 10 years
       Maximum Volume Jobs = 1
}

Pool {
       Name = DiffProd
       Label Format = "DiffProd"
       Pool Type = Backup
       Recycle = yes
       AutoPrune = yes
       Volume Retention =  40 days
       Maximum Volume Jobs = 1
}

Pool {
       Name = IncrProd
       Label Format = "IncrProd"
       Pool Type = Backup
       Recycle = yes
       AutoPrune = yes
       Volume Retention =  10 days
       Maximum Volume Jobs = 1
}
Differential 2nd-5th sun at 16:05
       Run = Incremental mon-sat at 16:05
}

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Bacula-users mailing list
Bacula-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Display posts from previous:
Reply to topic Page 1 of 3
Goto page 1, 2, 3  Next
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
  


Magic SEO URL for phpBB