Backup Central

On Wed, 21 Jul 2004 at 12:00am, Stefan G. Weichinger wrote

Hi, Kris,

on Dienstag, 20. Juli 2004 at 23:14 you wrote to amanda-users:

KV> The box is running redhat 9 with 2.4.20 kernel and ext3 filesystem.
KV> Below is the most recent sendsize.debug

KV> sendsize[27747]: time 11114.784: Total bytes written: 429923983360 (400GB, 37MB/s)

ok ...

KV> sendsize[27747]: time 18815.342: Total bytes written: 69237104640 (64GB, 8.6MB/s)

not ok ---

Actually, both of those are *very* slow. Remember, those are estimates,
and they "write" to /dev/null. When tar does that, it doesn't actually
read the bits off the spindles, it just stats the files. On my big
(Linux) file servers, I get estimate rates on the order or GB/s -- up to
80GB/s in some cases. One difference, though, is that I'm using XFS.

I would:

- split venus:/home into several DLEs (via exclude/include)

I don't know that this will help.

- exclude unnecessary subdirs/files (./qa/build-main-branch-rfexamples/rfexamples-20040719/customer_test/Nestoras4/Freq
Domain/Linux_temp-g seems like a candidate to me)

That's a good idea, of course.

This would spawn several sendsize-processes in parallel ...

Actually, if you look at sendsize*debug, the estimates occur sequentially,
not in parallel. ICBW, but that's how it looks to me.

Kris, I think that you need to some performance testing/optimizing of your
system. What controllers are you using? Have you tested with bonnie++
and/or tiobench? Are there mount parameters to ext3 you can play with
(data=writeback comes to mind)? You may also want to spend some time on
bugzilla and see if there's some other kernel-foo you can apply to RH's
kernel (I assume you're using RH's kernel -- if not vanilla 2.4.20 is
awfully old...) to speed up disk stuff.

--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

On Tuesday 20 July 2004 19:51, Mike Fedyk wrote:
Kris Vassallo wrote:
The disks in the venus box are all SATA 150 drives, SCSI is way
out of the price range for this amount of space. If venus is the
machine that is taking forever to do the estimates, is it possible
that 1. estimates start on all machines, 2. the estimates finish
on the smaller remote file systems first; these systems begin to
dump. 3. now along with the backup server trying to do an estimate
on its own disks, its also dealing with a dump coming in from
remote systems and all of this together is slowing it down? Do I
have any valid ideas here?

On my Amanda 2.4.4p2 from Fedora Core 2, the dumps wait until all
estimates finish before starting.

Is there an option to change that?

Unforch no. The way amanda works, amanda requires full knowledge of
what she is supposed to do in order to work out a scenario that will
fit the available storage space. This only takes a couple of seconds
once all the estimates are in and then the dumpers are launched, one
per spindle number.

You would be better off to put in some spindle numbers so that amanda
can know for sure that she can access each disk exclusively,
otherwise she may launch 3 or more dumpers all attacking the same
disk(array) which will lead to some time loss due to head thrashing
as the heads seeek back and forth to satisfy more than one dumper.

I'd think each raid array should have its own spindle number. I don't
run any raids here at home, but I do give every disk that amanda
touches its own spindle number even if its in another machine playing
client. Each partition on that disk has the same spindle number in
the disklist, and it did cut my times down by at least a half an hour
overall in an approximately 4 hour run.

Also, as has been noted here, the lack of a holding disk can slow it
down quite a bit once the backups are actually being done because the
drive cannot service more than one data stream at a time. Without
that holding disk buffer, any cpu based compression may slow it down
enough that the drive will do some "shoe shining" as it stops for
lack of data, rewinds a bit and then starts back up almost instantly
as its buffer fills. Atapi/ide disks big enough to buffer what you
want to do aren't too much over a $100 bill these days, or often come
with a rebate form that reduces them to that price range.

--
Cheers, Gene
There are 4 boxes to be used in defense of liberty.
Soap, ballot, jury, and ammo.
Please use in that order, starting now. -Ed Howdershelt, Author
Additions to this message made by Gene Heskett are Copyright 2004,
Maurice E. Heskett, all rights reserved.

Hi, Joshua Baker-LePain,

on Mittwoch, 21. Juli 2004 at 02:28 you wrote to amanda-users:

JBL> On Wed, 21 Jul 2004 at 12:00am, Stefan G. Weichinger wrote

KV> sendsize[27747]: time 11114.784: Total bytes written: 429923983360 (400GB, 37MB/s)

ok ...

KV> sendsize[27747]: time 18815.342: Total bytes written: 69237104640 (64GB, 8.6MB/s)

not ok ---

JBL> Actually, both of those are *very* slow. Remember, those are estimates,
JBL> and they "write" to /dev/null. When tar does that, it doesn't actually
JBL> read the bits off the spindles, it just stats the files. On my big
JBL> (Linux) file servers, I get estimate rates on the order or GB/s -- up to
JBL> 80GB/s in some cases. One difference, though, is that I'm using XFS.

Yes, I didn't think of the fact that this is estimating. My first "ok"
was meant as "Ok, but ..." ;-)

I think that one of the main reasons for the slow estimates here is
the fact that there seem to be loads of small files (think of the
CVS).

So tar has to stat all those many files and slows down ...

This in "cooperation" with filesystem and maybe kernel-related issues.

Converting to Reiser or XFS would help maybe, on the other hand you
would get more cpu-load with Reiser and such.

I would:

- split venus:/home into several DLEs (via exclude/include)

JBL> I don't know that this will help.

If there are high-level-subdirs for CVS and others that are not, Kris
could define specific dumptypes to speed things up for the DLEs, using
different exclude-patterns for different types of data.

This would spawn several sendsize-processes in parallel ...

JBL> Actually, if you look at sendsize*debug, the estimates occur sequentially,
JBL> not in parallel. ICBW, but that's how it looks to me.

Right again. I should not post after 14 hours of work ... I just
thought of the output of amstatus and didn't do the lookup you did.

But I will be quiet for another 14 hours now ;-)

best regards,
Stefan

Joshua Baker-LePain wrote:
On Wed, 21 Jul 2004 at 12:00am, Stefan G. Weichinger wrote
[...]

Kris, I think that you need to some performance testing/optimizing of your
system. What controllers are you using? Have you tested with bonnie++
and/or tiobench? Are there mount parameters to ext3 you can play with
(data=writeback comes to mind)? You may also want to spend some time on
bugzilla and see if there's some other kernel-foo you can apply to RH's
kernel (I assume you're using RH's kernel -- if not vanilla 2.4.20 is
awfully old...) to speed up disk stuff.

Try the most simple tests like "hdparm -tT" if it's possible on that RAID
array, also try to doublecheck that all discs are running in DMA mode
(hdparm -d) or check the info from your RAID controller.

As I said I'm not sure if it works on your RAID setup, if not you'll need
to check the performance with the other tools mentioned.

/Andreas

On Wed, 21 Jul 2004, Paul Bijnens wrote:
If you install dump for ext2, then you should also try out that one.
Dump takes only a few seconds or minutes compared to gnutar for such
filesystems.

I thought you do not want to use dump on Linux, since it's unsafe?

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert < at > linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

On Tue, 2004-07-20 at 17:28, Joshua Baker-LePain wrote: On Wed, 21 Jul 2004 at 12:00am, Stefan G. Weichinger wrote

Hi, Kris,

on Dienstag, 20. Juli 2004 at 23:14 you wrote to amanda-users:

KV> The box is running redhat 9 with 2.4.20 kernel and ext3 filesystem.
KV> Below is the most recent sendsize.debug

KV> sendsize[27747]: time 11114.784: Total bytes written: 429923983360 (400GB, 37MB/s)

ok ...

KV> sendsize[27747]: time 18815.342: Total bytes written: 69237104640 (64GB, 8.6MB/s)

not ok ---

Actually, both of those are *very* slow. Remember, those are estimates,
and they "write" to /dev/null. When tar does that, it doesn't actually
read the bits off the spindles, it just stats the files. On my big
(Linux) file servers, I get estimate rates on the order or GB/s -- up to
80GB/s in some cases.  One difference, though, is that I'm using XFS. Maybe I am missing something here, but do I need to have the estimates? Does something depend on this? If I just ripped this out somehow (no idea how I would go about doing this) then waiting 7 hours for an estimate would no longer be a problem... right?
I would:

- split venus:/home into several DLEs (via exclude/include)

I don't know that this will help.

- exclude unnecessary subdirs/files (./qa/build-main-branch-rfexamples/rfexamples-20040719/customer_test/Nestoras4/Freq
Domain/Linux_temp-g seems like a candidate to me)

That's a good idea, of course.

This would spawn several sendsize-processes in parallel ...

Actually, if you look at sendsize*debug, the estimates occur sequentially,
not in parallel. ICBW, but that's how it looks to me.

Kris, I think that you need to some performance testing/optimizing of your
system.  What controllers are you using? I am using a 3ware 12 port RAID card. Initially when I set the card up I did some benchmarking and the read/write speeds to the drives being backed up were excellent, what they were doesn't come to mind at this moment. I am trying not to take this machine down for any reason as it is a HUGE hassle, but it looks like I am going to have to. Have you tested with bonnie++
and/or tiobench? Are there mount parameters to ext3 you can play with
(data=writeback comes to mind)? I tried using noatime for the run last night and that didn't seem to help worth squat. It seems as if i set data=writeback I might as well turn off journaling completely because there is no guarantee that old data from the journal wont end up getting written back to the disk in the event of the system going down. However, would this really speed things up? Dramatically? You may also want to spend some time on
bugzilla and see if there's some other kernel-foo you can apply to RH's
kernel (I assume you're using RH's kernel -- if not vanilla 2.4.20 is
awfully old...) to speed up disk stuff. I am using redhat's kernel 2.4.20-20.9smp

On Tue, 2004-07-20 at 14:37, Stefan G. Weichinger wrote: Hi, Frank,

on Dienstag, 20. Juli 2004 at 07:41 you wrote to amanda-users:

420GB is not the total amount per night. Something is bogging this down
though and I don't know what. I am not using holding disks because the
majority of data is being backed up from one set of disks to another on
the same machine. This one machine has a set of RAID 10 disks. These
disks are backed up by amanda and put onto a set of RAID 5 disks.

FS> OK, I was assuming a different setup. Having a holding disk would let
FS> you run multiple dumps in parallel. Wouldn't help much (if any) when
FS> its all on one machine, but can really speed up your overall time if
FS> you have multiple clients.

Given Joshua's note about having data and backup on the same
controller I would just suggest adding a cheap'n'huge IDE-drive (and
controller, if necessary) for a holdingdisk.

This will speed things up locally, too. Think parallel dumping AND the
fact that people could access data at ~normal speed even while the
holdingdisk is still feeding the tape (while this is still not the
solution here, estimates ain't done on the holdingdisk ....)

Having a separate holdingdisk is never a bad thing with AMANDA IMHO.

As far
as assigning spindle #s goes I don't quite understand why I would set
that. I have inparallel set to 4 and then didn't define maxdumps, so I
would assume that not more than 1 dumper would get started on a machine
at once. Am I getting this right?

FS> I think maxdumps defaults to 2 but I may be wrong (someone else should
FS> jump in here).

It is 10. ( grep -r "define MAXDUMPS" amanda-2.4.4-p3 )

Estimate Time (hrs:min) 7:30

FS> Here's your runtime problem, 7.5 hours for estimates .

Yep.

Run Time (hrs:min) 10:35
Dump Time (hrs:min) 2:52 0:29 2:23

FS> Three hours for dumps doesn't seem too bad. It could probably
FS> be improved some, but the estimates are what's killing you.

Yep again.

FS> As for the estimates, are you using dump or tar? Look in the
FS> *debug files on the clients and see which one was taking all the time
FS> (I'm guessing venus since it looks like you did a force on bda1).
FS> Does that filesystem have millions of small files?
FS> I'm not sure of the best way to speed up estimates, other than a
FS> faster disk system. Perhaps someone else on the list has some ideas.

My idea is to request more details here.

Relevant dumptype-definition, local/remote-info, df venus:/home, etc FYI:
define dumptype hard-disk-tar-fast-compress {
global
comment "Back up to hard disk instead of tape - using tar with compress client fast"
compress client fast
program "GNUTAR"
}

df /home
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 721091784 438381716 246080668 65% /home

DLE: venus.xxxx /home hard-disk-tar-fast-compress
Not sure what you mean about local/remote info here.

On Wed, 21 Jul 2004 at 12:17pm, Kris Vassallo wrote

On Tue, 2004-07-20 at 17:28, Joshua Baker-LePain wrote:

Actually, both of those are *very* slow. Remember, those are estimates,
and they "write" to /dev/null. When tar does that, it doesn't actually
read the bits off the spindles, it just stats the files. On my big
(Linux) file servers, I get estimate rates on the order or GB/s -- up to
80GB/s in some cases. One difference, though, is that I'm using XFS.

Maybe I am missing something here, but do I need to have the estimates?
Does something depend on this? If I just ripped this out somehow (no
idea how I would go about doing this) then waiting 7 hours for an
estimate would no longer be a problem... right?

You need to have some form of the estimates. Amanda uses them to decide
what backups to run each night (whether to demote or promote certain DLEs,
e.g.). Some folks on the list have talked of replacing tar for estimates
with some more efficient method. But that involves a script that detects
whether tar is being run to do estimates or the actual backups, and runs
an appropriate command. Not trivial -- the discussion should be in the
archives.

Kris, I think that you need to some performance testing/optimizing of your
system. What controllers are you using?

I am using a 3ware 12 port RAID card. Initially when I set the card up I
did some benchmarking and the read/write speeds to the drives being
backed up were excellent, what they were doesn't come to mind at this
moment. I am trying not to take this machine down for any reason as it
is a HUGE hassle, but it looks like I am going to have to.

One of my 3ware based systems (single 7500-8, hardware RAID5) is much
slower than the others doing the estimates. I *think* it's due to large
numbers of small files -- at least, it helped a lot when I had the user
clean up. But it was never as bad as your case -- it's a 1TB filesystem,
and estimates were taking about 1.5 hours at worst. Also, it's using XFS.

Have you tested with bonnie++
and/or tiobench? Are there mount parameters to ext3 you can play with
(data=writeback comes to mind)?

I tried using noatime for the run last night and that didn't seem to
help worth squat. It seems as if i set data=writeback I might as well
turn off journaling completely because there is no guarantee that old
data from the journal wont end up getting written back to the disk in
the event of the system going down. However, would this really speed
things up? Dramatically?

I don't know, as I don't use ext3 for this type of stuff. My suspicion,
actually, is no -- the problem isn't throughput, just the huge number of
files.

You may also want to spend some time on
bugzilla and see if there's some other kernel-foo you can apply to RH's
kernel (I assume you're using RH's kernel -- if not vanilla 2.4.20 is
awfully old...) to speed up disk stuff.

I am using redhat's kernel 2.4.20-20.9smp

Is it too disruptive to just reboot the system? It'd be nice to try a
couple of other kernels, like a vanilla 2.4.26. Also, I'd ask on a redhat
list and/or an ext2/3 list about any kernel tweaks to get better
performance for lots of small files.

--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

On Tue, 2004-07-20 at 16:05, Frank Smith wrote: --On Tuesday, July 20, 2004 14:35:53 -0700 Kris Vassallo <kris < at > linuxcertified.com> wrote:
NOTES:

driver: WARNING: /tmp: not 102400 KB free.

I overlooked this last night. I've never seen this message myself,
but perhaps it is relevant. Any thoughts, anyone?


I am using tar to do this. The bda1 system is a CVS server which gets
hammered on all day long and does have tons of smaller files as well as
a decent amount of larger ones.

As Stefan mentioned, there are probably subdirectories you could exclude
from the backup to speed things up. You mentioned part of it was used
for CVS, perhaps you can exclude some of the build trees and just backup
the source trees. Scrap the CVS part of it, the CVS repository is on a completely different client and that client isn't taking all that long to backup. However, point well taken, I'm sure that there is plenty of junk that does not need to be backed up. So fixing this would reduce the backup time as a whole but would still leave me with outrageous estimate times on the other server.
The disks in the venus box are all SATA 150 drives, SCSI is way out of
the price range for this amount of space. If venus is the machine that
is taking forever to do the estimates, is it possible that 1. estimates
start on all machines, 2. the estimates finish on the smaller remote
file systems first; these systems begin to dump. 3. now along with the
backup server trying to do an estimate on its own disks, its also
dealing with a dump coming in from remote systems and all of this
together is slowing it down? Do I have any valid ideas here?

Possible, although the estimates write to /dev/null, so the remote
dumps shouldn't be slowing them down unless it's your controller
limiting you and not the disks themselves. You could try commenting
out all the other filesystems in your disklist and see if the estimate
still takes as long. Will do! Is the system otherwise idle when you are running Amanda? If
the disks are fairly active (whether from user activity or perhaps
automated nightly builds) it will slow down your backups considerably. There are nightly builds, however I have set the backup to run during a time at which disk access is minimal.

It could also be kernel related. Our first attempt at Linux
fileservers had problems under heavy load, the sytem would slow to
a crawl (and sometimes appear to hang) under concurrent loads (a
CVS build and an rsync of the filesystem in our case). Moving from
a 2.4 kernel to 2.6 solved the problem completely. Well, if I can't get this going with anything else, then I am going to have to try 2.6. That will be my last resort (along with migrating to a new file system) as there is going to be an insane amount of work involved. I can't just plop in the 2.6 kernel without breaking the module utils and all sorts of other things. This probably means building a similar piece of hardware up and then building the kernel on that to make sure it boots and then replacing the system disk when its ready. That along with maintaining minimal down time and getting screamed at by impatient developers... I can already predict a week without sleep and the headache coming on.. oie!
Frank

-Kris

Frank Smith wrote:

limiting you and not the disks themselves. You could try commenting
out all the other filesystems in your disklist and see if the estimate
still takes as long.


Why do estimates all start and finish at the exact same time for all
volumes on a single client?

Geert Uytterhoeven wrote:

On Wed, 21 Jul 2004, Paul Bijnens wrote:

If you install dump for ext2, then you should also try out that one.
Dump takes only a few seconds or minutes compared to gnutar for such
filesystems.


I thought you do not want to use dump on Linux, since it's unsafe?

Dump should be used on a non-active filesystem. Yes, Linus once
made some comments about dump inherently being unsafe, especially
while he was building the vfs filesystem in the kernel.
In that time dump was also without maintainers, and did contain
quiet a few bugs. That has changed, and dump has considerably
improved. It should not be less safe than dump on solaris.

Moreover, if you do snapshotting using the logical volume manager,
you can make backups on a perfectly quiet filesystem. Using ext3
has a little difficulty last time I tried, that the snapshot filesystem
needs repairing (from the redo logs), and that needs write access
to the filesystem, which you haven't on the snapshot. (That was on
Redhat 9; on RH 8 it did work fine; never tried on fedora)

But, personally I use gnutar, because I have a mixed environment, and
need to able to restore to different OS'es. That means my real life
experience with dump is limited, i.e. I could be wrong :-)


--
Paul Bijnens, Xplanation Tel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM Fax +32 16 397.512
http://www.xplanation.com/ email: Paul.Bijnens < at > xplanation.com
***********************************************************************
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?" ... YES ... Phew ... I'm out *
***********************************************************************

Joshua Baker-LePain wrote:

Is it too disruptive to just reboot the system? It'd be nice to try a


Boo! I sincerely doubt rebooting will help unless there is a kernel
problem.

couple of other kernels, like a vanilla 2.4.26. Also, I'd ask on a redhat
list and/or an ext2/3 list about any kernel tweaks to get better
performance for lots of small files.

Are you using htree (indexed directories)[1]?

I would look into using the LD_PRELOAD library that sorts the directory
entries based on physical layout on disk (sort by inode number).

Mike

[1] tune2fs -l /dev/XXX |grep index

Mike Fedyk wrote:

Why do estimates all start and finish at the exact same time for all
volumes on a single client?



They don't.

The amanda server asks for estimates in one request for all disks and
levels. Then the client runs them, and when finished them all, sends
the reply in one answer.

In the client debugfiles /tmp/amanda/sendsize.*.debug, you find
the start and end time of each of the estimates.


--
Paul Bijnens, Xplanation Tel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM Fax +32 16 397.512
http://www.xplanation.com/ email: Paul.Bijnens < at > xplanation.com
***********************************************************************
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?" ... YES ... Phew ... I'm out *
***********************************************************************