Backup Central

I'd really appreciate any ideas you guys have on this... I'm stuck...

We are using backupPC to backup about 20 machines in the office. Of these,
one machine is a large raid array holding millions of development files
(lots of kernel trees, cvs branches, etc) and comprising of about 200GB.
Over the past few months, backupPC has slowly but steadily gone from taking
1 entire day to do a full backup of this array to taking roughly 3. There
hasn't been anywhere near a 3x increase in the number/size of files on the
array...

The backupPC drive is a 250GB ata100 internal drive. We are using reiserfs
with the r5 hash mounted with noatime. I'm reasonably certain this problem
has something to do with that filesystem we use... What I can't figure out
is why the filesystem's performance has degraded so much. It was my
understanding that reiserfs handled lots of files/links of various sizes
quite well...

The cpu is never a bottleneck when backuppc does anything, be it a
backup/link/nightly/etc. When I call the tar command directly and grab the
raid array's files (over rsh), but dump the output to null, a "full
backup" takes roughly 8 hours. This tells me the raid array is speedy and
not the problem. Also, when I try doing a full backup with a spare
computer and fresh backuppc setup (also with reiserfs), it takes roughly a
day. This again makes me think our filesystem has degraded tremendously
with the addition of more and more hard links/new files... even though it
really shouldn't...

Here's an interesting part I think may be key: When I go into a leaf cpool
directory (say cpool/a/b/c) and run 'find', it takes about .5 seconds to
display the ~1000 files. Fine. However if I try and run 'stat *', it takes
far too long, even for tiny files. It takes about 45 seconds to do a 'stat
*' or 'ls' in one of these directories. (assuming no caching has already
happened). This is where the root of the slowdown is. If it takes that
long to stat each of these tiny files, backuppc can not blaze through
hundreds of tiny files per second like it used to.

I really appreciate any help/suggestions on this.

Thanks,

-Ross Skaliotis


-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 06/07 03:28 , Ross Skaliotis wrote:
The backupPC drive is a 250GB ata100 internal drive. We are using reiserfs
with the r5 hash mounted with noatime. I'm reasonably certain this problem
has something to do with that filesystem we use... What I can't figure out
is why the filesystem's performance has degraded so much. It was my
understanding that reiserfs handled lots of files/links of various sizes
quite well...

Try asking the reiserfs list?
(not that it isn't a valid question here; but they might have some ideas).

I found that 3ware RAID controllers did wonders for disk speed; and
honest-to-god SCSI on a fast CPU works better yet. (tho I haven't really had
a chance to test all the variables in isolation).

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Hi,

On Mon, 2004-06-07 at 21:28, Ross Skaliotis wrote:
I'd really appreciate any ideas you guys have on this... I'm stuck...

<snip>

Here's an interesting part I think may be key: When I go into a leaf cpool
directory (say cpool/a/b/c) and run 'find', it takes about .5 seconds to
display the ~1000 files. Fine. However if I try and run 'stat *', it takes
far too long, even for tiny files. It takes about 45 seconds to do a 'stat
*' or 'ls' in one of these directories. (assuming no caching has already
happened). This is where the root of the slowdown is. If it takes that
long to stat each of these tiny files, backuppc can not blaze through
hundreds of tiny files per second like it used to.

I really appreciate any help/suggestions on this.

I've got more or less the same experience with reiserfs. Reiserfs v3
seems to have serious performance issues running stat() calls on
filesystems with a lot of files or directories with a lot of files, not
exactly sure which, I suspect the former.
Btw: are you sure 'ls' takes long? Or is 'ls' fast and 'ls -l' slow?
Without -l 'ls' shouldn't do stat() calls afaik.

I used to have an ext3 fs, which rendered backuppc backups really,
really slow. Switching to reiserfs solved that. For me the trouble
surfaced with the running of backuppc_nightly. Using the patch with the
external program to run that in inode order solved that mostly, so
basically I don't have any performance issues at the moment.

My pool consists of 83.63GB comprising 2758760 files and 4369
directories. A lot of those 2.7 M files are very small (2 KB).

I had tried jfs but that was a disaster. Became extremely slow, even
worse than ext3. I haven't tried reiser v4 (yet), so no idea if that
will improve matters. Perhaps people running xfs can share their
experiences?

Thanks,

-Ross Skaliotis

Regards,

Guus




-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Mon, 2004-06-07 at 15:58, Guus Houtzager - Luna.nl wrote:

I've got more or less the same experience with reiserfs. Reiserfs v3
seems to have serious performance issues running stat() calls on
filesystems with a lot of files or directories with a lot of files, not
exactly sure which, I suspect the former.
Btw: are you sure 'ls' takes long? Or is 'ls' fast and 'ls -l' slow?
Without -l 'ls' shouldn't do stat() calls afaik.

Just guessing, I'd say the inodes involved are splattered all over
the disk and the time involved is mostly waiting for the disk
head to seek to each one. I'm not sure what you can do about
that on a very active filesystem other than add RAM to cache
more of them.

---
Les Mikesell
les < at > futuresource.com




-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Guus Houtzager - Luna.nl wrote:

I used to have an ext3 fs, which rendered backuppc backups really,
really slow. Switching to reiserfs solved that. For me the trouble
surfaced with the running of backuppc_nightly. Using the patch with the
external program to run that in inode order solved that mostly, so
basically I don't have any performance issues at the moment.

What kernel were you using? I wonder if ext3 with the dir_index feature
turned on (available in 2.6 kernels) helps things... Right now my
backuppc servers are using reiserfs.

-Dave



-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 06/07 04:27 , Les Mikesell wrote:
Just guessing, I'd say the inodes involved are splattered all over
the disk and the time involved is mostly waiting for the disk
head to seek to each one. I'm not sure what you can do about
that on a very active filesystem other than add RAM to cache
more of them.

this is one of the places where SCSI (actually, drives that can do Tagged
Command Queueing) win.
TCQ lets the drive re-order the sequence of reads from the disk; so the
firmware can optimize the order in which tracks are read, possibly saving
several rotations of the media and a number of seeks.

so this is why SCSI wins in a lot of cases, when trying to process a lot of
requests all at once.

for some fascinating discussion of why SCSI often costs more, read this:
http://www.usenix.org/publications/library/proceedings/fast03/tech/full_papers/anderson/anderson_html/index.html

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Hi,

On Tue, 2004-06-08 at 00:21, David Rees wrote:
Guus Houtzager - Luna.nl wrote:

I used to have an ext3 fs, which rendered backuppc backups really,
really slow. Switching to reiserfs solved that. For me the trouble
surfaced with the running of backuppc_nightly. Using the patch with the
external program to run that in inode order solved that mostly, so
basically I don't have any performance issues at the moment.

What kernel were you using? I wonder if ext3 with the dir_index feature
turned on (available in 2.6 kernels) helps things... Right now my
backuppc servers are using reiserfs.

That was on a 2.4 kernel. I wasn't planning on running 2.6 on a
production server but I needed the performance increase to keep things
acceptable.

-Dave

Regards,

Guus



-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Hi,

On Tue, 2004-06-08 at 00:57, Carl Wilhelm Soderstrom wrote:
On 06/07 04:27 , Les Mikesell wrote:
Just guessing, I'd say the inodes involved are splattered all over
the disk and the time involved is mostly waiting for the disk
head to seek to each one. I'm not sure what you can do about
that on a very active filesystem other than add RAM to cache
more of them.

this is one of the places where SCSI (actually, drives that can do Tagged
Command Queueing) win.
TCQ lets the drive re-order the sequence of reads from the disk; so the
firmware can optimize the order in which tracks are read, possibly saving
several rotations of the media and a number of seeks.

so this is why SCSI wins in a lot of cases, when trying to process a lot of
requests all at once.

for some fascinating discussion of why SCSI often costs more, read this:
http://www.usenix.org/publications/library/proceedings/fast03/tech/full_papers/anderson/anderson_html/index.html

Yeah I know.... I had hoped to avoid some of those issues by using a
3ware card, but that obviously didn't cure it all. Those 3ware cards do
deliver acceptable performance luckily.
I think SATA-II implements TCQ aswell (WD raptor drives do) so maybe
people on a tight budget (ie. all of us ) will get decent
performance when sata-ii is released.

Regards,

Guus



-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Tue, 2004-06-08 at 02:50, Guus Houtzager - Luna.nl wrote:

for some fascinating discussion of why SCSI often costs more, read this:
http://www.usenix.org/publications/library/proceedings/fast03/tech/full_papers/anderson/anderson_html/index.html

Yeah I know.... I had hoped to avoid some of those issues by using a
3ware card, but that obviously didn't cure it all. Those 3ware cards do
deliver acceptable performance luckily.

SCSI is fine when you have lots of concurrent operations but I'd
expect backuppc to wait for the results of each stat before
issuing the next command so you are going to wait for head
motion regardless of the hardware. Linux is pretty aggressive
about caching in RAM and should know more than SCSI drives
about reordering seeks in the queue. I'd max out the RAM
in the box before getting more expensive drives. With a
gig or two you might cache all the inodes.

---
Les Mikesell
les < at > futuresource.com




-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

I got a useful response to my question over at the reiserfs list:

Reiserfs allocates inodes dynamically, so they're spread all over the
filesystem. So, doing a 'ls -l' in a directory causes major head-seeking.
Other filesystems, including XFS apparently, group inodes together on the
disk. I would assume this solves the problem at the expense of the
filesystem having a fixed number of inodes after creation.

I'm going to give xfs a shot. I'll post my results.

-Ross

On Tue, 8 Jun 2004, Les Mikesell wrote:

On Tue, 2004-06-08 at 02:50, Guus Houtzager - Luna.nl wrote:

for some fascinating discussion of why SCSI often costs more, read this:
http://www.usenix.org/publications/library/proceedings/fast03/tech/full_papers/anderson/anderson_html/index.html

Yeah I know.... I had hoped to avoid some of those issues by using a
3ware card, but that obviously didn't cure it all. Those 3ware cards do
deliver acceptable performance luckily.

SCSI is fine when you have lots of concurrent operations but I'd
expect backuppc to wait for the results of each stat before
issuing the next command so you are going to wait for head
motion regardless of the hardware. Linux is pretty aggressive
about caching in RAM and should know more than SCSI drives
about reordering seeks in the queue. I'd max out the RAM
in the box before getting more expensive drives. With a
gig or two you might cache all the inodes.

---
Les Mikesell
les < at > futuresource.com




-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/



-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 06/08 09:50 , Guus Houtzager - Luna.nl wrote:
I think SATA-II implements TCQ aswell (WD raptor drives do) so maybe
people on a tight budget (ie. all of us ) will get decent
performance when sata-ii is released.

I think TCQ is not a required part of the SATA-II spec; so there might very
well be some drives that have it, and some that don't.

WD Raptors are indeed very cool tho. I'm definitely going for them, if I
build a new box for myself in the near future.

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 06/08 07:40 , Les Mikesell wrote:
SCSI is fine when you have lots of concurrent operations but I'd
expect backuppc to wait for the results of each stat before
issuing the next command so you are going to wait for head
motion regardless of the hardware.

I'm not familiar with Backuppc's internal architecture, so I don't know.
This doesn't sound like a very efficient way to do things, if it were true
tho. More likely; perl just tells the kernel "read/write these files to/from
disk". The kernel then worries about how to best get the data to/from the
disk.

tho maybe I'm missing what you're getting at here.

Linux is pretty aggressive
about caching in RAM and should know more than SCSI drives
about reordering seeks in the queue.

can't. remember that what the modern drive presents to the OS, has little or
no relation to the way things are physically laid out on disk. the OS
doesn't control head seeks, because it only has an abstracted notion of
where things are on disk. the drive firmware itself handles knowing where
stuff is (otherwise transparent block reallocation wouldn't work).

I'd max out the RAM
in the box before getting more expensive drives. With a
gig or two you might cache all the inodes.

this is definitely a good idea. at times, Backuppc will take all the memory
and CPU you can give it.

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Tue, 2004-06-08 at 09:37, Ross Skaliotis wrote:
I got a useful response to my question over at the reiserfs list:

Reiserfs allocates inodes dynamically, so they're spread all over the
filesystem. So, doing a 'ls -l' in a directory causes major head-seeking.
Other filesystems, including XFS apparently, group inodes together on the
disk. I would assume this solves the problem at the expense of the
filesystem having a fixed number of inodes after creation.

I'm going to give xfs a shot. I'll post my results.

Out of curiosity: how much RAM do you have? Linux 2.2 kernels used
to have an inode-max setting. I'm not sure if it is completely
dynamic in 2.4+ or if there is some other way to control the
cache size.

---
Les Mikesell
les < at > futuresource.com




-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Tue, 2004-06-08 at 09:52, Carl Wilhelm Soderstrom wrote:

I'm not familiar with Backuppc's internal architecture, so I don't know.
This doesn't sound like a very efficient way to do things, if it were true
tho. More likely; perl just tells the kernel "read/write these files to/from
disk". The kernel then worries about how to best get the data to/from the
disk.

tho maybe I'm missing what you're getting at here.

Reiserfs does just fine at read/write/create. The slow operation that
is essential for backuppc is the nightly walk through the gazillion
pool files checking for ones that only have one link, meaning they
have been deleted from all of the individual pc backups and are
no longer needed. About the only way you can do this is to read
through the directory (which happens quickly as noted by a plain
'ls'), then stat each file, one-by-one and delete the ones with
only one link. You can re-order writes because they are buffered
and queue, but reads (and stat()'s) always wait for the results
so nothing the drive can do will help. Someone has already noted
that this goes much faster if you read the whole directory first,
then do the stats in inode-number which is about the best you
can do.


Linux is pretty aggressive
about caching in RAM and should know more than SCSI drives
about reordering seeks in the queue.

can't. remember that what the modern drive presents to the OS, has little or
no relation to the way things are physically laid out on disk. the OS
doesn't control head seeks, because it only has an abstracted notion of
where things are on disk. the drive firmware itself handles knowing where
stuff is (otherwise transparent block reallocation wouldn't work).

Sorting the queue and re-ordering has been in unix filesystem code for
decades. Maybe it has been removed from Linux recently because of the
increasing intelligence of devices, but I'd expect ascending values
to still represent the same direction of motion in nearly all cases
and head motion is about 1000 times slower than any other operation.

I'd max out the RAM
in the box before getting more expensive drives. With a
gig or two you might cache all the inodes.

this is definitely a good idea. at times, Backuppc will take all the memory
and CPU you can give it.

In this case it may also need some help with kernel tuning to make it
cache all the inodes on a 250 gig drive. Does anyone know how to
change that in 2.4 or 2.6 kernels.

---
Les Mikesell
les < at > futuresource.com




-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 06/08 10:32 , Les Mikesell wrote:
tho maybe I'm missing what you're getting at here.

Reiserfs does just fine at read/write/create. The slow operation that
is essential for backuppc is the nightly walk through the gazillion
pool files checking for ones that only have one link, meaning they
have been deleted from all of the individual pc backups and are
no longer needed.

ah. I see.

Sorting the queue and re-ordering has been in unix filesystem code for
decades.

certainly reads and writes are 'clustered' together for performance reasons;
and the I/O queue is re-organized in that way.

Maybe it has been removed from Linux recently because of the
increasing intelligence of devices, but I'd expect ascending values
to still represent the same direction of motion in nearly all cases

not necessarily.
like I said; what the drive shows you, and what actually happens on the
platters are two very different things, and are not always congruent. the
most trivial example is a drive allocating a new block somewhere else on the
platter, to take the place of one where corruption has occurred for some
reason.

also, look at Cylinder/Head/Sector figures... those plainly don't apply when
the drive swears it has 65535 cylinders, 16 heads, and 255 sectors... but
there's only one physicall platter and head.

there usually is some relation tho; which is why Mac OSX's 'Hot File
Clustering' works... it moves frequently-used files near to the beginning of
the partition (which is *usually* near the outside of the disk), for better
I/O performance.

with the older MFM and RLL drives, things were probably different.

--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
This SF.Net email is sponsored by: GNOME Foundation
Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event.
GNOME Users and Developers European Conference, 28-30th June in Norway
http://2004/guadec.org
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/