Backup Central

Hi,

Some time ago I had performance problems with backuppc, but those were
solved with a switch from ext3 to reiser 3.
Now, about a month further along, BackupPC_nightly is taking very long,
about 8.5 hours. Because that blocks any backup an restore processes,
this doesn't make me very happy.
I'm afraid my I/O subsystem just isn't fast enough, but then again, my
setup isn't that ridiculously large I think. Is there anything I can do
to speed up this process?

Specs and logfiles:

Duron 1300, 1 GB ram, 3ware 7504 raidcontroller with
4 200 GB Western Digital 7200 rpm disks in raid 5 configuration.
The /var partition is 550 GB large, reiser3.
I tried the suggestion from 3ware to set /proc/sys/vm/min-readahead
en max-read-ahead to 512 instead of 3 and 31, didn't help. I have
mounted /var with the noatime option.
This machine is running Debian Linux, stable with a 2.4.24 kernel.
BackupPC version is 2.0.2.

2004/3/30 03:06:21 Running BackupPC_nightly (pid=18606)
2004/3/30 03:06:22 Pool nightly clean removed 0 files of size 0.00GB
2004/3/30 03:06:22 Pool is 0.00GB, 0 files (0 repeated, 0 max chain, 2
max links), 1 directories
2004/3/30 04:00:01 Next wakeup is 2004/3/30 05:00:00
2004/3/30 05:00:00 Next wakeup is 2004/3/30 06:00:00
2004/3/30 06:00:00 Next wakeup is 2004/3/30 07:00:00
2004/3/30 07:00:00 Next wakeup is 2004/3/30 08:00:00
2004/3/30 08:00:00 Next wakeup is 2004/3/30 09:00:00
2004/3/30 09:00:00 Next wakeup is 2004/3/30 10:00:00
2004/3/30 10:00:00 Next wakeup is 2004/3/30 11:00:00
2004/3/30 11:00:00 Next wakeup is 2004/3/30 12:00:00
2004/3/30 11:48:46 Cpool nightly clean removed 27189 files of size
2.60GB
2004/3/30 11:48:46 Cpool is 78.41GB, 2626854 files (136 repeated, 11 max
chain,
48489 max links), 4369 directories

132 full backups of total size 355.75GB (prior to pooling and
compression)
233 incr backups of total size 65.25GB (prior to pooling and
compression).

BackupPC_link is running fast. Linking a full backup of about a million
files takes about 10 minutes.

Any suggestions?
TIA!

Kind regards,

--
Guus Houtzager Email: guus < at > houtzager.net
PGP fingerprint = 5E E6 96 35 F0 64 34 14 CC 03 2B 36 71 FB 4B 5D
"A)bort, R)etry, I)nfluence with large hammer."



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Tue, 2004-03-30 at 04:44, Guus Houtzager wrote:

Some time ago I had performance problems with backuppc, but those were
solved with a switch from ext3 to reiser 3.
Now, about a month further along, BackupPC_nightly is taking very long,
about 8.5 hours. Because that blocks any backup an restore processes,
this doesn't make me very happy.
I'm afraid my I/O subsystem just isn't fast enough, but then again, my
setup isn't that ridiculously large I think. Is there anything I can do
to speed up this process?


Most Linux distributions come with the 'slocate' package
configured to walk all the local filesystems nightly
building the database for the locate command. On RedHat,
a list of paths to avoid is configured in /etc/updatedb.conf.
I don't know where other distbutions keep it, but it should
help keep that from running over the backuppc files.

Also, make sure that the nightly run actually completes - if
you kill it or reboot the machine it will take longer next
time since there will be more links to delete. If it is
behind, it should be able to catch up by running over a
weekend.

---
Les Mikesell
les < at > futuresource.com




-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 03/30 06:25 , daniel.poelzleithner wrote:
Did you make a benchmark to test if the software raid driver of the
kernel is faster then the 3ware driver ? The most hw-raid 5 cards do not
have a real processor for the checksum code.

3ware cards do have a real processor onboard. It's an i960.
Our tests have shown that 3ware cards beat software raid by a huge margin;
not to mention being a lot less hassle when a drive dies. 3ware cards let
you hot-swap drives, and the system doesn't crash. with software RAID, the
system will often crash when a disk dies (but it does still save your data).

Carl Soderstrom.
--
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Tue, 2004-03-30 at 15:46, Les Mikesell wrote:
On Tue, 2004-03-30 at 04:44, Guus Houtzager wrote:

<snip>

Most Linux distributions come with the 'slocate' package
configured to walk all the local filesystems nightly
building the database for the locate command. On RedHat,
a list of paths to avoid is configured in /etc/updatedb.conf.
I don't know where other distbutions keep it, but it should
help keep that from running over the backuppc files.

Ah, I hadn't thought of that! I have excluded /var/lib/backuppc, I'll
check after tomorrows run if that helps.

Also, make sure that the nightly run actually completes - if
you kill it or reboot the machine it will take longer next
time since there will be more links to delete. If it is
behind, it should be able to catch up by running over a
weekend.

Yes, it finishes correctly every time, just takes a very long time.

---
Les Mikesell
les < at > futuresource.com

Thanks for your suggestions!

Regards,

--
Guus Houtzager Email: guus < at > houtzager.net
PGP fingerprint = 5E E6 96 35 F0 64 34 14 CC 03 2B 36 71 FB 4B 5D
"A)bort, R)etry, I)nfluence with large hammer."



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Tue, 2004-03-30 at 18:25, daniel.poelzleithner wrote:
Guus Houtzager wrote:

<snip>

Try a 2.6 Kernel, it's io sub-system is much faster.

That's also worth a shot. I'm not too keen yet on running 2.6 on a
production machine, but at least it would give me an indication.

An external reiserfs log would increase the speed i think.
Did you make a benchmark to test if the software raid driver of the
kernel is faster then the 3ware driver ? The most hw-raid 5 cards do not
have a real processor for the checksum code. In fact, the software raid
driver is often faster than hardware raid 5, because it contains
optimized assambler code for every processor type.

3ware cards do the XOR-ing in hardware I believe, but I haven't tested
that.

Maybe, a Duron is not the best choice. It doesn't have so much L2 Cache
than a Athlon have, this speeds raid5 up.

True, but my experience is that only the backuppc_dump processes use a
lot of cpu, probably the rsync checksum stuff and the actual ssh
transfers. During the run of backuppc_nightly load is about 2.5, a few %
user, about 10 % system and the rest idle. So I guess the cpu isn't the
issue here. But if I were to build a new server, I would use a more
powerfull cpu.

BackupPC_link is running fast. Linking a full backup of about a million
files takes about 10 minutes.

Maybe backuppc could use optional threads while cleaning up the tree,
because raid systems can handle more than one access to a filesystem at
once. With the 2.6 io-scheduler this could increase the speed.

regards
Daniel

Thanks for your suggestions!

Regards,

--
Guus Houtzager Email: guus < at > houtzager.net
PGP fingerprint = 5E E6 96 35 F0 64 34 14 CC 03 2B 36 71 FB 4B 5D
"A)bort, R)etry, I)nfluence with large hammer."



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Guus Houtzager wrote:


Now, about a month further along, BackupPC_nightly is taking very long,
about 8.5 hours. Because that blocks any backup an restore processes,
this doesn't make me very happy.

Duron 1300, 1 GB ram, 3ware 7504 raidcontroller with
4 200 GB Western Digital 7200 rpm disks in raid 5 configuration.
The /var partition is 550 GB large, reiser3.
I tried the suggestion from 3ware to set /proc/sys/vm/min-readahead
en max-read-ahead to 512 instead of 3 and 31, didn't help. I have
mounted /var with the noatime option.
This machine is running Debian Linux, stable with a 2.4.24 kernel.
BackupPC version is 2.0.2.

Try a 2.6 Kernel, it's io sub-system is much faster.
An external reiserfs log would increase the speed i think.

Did you make a benchmark to test if the software raid driver of the
kernel is faster then the 3ware driver ? The most hw-raid 5 cards do not
have a real processor for the checksum code. In fact, the software raid
driver is often faster than hardware raid 5, because it contains
optimized assambler code for every processor type.

Maybe, a Duron is not the best choice. It doesn't have so much L2 Cache
than a Athlon have, this speeds raid5 up.

BackupPC_link is running fast. Linking a full backup of about a million
files takes about 10 minutes.

Maybe backuppc could use optional threads while cleaning up the tree,
because raid systems can handle more than one access to a filesystem at
once. With the 2.6 io-scheduler this could increase the speed.

regards
Daniel


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Tue, Mar 30, 2004 at 12:44:21, Guus Houtzager wrote:

Hi Guus,

Some time ago I had performance problems with backuppc, but those were
solved with a switch from ext3 to reiser 3.
Now, about a month further along, BackupPC_nightly is taking very long,
about 8.5 hours. Because that blocks any backup an restore processes,
this doesn't make me very happy.
I'm afraid my I/O subsystem just isn't fast enough, but then again, my
setup isn't that ridiculously large I think. Is there anything I can do
to speed up this process?


FWIW, I have the same problem :-/

My hardware is a P4 2.6 GHz 1GB RAM, 3ware 8500-S8 controller.

I'm using it with 7+1 200GB Maxtor SATA drives, the filesystem is XFS.
Everything but BackupPC_nightly is running very fast.

I'll try some of the hints given in this thread and see if it helps.

Lars
--
Lars Anderson mailto:lsa < at > business.aau.dk
Faculty of Social Sciences http://www.business.aau.dk/~lsa/
Aalborg University Voice: +45 96358225, Fax: +45 98153505
Denmark Office: Fib4-117


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Me too. Athlon 2200XP, 512MB Ram, 3ware 7xxx controller.
BackupPC_nightly takes about 7 hours.

My solution is to schedule around it. I run incrementals during the week
that are always finished by midnight, and my weekly full backups
starting on Friday evening. That takes about 24 hours, so I added a line
to the code so that BackupPC_nightly does not run on Saturdays.

FWIW, I have the same problem :-/

My hardware is a P4 2.6 GHz 1GB RAM, 3ware 8500-S8 controller.

I'm using it with 7+1 200GB Maxtor SATA drives, the filesystem is XFS.
Everything but BackupPC_nightly is running very fast.

I'll try some of the hints given in this thread and see if it helps.

Lars



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Interesting,

I have an Athlon XP 2100+ running on a software RAID0 and the pool is
around 240gig. Nightly takes around 5 minutes.

Doug

Scott Hanson wrote:

Me too. Athlon 2200XP, 512MB Ram, 3ware 7xxx controller.
BackupPC_nightly takes about 7 hours.

My solution is to schedule around it. I run incrementals during the
week that are always finished by midnight, and my weekly full backups
starting on Friday evening. That takes about 24 hours, so I added a
line to the code so that BackupPC_nightly does not run on Saturdays.

FWIW, I have the same problem :-/

My hardware is a P4 2.6 GHz 1GB RAM, 3ware 8500-S8 controller.

I'm using it with 7+1 200GB Maxtor SATA drives, the filesystem is XFS.
Everything but BackupPC_nightly is running very fast.

I'll try some of the hints given in this thread and see if it helps.

Lars




-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/






-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

I also have the same problem. I recently switched from ext3 to reiser3.

On ext3 BackupPC_nightly took less than 30 minutes, with reiser3 it takes
more than 120 minutes.

I have already excluded backuppc from the updatedb.conf and slocate daily
cron job.

I'm using software raid 5 (4 drives) on a Pentium4 2.4 with 512MB of RAM
(Fedora Core 1).


Dale Renton


----- Original Message -----
From: "Guus Houtzager" <guus < at > houtzager.net>
To: <backuppc-users < at > lists.sourceforge.net>
Sent: Tuesday, March 30, 2004 6:44 AM
Subject: [BackupPC-users] BackupPC_nightly taking very long


Hi,

Some time ago I had performance problems with backuppc, but those were
solved with a switch from ext3 to reiser 3.
Now, about a month further along, BackupPC_nightly is taking very long,
about 8.5 hours. Because that blocks any backup an restore processes,
this doesn't make me very happy.
I'm afraid my I/O subsystem just isn't fast enough, but then again, my
setup isn't that ridiculously large I think. Is there anything I can do
to speed up this process?

Specs and logfiles:

Duron 1300, 1 GB ram, 3ware 7504 raidcontroller with
4 200 GB Western Digital 7200 rpm disks in raid 5 configuration.
The /var partition is 550 GB large, reiser3.
I tried the suggestion from 3ware to set /proc/sys/vm/min-readahead
en max-read-ahead to 512 instead of 3 and 31, didn't help. I have
mounted /var with the noatime option.
This machine is running Debian Linux, stable with a 2.4.24 kernel.
BackupPC version is 2.0.2.

2004/3/30 03:06:21 Running BackupPC_nightly (pid=18606)
2004/3/30 03:06:22 Pool nightly clean removed 0 files of size 0.00GB
2004/3/30 03:06:22 Pool is 0.00GB, 0 files (0 repeated, 0 max chain, 2
max links), 1 directories
2004/3/30 04:00:01 Next wakeup is 2004/3/30 05:00:00
2004/3/30 05:00:00 Next wakeup is 2004/3/30 06:00:00
2004/3/30 06:00:00 Next wakeup is 2004/3/30 07:00:00
2004/3/30 07:00:00 Next wakeup is 2004/3/30 08:00:00
2004/3/30 08:00:00 Next wakeup is 2004/3/30 09:00:00
2004/3/30 09:00:00 Next wakeup is 2004/3/30 10:00:00
2004/3/30 10:00:00 Next wakeup is 2004/3/30 11:00:00
2004/3/30 11:00:00 Next wakeup is 2004/3/30 12:00:00
2004/3/30 11:48:46 Cpool nightly clean removed 27189 files of size
2.60GB
2004/3/30 11:48:46 Cpool is 78.41GB, 2626854 files (136 repeated, 11 max
chain,
48489 max links), 4369 directories

132 full backups of total size 355.75GB (prior to pooling and
compression)
233 incr backups of total size 65.25GB (prior to pooling and
compression).

BackupPC_link is running fast. Linking a full backup of about a million
files takes about 10 minutes.

Any suggestions?
TIA!

Kind regards,

--
Guus Houtzager Email: guus < at > houtzager.net
PGP fingerprint = 5E E6 96 35 F0 64 34 14 CC 03 2B 36 71 FB 4B 5D
"A)bort, R)etry, I)nfluence with large hammer."



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

I think the deciding factor is the number of files in the pool. I have
two BackupPC installations. One with 80 Gigs and 2500 files takes 5
minutes. The other with 45 Gigs and 2.5 million files takes 8 hours.

Doug Lytle wrote:
Interesting,

I have an Athlon XP 2100+ running on a software RAID0 and the pool is
around 240gig. Nightly takes around 5 minutes.

Doug

--
Scott Hanson Network Services
Netlife GmbH Phone: +49 40 28415 0
Millerntorplatz 1 Fax: +49 40 28415 999
20359 Hamburg, Germany http://www.netlife.de


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Guus Houtzager writes:

Some time ago I had performance problems with backuppc, but those were
solved with a switch from ext3 to reiser 3.
Now, about a month further along, BackupPC_nightly is taking very long,
about 8.5 hours. Because that blocks any backup an restore processes,
this doesn't make me very happy.
I'm afraid my I/O subsystem just isn't fast enough, but then again, my
setup isn't that ridiculously large I think. Is there anything I can do
to speed up this process?

It sounds like BackupPC generally runs faster on reiser3, except
for BackupPC_nightly, which seems to run faster on ext3...

Since a number of people are having problems with BackupPC_nightly
running slow, I propose the following:

- You could choose to run BackupPC_nightly during the day, if
your backups generally run at night. The first entry of
$Conf{WakeupSchedule} is when BackupPC_nightly runs, so
simply re-order the entries so that the first one is when you
want to run BackupPC_nightly.

- I will look into adding a feature to BackupPC-2.1.0beta1 (maybe
target for this weekend), which allows BackupPC_nightly to just
check a part of the pool each night. The only significant issue
is that the pool stats (number of files etc) will be harder to
update correctly. The only cost here is a slight increase in
storage since unused pool files are not deleted sooner.

- Or instead (or in addition) I could set things up so BackupPC_nightly
only does the pool check say every other night. This won't reduce
the run time, but the average impact will be half.

- In a later version (post 2.1.0) I will look into having
BackupPC_nightly fork itself a configurable number of times
so that it runs in parallel on different parts of the pool.
Again, the tricky issue here is collecting the stats output
from each child.

Craig


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Craig Barratt wrote:

Since a number of people are having problems with BackupPC_nightly
running slow, I propose the following:
[...]

Another idea i had with the reiserfs slowdown. Unlink in ext2 is a
really simple remove of one inode. Reiserfs has to update much more
meta-data which can't be so fast i think.

Craig: Could you please trie to patch backuppc, that he only save which
files will be deleted in a list. And then run the unlinks directly one
after another. I think that could solve the problem.

regards
Daniel

"daniel.poelzleithner" writes:

Since a number of people are having problems with BackupPC_nightly
running slow, I propose the following:
[...]

Another idea i had with the reiserfs slowdown. Unlink in ext2 is a
really simple remove of one inode. Reiserfs has to update much more
meta-data which can't be so fast i think.

Craig: Could you please trie to patch backuppc, that he only save which
files will be deleted in a list. And then run the unlinks directly one
after another. I think that could solve the problem.

I'm not sure that this would help that much. From Guus' email:

2004/3/30 11:48:46 Cpool nightly clean removed 27189 files of size 2.60GB

So it only removed 27189 files. I can't imagine that took very long.
I suspect the slow part is scanning the directories and stat()ing
each of his 2.6M files.

I guess one test would be to run BackupPC_nightly twice in a row
(manually, as the BackupPC user, without BackupPC running) and
see if the running time is a lot faster the second time - the
second time there shouldn't be any files to remove. This test
isn't ideal since caching will make the second run faster anyhow.

Craig


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Craig Barratt wrote:

So it only removed 27189 files. I can't imagine that took very long.
I suspect the slow part is scanning the directories and stat()ing
each of his 2.6M files.

I guess one test would be to run BackupPC_nightly twice in a row
(manually, as the BackupPC user, without BackupPC running) and
see if the running time is a lot faster the second time - the
second time there shouldn't be any files to remove. This test
isn't ideal since caching will make the second run faster anyhow.

But accessing to files is much faster because it's a b-tree design

Please run following:

time find /YOUR/CPOOL -type f -exec stat {} >/dev/null \;

How long does this take on large pools ?


I had a idea. Reiserfs uses a hash key for each file. When there a much
files on a filesystem this can result in hash collisions which have a
deep impact on performance. I don't know if this could be solved by more
directory levels.

See:
http://www.namesys.com/mount-options.html

tea is for such number of files the better choice anyway.

regards
Daniel