Backup Central

I started the first backup to an empty directory on a second server.
I'm wandering about the low speed of ca. 1.6MB/s

Both servers are DL380 with dual PIII 833MHz running debian, python2.3,
librsync 0.9.7-1 and rdiff-backup 1.0.0-0.cvs20050819 (thanks to Dean
Gaudet for the debian package).
They are directly connected by gigabit.

I started rdiff-backup this way:
rdiff-backup -v5 --terminal-verbosity=4 --ssh-no-compression /mnt/files/ 192.168.0.1::/mnt/raid/backup/fileserver2/

Everything looks fine:
CPU-usage on the source-server is ca.15% for ssh and ca. 6% for rdiff-backup
CPU-usage on the destination is ca. 18% for sshd and ca. 12% for rdiff-backup
No swap-space is used.

rdiff-backup needs 132s to transfer a 140MB file (1.1MB/s).
While rdiff-backup works on the next file scp transfers the same file in 15s (9.3MB/s)!

Since we want to backup one tera-byte of data this would last more than a week.

Has anyone hints how to speedup rdiff-backup?

Carsten

Carsten Lorenz wrote:

rdiff-backup needs 132s to transfer a 140MB file (1.1MB/s).
While rdiff-backup works on the next file scp transfers the same file in 15s (9.3MB/s)!



What does that mean? scp = 9.3MB/s rdiff-backup = 1.1MB/s

Since we want to backup one tera-byte of data this would last more than a week.



That be bad :)

Has anyone hints how to speedup rdiff-backup?



Well if you want to speed up rdiff-backup using ssh/scp as transport you
might need to look into:

high performance ssh

or if you are copying from a Local "secure" network just use ftp or hell
why not NFS.

http://www.gentoo-portage.com/net-misc/openssh/USE

Gentoo ass a high perf. flag for ssh.

http://www.psc.edu/networking/projects/hpn-ssh/

cheers,

Steve C

--
ION Network Solutions
Steve Clement
Unix System Administrator
209, rue des Romains
L-8041 Bertrange
Tel: +352 261 276-2
Fax: +352 261 276-9
mailto:steve < at > ion.lu
http://www.ion.lu

Steve Clement wrote:

Carsten Lorenz wrote:


rdiff-backup needs 132s to transfer a 140MB file (1.1MB/s).
While rdiff-backup works on the next file scp transfers the same file in 15s (9.3MB/s)!


What does that mean? scp = 9.3MB/s rdiff-backup = 1.1MB/s


If i understand, what you mean, yes
scp transfers this file 9 times faster than rdiff-backup

Since we want to backup one tera-byte of data this would last more than a week.


That be bad :)


ETA is Thursday.

Has anyone hints how to speedup rdiff-backup?


Well if you want to speed up rdiff-backup using ssh/scp as transport you
might need to look into:

high performance ssh

or if you are copying from a Local "secure" network just use ftp or hell
why not NFS.


We thought about using NFS. Since we use a separate "Network" consisting
of both servers and a cable, it can be configured to be safe.
We will try it, when we create the backup in the opposite direction.

Since scp and rdiff-backup/ssh both uses ssh2 transferring data, why is
rdiff-backup 9 times slower, while the CPU-usage is low?
I can't see where the bottleneck is.

Thanks, Carsten

On Fri, Oct 14, 2005 at 05:21:56PM +0200, Carsten Lorenz wrote:
We thought about using NFS. Since we use a separate "Network" consisting
of both servers and a cable, it can be configured to be safe.
We will try it, when we create the backup in the opposite direction.

Since scp and rdiff-backup/ssh both uses ssh2 transferring data, why is
rdiff-backup 9 times slower, while the CPU-usage is low?
I can't see where the bottleneck is.

Are you using FreeBSD? I have the same problems. See these messages:
http://lists.gnu.org/archive/html/rdiff-backup-users/2004-08/msg00029.html

I was never able to figure it out either. I just use NFS for large
file systems and SSH for small ones.

On Fri, 14 Oct 2005, Carsten Lorenz wrote:

Since we want to backup one tera-byte of data this would last more than a week.

i've never looked closely at why the first backup is so slow -- and i've
heard the report from lots of folks... i tend to use rsync for the initial
backup, and then use rdiff-backup with "--force" the next time (which will
generate one increment and create the rdiff-backup-data subdirectory).

the incremental runs are a lot faster ... although i've never tried
it on a TB of data. here are the average stats for daily backups on a
filesystem i maintain with a couple hundred mail/web accounts, with
the backup occuring over a 768k dsl line:

--------------[ Average of 31 stat files ]--------------
ElapsedTime 12966.16 (3 hours 36 minutes 6.16 seconds)
SourceFiles 1153830.35484
SourceFileSize 108174471204.0 (101 GB)
MirrorFiles 1153245.22581
MirrorFileSize 108115114552.0 (101 GB)
NewFiles 872.193548387
NewFileSize 173906366.29 (166 MB)
DeletedFiles 287.064516129
DeletedFileSize 136894182.742 (131 MB)
ChangedFiles 9703.90322581
ChangedSourceSize 9800370636.0 (9.13 GB)
ChangedMirrorSize 9778026167.13 (9.11 GB)
IncrementFiles 10880.3548387
IncrementFileSize 321359631.484 (306 MB)
TotalDestinationSizeChange 380716283.903 (363 MB)
Errors 0


Has anyone hints how to speedup rdiff-backup?

mounting the backup filesystem noatime,nodiratime probably helps a bit.

-dean

Carsten Lorenz wrote:
I started the first backup to an empty directory on a second server.
I'm wandering about the low speed of ca. 1.6MB/s

Both servers are DL380 with dual PIII 833MHz running debian, python2.3,
librsync 0.9.7-1 and rdiff-backup 1.0.0-0.cvs20050819 (thanks to Dean
Gaudet for the debian package).
They are directly connected by gigabit.

I started rdiff-backup this way:
rdiff-backup -v5 --terminal-verbosity=4 --ssh-no-compression /mnt/files/ 192.168.0.1::/mnt/raid/backup/fileserver2/

Everything looks fine:
CPU-usage on the source-server is ca.15% for ssh and ca. 6% for rdiff-backup
CPU-usage on the destination is ca. 18% for sshd and ca. 12% for rdiff-backup
No swap-space is used.

rdiff-backup needs 132s to transfer a 140MB file (1.1MB/s).
While rdiff-backup works on the next file scp transfers the same file in 15s (9.3MB/s)!

Since we want to backup one tera-byte of data this would last more than a week.

Has anyone hints how to speedup rdiff-backup?

Carsten

What filesystems are your source and destinatin partition? What are the mount
options? Could you give me the output of "mount" of both machines?

Also, when copying with SCP, do you copy to the same target partition as you
would with rdiff-backup?

On Fri, 14 Oct 2005, Carsten Lorenz wrote:

Has anyone hints how to speedup rdiff-backup?

i just had a random idea while reading one of the other threads...

maybe fsync is the culprit for the slow initial backups (compared to
rsync) ... it would also slow down increments, but assuming a typical rate
of file change that wouldn't be as noticable.

what would probably be telling is to get some "strace -r -T" output during
an initial backup and see just where it's spending its time. i tend to
take such output and run it through sort -t\< -k 2,2 ... which would sort
on the <n.nnnn> syscall timings... look for the worst culprits then work
backwards to figure out wtf rdiff-backup was trying to do when it hit
those worst cases... but if you send me some output i can do the data
mining.

-dean

"Carsten Lorenz" <clorenz < at > hamburg.fcb.com>
wrote the following on Fri, 14 Oct 2005 10:03:37 +0200
I started the first backup to an empty directory on a second server.
...
Everything looks fine:
CPU-usage on the source-server is ca.15% for ssh and ca. 6% for rdiff-backup
CPU-usage on the destination is ca. 18% for sshd and ca. 12% for rdiff-backup
No swap-space is used.

rdiff-backup needs 132s to transfer a 140MB file (1.1MB/s). While
rdiff-backup works on the next file scp transfers the same file in
15s (9.3MB/s)!

Hmm, when other people complained about slow rdiff-backups I thought
the problem was CPU usage but obviously this isn't the problem in your
case.

Try editing your Globals.py file and increasing blocksize and
conn_blocksize by a lot. For instance, try:

blocksize = 524288
conn_bufsize = 1572864

I'd be curious to see if this makes much of a difference.


--
Ben Escoto

dean gaudet <dean-list-rdiff-backup < at > arctic.org>
wrote the following on Fri, 14 Oct 2005 19:58:32 -0700 (PDT)
On Fri, 14 Oct 2005, Carsten Lorenz wrote:

Has anyone hints how to speedup rdiff-backup?

i just had a random idea while reading one of the other threads...

maybe fsync is the culprit for the slow initial backups (compared to
rsync) ... it would also slow down increments, but assuming a typical rate
of file change that wouldn't be as noticable.

what would probably be telling is to get some "strace -r -T" output during
an initial backup and see just where it's spending its time. i tend to
take such output and run it through sort -t\< -k 2,2 ... which would sort
on the <n.nnnn> syscall timings... look for the worst culprits then work
backwards to figure out wtf rdiff-backup was trying to do when it hit
those worst cases... but if you send me some output i can do the data
mining.

rdiff-backup doesn't fsync on an initial backup, except for one global
sync() at the end before it writes the current_mirror file. The
fsync()ing is only when doing incremental backups so it can regress if
necessary. But on an initial backup there is nothing to regress to so
fsyncing isn't necessary.

The strace stuff is interesting, but you may want to try increasing
those bufsizes first, since it would just take a few seconds and you
might see an improvement.


--
Ben Escoto

Ben Escoto wrote: "Carsten Lorenz" <clorenz < at > hamburg.fcb.com> ([email]clorenz < at > hamburg.fcb.com[/email])
wrote the following on Fri, 14 Oct 2005 10:03:37 +0200
I started the first backup to an empty directory on a second server.
...
Everything looks fine:
CPU-usage on the source-server is ca.15% for ssh and ca. 6% for rdiff-backup
CPU-usage on the destination is ca. 18% for sshd and ca. 12% for rdiff-backup
No swap-space is used.

rdiff-backup needs 132s to transfer a 140MB file (1.1MB/s). While
rdiff-backup works on the next file scp transfers the same file in
15s (9.3MB/s)!

Hmm, when other people complained about slow rdiff-backups I thought
the problem was CPU usage but obviously this isn't the problem in your
case.

Try editing your Globals.py file and increasing blocksize and
conn_blocksize by a lot. For instance, try:

blocksize = 524288
conn_bufsize = 1572864

I'd be curious to see if this makes much of a difference.
i will try that when i start the backup in the opposite direction. ETA for actual run is Thursday.

Carsten

Wiebe Cazemier wrote:

Carsten Lorenz wrote:

I started the first backup to an empty directory on a second server.
I'm wandering about the low speed of ca. 1.6MB/s

Both servers are DL380 with dual PIII 833MHz running debian, python2.3,
librsync 0.9.7-1 and rdiff-backup 1.0.0-0.cvs20050819 (thanks to Dean
Gaudet for the debian package).
They are directly connected by gigabit.

I started rdiff-backup this way:
rdiff-backup -v5 --terminal-verbosity=4 --ssh-no-compression
/mnt/files/ 192.168.0.1::/mnt/raid/backup/fileserver2/

Everything looks fine:
CPU-usage on the source-server is ca.15% for ssh and ca. 6% for
rdiff-backup
CPU-usage on the destination is ca. 18% for sshd and ca. 12% for
rdiff-backup
No swap-space is used.

rdiff-backup needs 132s to transfer a 140MB file (1.1MB/s).
While rdiff-backup works on the next file scp transfers the same file
in 15s (9.3MB/s)!

Since we want to backup one tera-byte of data this would last more
than a week.

Has anyone hints how to speedup rdiff-backup?

Carsten


What filesystems are your source and destinatin partition? What are
the mount options? Could you give me the output of "mount" of both
machines?

I don't think that the file system interferes here.

Both file systems are xfs on raid5 systems.
This is the actual output from mount:

source:
/dev/sde1 on /mnt/files type xfs (rw,noatime,nodiratime,logbufs=8)

destination (4 partitions through LVM2):
/dev/mapper/VG1-LOGVOL1 on /mnt/raid type xfs
(rw,noatime,nodiratime,logbufs=8)

We added the additional options for testing, but they don't change anything.
Both can of cause transfer more than 1.6MB/s. Reading or writing files. ;->
Copying local to an identical raid-system transfers ca. 100MB/s

Also, when copying with SCP, do you copy to the same target partition
as you would with rdiff-backup?

Yes and i used the same file and a different file with identical results.

Carsten

Ben Escoto wrote:

Hmm, when other people complained about slow rdiff-backups I thought
the problem was CPU usage but obviously this isn't the problem in your
case.

Try editing your Globals.py file and increasing blocksize and
conn_blocksize by a lot. For instance, try:

blocksize = 524288
conn_bufsize = 1572864

I'd be curious to see if this makes much of a difference.


It makes a difference.

To backup 1.13TB it lasts from 12th 2:00pm to 20th 10:30pm which makes
2.1KB/s

I did some short tests (the first 20 minutes of the same backup).
Your values work with 3.4KB/s
Larger buffers makes it slower.

Carsten

"Carsten Lorenz" <clorenz < at > hamburg.fcb.com>
wrote the following on Mon, 24 Oct 2005 13:41:08 +0200

To backup 1.13TB it lasts from 12th 2:00pm to 20th 10:30pm which makes
2.1KB/s

More than 8 days?? Still though, isn't 1.13TB over 8 days about
1.7MB/s?

I did some short tests (the first 20 minutes of the same backup).
Your values work with 3.4KB/s
Larger buffers makes it slower.

Oops, I just increased the buffer sizes in 1.1.0, since this issue
reminded me that they've been the same size for years. I'm not sure
why larger buffers would slow things down, and would love to know what
the bottleneck on your system (and Andy Wettstein's) is.


--
Ben Escoto