Backup Central

On Mon, 6 Feb 2006, Vadim Kouzmine wrote:

Let me summarize the information I've gathered during my tests, on two
platforms (Gentoo, Trustix 2.2) and different versions of python (2.2.3,
2.3.5, 2.4.2):

- on 3GHz P4 Xeon I get ~2MB/s steady transfer through ssh, on P4 3GHz
it's ~1.9MB/s;
- I made tests ssh-ing to LOCALHOST, so no network is involved here.

actually we can get rid of ssh entirely as well and just use pipes
between the rdiff-backup and "rdiff-backup --server"... to reduce the
variables even further.

when i compare:

rdiff-backup src dst1

with:

rdiff-backup --remote-schema '%s' src 'rdiff-backup --server'::dst2

i'm seeing an avg of 0.9s real-time for the first and 1.4s real-time
for the second using an extracted copy of rdiff-backup-1.0.4.tar.gz as
the src. (the results are quite repeatable so i didn't bother with a
larger src.)

my suspicion would be something serializing in the protocol...

-dean

On Mon, 6 Feb 2006, Vadim Kouzmine wrote:

Let me summarize the information I've gathered during my tests, on two
platforms (Gentoo, Trustix 2.2) and different versions of python (2.2.3,
2.3.5, 2.4.2):

- on 3GHz P4 Xeon I get ~2MB/s steady transfer through ssh, on P4 3GHz
it's ~1.9MB/s;
- I made tests ssh-ing to LOCALHOST, so no network is involved here.

Suddenly it's changed for me. Now I get about 4MB/s average on the same
hardware, same data, through the network. This is what's changed:

1) rdiff-backup 1.1.5 vs. 1.0.4
2) python 2.4.2 vs 2.2.3
3) called as:
"--remote-schema 'ssh -c blowfish %s /usr/local/Python/bin/rdiff-backup
--server --restrict-read-only /'"
vs
"--ssh-no-compression --restrict-read-only /"
4) openssh 4.3 vs 4.2
5) destination is external USB drive (actually slow) vs local RAID array

I don't have time to identify what exactly made it faster, but it's now
2 (!) times faster and it's great!


actually we can get rid of ssh entirely as well and just use pipes
between the rdiff-backup and "rdiff-backup --server"... to reduce the
variables even further.

when i compare:

rdiff-backup src dst1

with:

rdiff-backup --remote-schema '%s' src 'rdiff-backup --server'::dst2

i'm seeing an avg of 0.9s real-time for the first and 1.4s real-time
for the second using an extracted copy of rdiff-backup-1.0.4.tar.gz as
the src. (the results are quite repeatable so i didn't bother with a
larger src.)

my suspicion would be something serializing in the protocol...

I'll check it with netcat tunnels, and probably with netcat through ssh
forwarded port, and post results. Any other idea to make a tunnel?

Vadim

On Mon, 2006-02-20 at 21:11 -0800, dean gaudet wrote:
actually we can get rid of ssh entirely as well and just use pipes
between the rdiff-backup and "rdiff-backup --server"... to reduce the
variables even further.

when i compare:

rdiff-backup src dst1

with:

rdiff-backup --remote-schema '%s' src 'rdiff-backup --server'::dst2

i'm seeing an avg of 0.9s real-time for the first and 1.4s real-time
for the second using an extracted copy of rdiff-backup-1.0.4.tar.gz as
the src. (the results are quite repeatable so i didn't bother with a
larger src.)

my suspicion would be something serializing in the protocol...

-dean

Some test results below. Backup src is single 512MB file with urandom
stuff. Servers are 3GHz P4 Xeon, gigabit lan. rdiff-backup 1.1.5 /
python 2.4.2. Time in SECONDS.

1) rdiff-backup src dst
16.9s

2) rdiff-backup --remote-schema '%s' src 'rdiff-backup --server'::dst
21.3s

3) rdiff-backup -remote-schema 'ssh %s rdiff-backup' server2::src dst
80.0s (76.7s with blowfish and 75.9s with arcfour)

4) rdiff-backup through netcat (no ssh/no encryption)
76.4s (just like with ssh?!)

5) rdiff-backup -remote-schema 'ssh -C %s rdiff-backup' server2::src dst
129.1s (compression makes it so much slower!)

6) scp server2:src/bigfile dst/
13.6s

7) rsync server2:src/ dst/
14.1s

So, yes, something is wrong with serializing. And looks like it's
seriously affected by latency, compare 2) with 3),4).

------
Just in case this is how I run it with netcat. Sure must be an easier
way :)

On src (remote server2):

nc -l -p 5555 <afifo | rdiff-backup --server >afifo

On dst (local receiver):

rdiff-backup --remote-schema 'nc %s 5555' server2:src dst

--
Vadim <kouzminv < at > gmail.com>