rdiff-backup painfully slow using NFS mount

Hello,

I was wondering if the use of NFS mounts is contraindicated for rdiff-backup? The documentation on the site alludes to some errors, but then release notes seem to indicate that bug fixes were put into place for those errors. I am running rdiff-backup version 1.2.8 installed via FreeBSD 7.2 ports. The system is an older P4 2.8 with 2 GiB of RAM and 100 Mbps network speeds.

It took rdiff-backup 1 hour 49 minutes and 7.88 seconds according to the log. The log stated it backed up 6634675372 bytes in 6547.88 seconds.

6634675372 bytes = 53077402976 bits

53077402976 / 6547.88 seconds = 8106044 bits per second or 8.11 Mbps

The NAS device I'm backing up to (ReadyNAS NV) does not allow inbound SSH (well, they blocked it and as of yet I haven't gotten around to breaking into it) so I need to do my backups via NFS over TCP and is direct connected by crossover cable.

Also, as an interesting note when I'm copying files from the server to the NAS directly, my MRTG graphs show outbound traffic approaching 80 Mbps, but when rdiff-backup is running, it shows traffic inbound AND outbound (see [url=http://archon.silvertree.org/rdiff.png)]http://archon.silvertree.org/rdiff.png)[/url].

This has me somewhat flummoxed as I like the way rdiff-backup seems to work, but I definitely need more speed. I know that subsequent backups should be quicker, but this is a pretty serious performance hit.

Thanks for any pointers,

Scott

The issue is that since you're using an NFS mount instead of SSH, the network transfer benefits of rdiff-backup are lost. Really, if you want to get reasonable performance, you need to install rdiff-backup on the NAS and enable SSH access.

JoshN

Scott Schappell wrote: [quote]Hello,

I was wondering if the use of NFS mounts is contraindicated for rdiff-backup? The documentation on the site alludes to some errors, but then release notes seem to indicate that bug fixes were put into place for those errors. I am running rdiff-backup version 1.2.8 installed via FreeBSD 7.2 ports. The system is an older P4 2.8 with 2 GiB of RAM and 100 Mbps network speeds.

It took rdiff-backup 1 hour 49 minutes and 7.88 seconds according to the log. The log stated it backed up 6634675372 bytes in 6547.88 seconds.

6634675372 bytes = 53077402976 bits

53077402976 / 6547.88 seconds = 8106044 bits per second or 8.11 Mbps

The NAS device I'm backing up to (ReadyNAS NV) does not allow inbound SSH (well, they blocked it and as of yet I haven't gotten around to breaking into it) so I need to do my backups via NFS over TCP and is direct connected by crossover cable.

Also, as an interesting note when I'm copying files from the server to the NAS directly, my MRTG graphs show outbound traffic approaching 80 Mbps, but when rdiff-backup is running, it shows traffic inbound AND outbound (see [url=http://archon.silvertree.org/rdiff.png%29]http://archon.silvertree.org/rdiff.png)[/url].

This has me somewhat flummoxed as I like the way rdiff-backup seems to work, but I definitely need more speed. I know that subsequent backups should be quicker, but this is a pretty serious performance hit.

Thanks for any pointers,

Scott [quote]

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org ([email]rdiff-backup-users < at > nongnu.org[/email])
[url=http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users]http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users[/url]
Wiki URL: [url=http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki]http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki[/url][/quote] [/quote]

So, then, to get optimal performance of rdiff-backup, it either needs to be a disk to disk copy on the local machine or transport via SSH?

I like the rdiff-backup methodology, and I trust it more than just doing rsync and both rsync and rdiff-backup are vastly superior to doing the dumps to a file I was doing.

I suppose I could rdiff-backup to a local disk, and then rsync that to the NFS mount, or is that ill advised?

Thanks for the reply, Josh

Scott
On Aug 12, 2009, at 09:57:47, Josh Nisly wrote:
[quote] The issue is that since you're using an NFS mount instead of SSH, the network transfer benefits of rdiff-backup are lost. Really, if you want to get reasonable performance, you need to install rdiff-backup on the NAS and enable SSH access.

JoshN

Scott Schappell wrote: [quote]Hello,

I was wondering if the use of NFS mounts is contraindicated for rdiff-backup? The documentation on the site alludes to some errors, but then release notes seem to indicate that bug fixes were put into place for those errors. I am running rdiff-backup version 1.2.8 installed via FreeBSD 7.2 ports. The system is an older P4 2.8 with 2 GiB of RAM and 100 Mbps network speeds.

It took rdiff-backup 1 hour 49 minutes and 7.88 seconds according to the log. The log stated it backed up 6634675372 bytes in 6547.88 seconds.

6634675372 bytes = 53077402976 bits

53077402976 / 6547.88 seconds = 8106044 bits per second or 8.11 Mbps

The NAS device I'm backing up to (ReadyNAS NV) does not allow inbound SSH (well, they blocked it and as of yet I haven't gotten around to breaking into it) so I need to do my backups via NFS over TCP and is direct connected by crossover cable.

Also, as an interesting note when I'm copying files from the server to the NAS directly, my MRTG graphs show outbound traffic approaching 80 Mbps, but when rdiff-backup is running, it shows traffic inbound AND outbound (see [url=http://archon.silvertree.org/rdiff.png%29]http://archon.silvertree.org/rdiff.png)[/url].

This has me somewhat flummoxed as I like the way rdiff-backup seems to work, but I definitely need more speed. I know that subsequent backups should be quicker, but this is a pretty serious performance hit.

Thanks for any pointers,

Scott [quote]
_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org ([email]rdiff-backup-users < at > nongnu.org[/email])
[url=http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users]http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users[/url]
Wiki URL: [url=http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki]http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki[/url][/quote] [/quote]

[/quote]

You are correct in your analysis. Using rdiff-backup locally + rsync to the NFS mount would certainly work.

JoshN

Scott Schappell wrote: [quote]So, then, to get optimal performance of rdiff-backup, it either needs to be a disk to disk copy on the local machine or transport via SSH?

I like the rdiff-backup methodology, and I trust it more than just doing rsync and both rsync and rdiff-backup are vastly superior to doing the dumps to a file I was doing.

I suppose I could rdiff-backup to a local disk, and then rsync that to the NFS mount, or is that ill advised?

Thanks for the reply, Josh

Scott
On Aug 12, 2009, at 09:57:47, Josh Nisly wrote:
[quote] The issue is that since you're using an NFS mount instead of SSH, the network transfer benefits of rdiff-backup are lost. Really, if you want to get reasonable performance, you need to install rdiff-backup on the NAS and enable SSH access.

JoshN

Scott Schappell wrote: [quote]Hello,

I was wondering if the use of NFS mounts is contraindicated for rdiff-backup? The documentation on the site alludes to some errors, but then release notes seem to indicate that bug fixes were put into place for those errors. I am running rdiff-backup version 1.2.8 installed via FreeBSD 7.2 ports. The system is an older P4 2.8 with 2 GiB of RAM and 100 Mbps network speeds.

It took rdiff-backup 1 hour 49 minutes and 7.88 seconds according to the log. The log stated it backed up 6634675372 bytes in 6547.88 seconds.

6634675372 bytes = 53077402976 bits

53077402976 / 6547.88 seconds = 8106044 bits per second or 8.11 Mbps

The NAS device I'm backing up to (ReadyNAS NV) does not allow inbound SSH (well, they blocked it and as of yet I haven't gotten around to breaking into it) so I need to do my backups via NFS over TCP and is direct connected by crossover cable.

Also, as an interesting note when I'm copying files from the server to the NAS directly, my MRTG graphs show outbound traffic approaching 80 Mbps, but when rdiff-backup is running, it shows traffic inbound AND outbound (see [url=http://archon.silvertree.org/rdiff.png%29]http://archon.silvertree.org/rdiff.png)[/url].

This has me somewhat flummoxed as I like the way rdiff-backup seems to work, but I definitely need more speed. I know that subsequent backups should be quicker, but this is a pretty serious performance hit.

Thanks for any pointers,

Scott [quote]
_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki[/quote] [/quote]

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org ([email]rdiff-backup-users < at > nongnu.org[/email])
[url=http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users]http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users[/url]
Wiki URL: [url=http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki]http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki[/url][/quote] [/quote]

Josh Nisly wrote:

[quote]You are correct in your analysis. Using rdiff-backup locally + rsync
to the NFS mount would certainly work.
[/quote]
Wouldn't that still leave the same fundamental problem - that using
NFS rather than rsync or rdiff-backup to access the backup storage
means that the system has to read the files from storage across the
network in order to work out what needs to be copied ?

Ie, taking the earlier example I gave (in thread "Measuring transfer
size") of adding 1 byte to the front of a 1G file :

If using an rsync server or rsync via ssh remote, rsync would
determine that the file contents had moved by 1 byte and would
transfer across the network only an initial block of data plus all
the checksums required. Ie only a few k traffic

If mounting a filesystem via <insert file sharing method> would mean
rsync reading the entire 1G file across the network, calculating that
it needs to insert 1 byte, doing that locally and writing out the
entire 1G file again. About 2G of network traffic.

With rsync (don't know about rdiff-backup) IIRC there are
combinations of options that would allow it to see the new
modification timestamp and file size, then just copy the entire file
without comparing it with the original. It would still be copying the
whole gig of file, but saves reading the gig of file from the backup
to do the compare.

--
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

On Wed, Aug 12, 2009 at 14:16, Simon Hobson<linux < at > thehobsons.co.uk> wrote:
[quote]Josh Nisly wrote:

[quote]You are correct in your analysis. Using rdiff-backup locally + rsync to
the NFS mount would certainly work.
[/quote]
Wouldn't that still leave the same fundamental problem
[/quote]
Yes. It would.

Magic isn't real. To truly know that a source and destination are
identical, the entire source must be read. As must the entire
destination. Period. If the two are on seperate computers joined by
a network, you can do this two ways:

1) Transfer the contents of one of them entirely, over the network.

2) Run a process on each to reduce blocks of each file to checksums,
and transfer just those checksums.

ISCSI and NFS are not capable of letting the remote machine run
processes on the server. SSH is. Rsync will run the comparison on
both machines.

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

On Aug 12, 2009, at 12:57:51, Billy Crook wrote:

[quote]ISCSI and NFS are not capable of letting the remote machine run
processes on the server. SSH is. Rsync will run the comparison on
both machines.

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
[/quote]
This is exactly what I needed to know. Guess it's time to throw a
freenas box together or get a new appliance that will let me run SSH.

Thanks :)

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Simon Hobson wrote:
[quote]Josh Nisly wrote:

[quote]You are correct in your analysis. Using rdiff-backup locally + rsync
to the NFS mount would certainly work.
[/quote]
Wouldn't that still leave the same fundamental problem - that using
NFS rather than rsync or rdiff-backup to access the backup storage
means that the system has to read the files from storage across the
network in order to work out what needs to be copied ?
[/quote]You're right, of course. The main difference is that the rdiff-backup
"snapshotting" would be fast (and therefore, more likely to get a
consistent snapshot), and would let the rsync process churn away in the
background.

JoshN

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

But the whole discussion still doesn't explain why it's so much slower during
the *first* run (that's what Scott's mail was about, if I'm not mistaken). In
that case, rsync/rdiff-backup should be (roughly) as fast as NFS, if there is
no processing bottleneck on the machine that's running both instances of rsync.
Rsync would figure out that there are no files on the destination, yet. It
would then transfer all data, without further checking the destination, AFAIK.

Patrick.

On 2009-08-13 04:11, Scott Schappell wrote:
[quote]This is exactly what I needed to know. Guess it's time to throw a
freenas box together or get a new appliance that will let me run SSH.

Thanks :)
[/quote]
--
Key ID: 0x86E346D4 http://patrick-nagel.net/key.asc
Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

---------- Forwarded message ----------
From: [b]James[/b] <packetsniffer < at > gmail.com ([email]packetsniffer < at > gmail.com[/email])>
Date: Thu, Aug 13, 2009 at 7:59 AM
Subject: Re: [rdiff-backup-users] rdiff-backup painfully slow using NFS mount
To: Patrick Nagel <mail < at > patrick-nagel.net ([email]mail < at > patrick-nagel.net[/email])>

I had bad performance with rsnapshot and rdiff-backup over an NFS mount also, to a raided Drobo Share device, it sucked so badly in the end the backup dragged on all night and into the day (it was a large backup). In the end I connected the device over USB2.0 and stopped using NFS, I think this was greatly to do with the Drobo Shares network performance, stating it was gigabit when it never seemed to perform at gigabit speeds (i think it was more of a market move, and was just a gigabit port). Also the device seemed to struggle with all the hard links and many small files being transferred.

Now that the device is connected over USB2.0 all is well.

James

On Thu, Aug 13, 2009 at 4:09 AM, Patrick Nagel <mail < at > patrick-nagel.net ([email]mail < at > patrick-nagel.net[/email])> wrote:
[quote]But the whole discussion still doesn&#39;t explain why it&#39;s so much slower during
the *first* run (that&#39;s what Scott&#39;s mail was about, if I&#39;m not mistaken). In
that case, rsync/rdiff-backup should be (roughly) as fast as NFS, if there is
no processing bottleneck on the machine that&#39;s running both instances of rsync.
Rsync would figure out that there are no files on the destination, yet. It
would then transfer all data, without further checking the destination, AFAIK.

Patrick.

On 2009-08-13 04:11, Scott Schappell wrote:
[quote]This is exactly what I needed to know. Guess it&#39;s time to throw a
freenas box together or get a new appliance that will let me run SSH.

Thanks :)
[/quote]

--
Key ID: 0x86E346D4            [url=http://patrick-nagel.net/key.asc]http://patrick-nagel.net/key.asc[/url]
Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org ([email]rdiff-backup-users < at > nongnu.org[/email])
[url=http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users]http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users[/url]
Wiki URL: [url=http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki]http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki[/url]

[/quote]

rdiff-backup gets most of its advantage by running running a server process on
the destination machine that keeps a lot of the disk access local. When you
run rdiff-backup on a networked drive you lose this advantage because the
client and server processes bot run on the local machine and all disk access
is across the wire.

You might find Duplicity (http://duplicity.nongnu.org) is better suited to your
arrangement. Duplicity also uses the rdiff algorithm but it keeps the metadata
locally and creates local backup archives which are uploaded when completed.

Dean

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

This is exactly what I was looking for. I'm going to join the
duplicity list, as I do have some rookie questions.

Thanks, Dean!
On Aug 12, 2009, at 22:42:34, Dean Cording wrote:

[quote]rdiff-backup gets most of its advantage by running running a server
process on
the destination machine that keeps a lot of the disk access local.
When you
run rdiff-backup on a networked drive you lose this advantage
because the
client and server processes bot run on the local machine and all
disk access
is across the wire.

You might find Duplicity (http://duplicity.nongnu.org) is better
suited to your
arrangement. Duplicity also uses the rdiff algorithm but it keeps
the metadata
locally and creates local backup archives which are uploaded when
completed.

Dean

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
[/quote]

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki