Backup Central

Hallo,

I like using "rsnapshot", and i like using it with nfs-shares in my LAN.

Usual way: one computer in the LAN is target for nearly all backups, its
nfs-share is mounted/symlinked to "/srv/backup" (or a similar
mountpoint).

A typical "hourly" backup for a machine with about 150 GByte data shows

[12/May/2011:05:35:02] /usr/bin/rsnapshot -c /etc/rsnap-deneb.conf
hourly: started
[12/May/2011:05:35:02] echo 13750 > /var/run/rsnap-deneb.pid
[12/May/2011:05:35:02] mv /srv/backdeneb/hourly.1/ /srv/backdeneb/_delete.13750/
[12/May/2011:05:35:02] /bin/cp -al /srv/backdeneb/hourly.0 /srv/backdeneb/hourly.1
[12/May/2011:06:04:22] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file --exclude=srv/backdeneb /. /srv/backdeneb/hourly.0/Wurzel/
[12/May/2011:06:05:33] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file /var /srv/backdeneb/hourly.0/./
[12/May/2011:06:05:42] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file /home /srv/backdeneb/hourly.0/./
[12/May/2011:06:11:28] touch /srv/backdeneb/hourly.0/
[12/May/2011:06:11:28] rm -f /var/run/rsnap-deneb.pid
[12/May/2011:06:11:28] /bin/rm -rf /srv/backdeneb/_delete.13750
[12/May/2011:06:28:06] /usr/bin/rsnapshot -c /etc/rsnap-deneb.conf hourly: completed successfully


---------------------------------------------------------

The job "cp -al" needs about 30 minutes. If I run this job directly on
the target machine (not over nfs) it only needs 2 minutes.

Is it possible to accelerate this job, p.e. by using a command which
tells the target machine to do this job?

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, May 12, 2011 at 12:30:00PM +0200, Helmut Hullen wrote:

The job "cp -al" needs about 30 minutes. If I run this job directly on
the target machine (not over nfs) it only needs 2 minutes.

Is it possible to accelerate this job, p.e. by using a command which
tells the target machine to do this job?

cmd_cp some_script

But this is a *nasty* hack.

--
David Cantrell | Minister for Arbitrary Justice

I apologize if I offended you personally,
I intended to do it professionally.
-- Steve Champeon, on the nanog list

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, David,

Du meintest am 12.05.11:

The job "cp -al" needs about 30 minutes. If I run this job directly
on the target machine (not over nfs) it only needs 2 minutes.

Is it possible to accelerate this job, p.e. by using a command which
tells the target machine to do this job?

cmd_cp some_script

But this is a *nasty* hack.

Maybe it's nasty. But I'd like it ...
Any idea (nasty or good looking)?

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, May 12, 2011 at 02:29:00PM +0200, Helmut Hullen wrote:
The job "cp -al" needs about 30 minutes. If I run this job directly
on the target machine (not over nfs) it only needs 2 minutes.

Is it possible to accelerate this job, p.e. by using a command which
tells the target machine to do this job?
cmd_cp some_script
But this is a *nasty* hack.
Maybe it's nasty. But I'd like it ...
Any idea (nasty or good looking)?

#!/bin/sh
ssh root < at > whateverhost cp -al /mnt/backups/hourly.0 /mnt/backups/hourly.1

(or whatever you'd normally use for a remote shell)

You could either hard-code the source and destination for the copy, or
if you want it to be a wee bit more general-purpose, parse the last two
arguments that the script receives and re-write them before passing
*all* arguments through to the remote cp, including -al.

Don't forget to handle paths with spaces in them properly, which is a
bit more tricky than normal in this case.

--
David Cantrell | London Perl Mongers Deputy Chief Heretic

Aluminum makes a nice hat.
All paranoids will tell you that.
But what most do not know
Is reflections will show
On the CIA's evil landsat.

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, David,

Du meintest am 12.05.11:

Is it possible to accelerate this job, p.e. by using a command
which tells the target machine to do this job?

cmd_cp some_script
But this is a *nasty* hack.
Maybe it's nasty. But I'd like it ...
Any idea (nasty or good looking)?

#!/bin/sh
ssh root < at > whateverhost cp -al /mnt/backups/hourly.0
/mnt/backups/hourly.1

(or whatever you'd normally use for a remote shell)

Thank you - I didn't remember the syntax, and "trial and error" is
nothing I'd use for backups.

You could either hard-code the source and destination for the copy,
or if you want it to be a wee bit more general-purpose, parse the
last two arguments that the script receives and re-write them before
passing *all* arguments through to the remote cp, including -al.

Am I right that "cp -al" is only needed for "hourly" (for the lowest
interval)?
Maybe "rm" could be treated in a similar way, for the "_delete"
directories. But that's a minor problem.

Don't forget to handle paths with spaces in them properly, which is a
bit more tricky than normal in this case.

And thanks again for these hints! I'll see which of my machines has to
be the guinea pig ...

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, May 12, 2011 at 03:19:00PM +0200, Helmut Hullen wrote:

Am I right that "cp -al" is only needed for "hourly" (for the lowest
interval)?

Yes.

--
David Cantrell | Enforcer, South London Linguistic Massive

There are many different types of sausages. The best are
from the north of England. The wurst are from Germany.
-- seen in alt.2eggs...

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, David,

Du meintest am 12.05.11:

The job "cp -al" needs about 30 minutes. If I run this job
directly on the target machine (not over nfs) it only needs 2
minutes.

Is it possible to accelerate this job, p.e. by using a command
which tells the target machine to do this job?

cmd_cp some_script

#!/bin/sh
ssh root < at > whateverhost cp -al /mnt/backups/hourly.0
/mnt/backups/hourly.1

I've first played with "cmd_rm", and my script works (but is still a bit
ugly).

In the "rsnapshot.conf" I define

cmd_rm /root/bin/fernrm

with the script "/root/bin/fernrm":


#! /bin/bash
# fuer rsnapshot mit NFS-Freigaben
# David Cantrell, Mailingliste "rsnapshot" 12. Mai 2011

Opt=$1
Pfad=$2
Datei=${Pfad##*/}
test "$Datei" || exit 1

Ziel=/Pfad/on/NFS-share

echo ssh root < at > 192.168.0.3 rm -rf $Ziel/$Datei
ssh root < at > 192.168.0.3 rm -rf $Ziel/$Datei
#

---------------------------------------

Times (on another machine, about 7 GByte over fast ethernet)

old way (deleting over nfs) about 21 minutes
this way (with ssh) about 90 seconds

The "echo" line puts some information into the rsnapshot log file (there
may be a better way).
The real directory name on the target machine (here: $Ziel) may be hard
coded; I haven't yet seen a reliable way to determine it from the source
machine.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, David,

Du meintest am 12.05.11:

The job "cp -al" needs about 30 minutes. If I run this job directly
on the target machine (not over nfs) it only needs 2 minutes.

Is it possible to accelerate this job, p.e. by using a command which
tells the target machine to do this job?

cmd_cp some_script


Just an additional information:

I've played with "link_dest".

Setting "link_dest 0" separates "cp -al" from rsync, and then it seems
to save time setting "cmd_cp" to "some_script" which lets the target cpu
do the job.

Setting "link_dest 1" needs as long as "cp -al" over nfs.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, David,

Du meintest am 12.05.11:

The job "cp -al" needs about 30 minutes. If I run this job
directly on the target machine (not over nfs) it only needs 2
minutes.

Is it possible to accelerate this job, p.e. by using a command
which tells the target machine to do this job?

cmd_cp some_script

And now it works, reliable and quick!

In "/etc/rsnapshot.conf"

cmd_cp /root/bin/ferncp

with the script "/root/bin/ferncp"

#! /bin/bash
# f?r rsnapshot mit NFS-Freigaben
# David Cantrell, Mailingliste "rsnapshot" 12. Mai 2011

# link_dest muss auf 0 (aus) gesetzt sein

Opt=$1
Pfad=$2
Datei=${Pfad##*/}
test "$Datei" || exit 1
Ziel=/path/on/the/nfs/target

echo ssh root < at > 192.168.0.3 cp -al $Ziel/hourly.0 $Ziel/hourly.1
ssh root < at > 192.168.0.3 cp -al $Ziel/hourly.0 $Ziel/hourly.1
#

(and a similar routine for "cmd_rm", as described some minutes ago)

---------------------------------------

Log file for about 7 GByte over fast ethernet:

old way (cp and rm over nfs)

[01/May/2011:17:20:02] /usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly:
started
[01/May/2011:17:20:02] echo 1218 > /var/run/rsnapshot.pid
[01/May/2011:17:20:03] mv /srv/backup/hourly.1/ /srv/backup/_delete.1218/
[01/May/2011:17:20:03] mv /srv/backup/hourly.0/ /srv/backup/hourly.1/
[01/May/2011:17:20:03] mkdir -m 0755 -p /srv/backup/hourly.0/
[01/May/2011:17:20:04] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file --exclude=srv/backup --link-dest=/srv/backup/hourly.1/Wurzel/ /. /srv/backup/hourly.0/Wurzel/
[01/May/2011:17:38:51] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file --link-dest=/srv/backup/hourly.1/./ /var /srv/backup/hourly.0/./
[01/May/2011:17:39:11] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file --link-dest=/srv/backup/hourly.1/./ /home /srv/backup/hourly.0/./
[01/May/2011:17:41:59] touch /srv/backup/hourly.0/
[01/May/2011:17:41:59] rm -f /var/run/rsnapshot.pid
[01/May/2011:17:41:59] /bin/rm -rf /srv/backup/_delete.1218
[01/May/2011:18:03:20] /usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly: completed successfully

new way (cp and rm via ssh on the target machine)

[12/May/2011:19:36:44] /usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly:
started
[12/May/2011:19:36:44] echo 23365 > /var/run/rsnapshot.pid
[12/May/2011:19:36:44] mv /srv/backup/hourly.1/ /srv/backup/_delete.23365/
[12/May/2011:19:36:44] /root/bin/ferncp -al /srv/backup/hourly.0 /srv/backup/hourly.1
[12/May/2011:19:38:43] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file --exclude=srv/backup /. /srv/backup/hourly.0/Wurzel/
[12/May/2011:19:42:20] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file /var /srv/backup/hourly.0/./
[12/May/2011:19:42:28] /usr/bin/rsync -ax --delete --numeric-ids --relative --delete-excluded --include=home/tmp/* --exclude-from=/etc/rsnapshot.d/exclude-file /home /srv/backup/hourly.0/./
[12/May/2011:19:43:13] touch /srv/backup/hourly.0/
[12/May/2011:19:43:13] rm -f /var/run/rsnapshot.pid
[12/May/2011:19:43:13] /root/bin/fernrm -rf /srv/backup/_delete.23365
[12/May/2011:19:44:42] /usr/bin/rsnapshot -c /etc/rsnapshot.conf hourly: completed successfully

Result: old way about 43 minutes, news way about 8 minutes.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, May 12, 2011 at 5:29 AM, Helmut Hullen <Hullen < at > t-online.de> wrote:
Du meintest am 12.05.11:
The job "cp -al" needs about 30 minutes. If I run this job directly
on the target machine (not over nfs) it only needs 2 minutes.

Is it possible to accelerate this job, p.e. by using a command which
tells the target machine to do this job?

cmd_cp   some_script

But this is a *nasty* hack.

Maybe it's nasty. But I'd like it ...
Any idea (nasty or good looking)?

I find myself somewhat perplexed as to why you're willing to remote
the cp and rm commands to that box, but _not_ willing to remote the
entire rsnapshot to that box? You're already trusting that box with
the valuable goodies involved in backing things up. You can also take
steps to allow that box to rsync stuff over without having full root
access to the boxes being backed up. So why not just let that box
handle the rsnapshot directly?

-scott

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Scott,

Du meintest am 12.05.11:

I find myself somewhat perplexed as to why you're willing to remote
the cp and rm commands to that box, but _not_ willing to remote the
entire rsnapshot to that box?

Quite simple:

df

shows (on many of my machines) more than 5 mounted partitions, and most
of them are no shares for all other machines in the LAN.

Some machines only have 1 disk, some others have 2 to 5 disks.

I use the "one filesystem" option in the rsnapshot.conf, and I define
which partitions have to be backed up.

And with that configuration I can simply define several targets for the
backup:

The ugliest: a subdirectory on the only disk.
The quickest: a dedicated disk in the same machine
The neurotic: another machine in the LAN or the WAN

(by the way: most of these systems are school servers in other cities in
germany - I'm allowed to look at them).

Some of these systems use two of the three possible ways - then it's the
easy way only to change the configuration file(s), not to install a part
of the backup system on another machine too.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

I'm still not sure I understand! I don't think the "one filesystem"
option is relevant (rsync pulling to the remote machine will apply it
just fine). One option you could do would be to define a conf file on
each machine, and then wrap the rsnapshot cron job on the central
server with a script to pull those files down and run rsnapshot on
each. Alternately, the individual machine could have a cron which
pushes the conf data to the central machine and remotely runs
rsnapshot over there (elsewhere in the thread you've already
established that you're willing to let the machine run such scripts
remotely on the backup server).

Regardless - where I'm going, here, is that doing rsnapshot over NFS
works, but isn't terribly efficient. At the point where you're
starting to work around this inefficiency by hacking in scripts to
poke through the NFS abstraction, you might be better off to just poke
_entirely_ through the NFS abstraction and use rsync. It will be
faster and less error prone.

-scott


On Thu, May 12, 2011 at 1:59 PM, Helmut Hullen <Hullen < at > t-online.de> wrote:
Hallo, Scott,

Du meintest am 12.05.11:

I find myself somewhat perplexed as to why you're willing to remote
the cp and rm commands to that box, but _not_ willing to remote the
entire rsnapshot to that box?

Quite simple:

       df

shows (on many of my machines) more than 5 mounted partitions, and most
of them are no shares for all other machines in the LAN.

Some machines only have 1 disk, some others have 2 to 5 disks.

I use the "one filesystem" option in the rsnapshot.conf, and I define
which partitions have to be backed up.

And with that configuration I can simply define several targets for the
backup:

The ugliest: a subdirectory on the only disk.
The quickest: a dedicated disk in the same machine
The neurotic: another machine in the LAN or the WAN

(by the way: most of these systems are school servers in other cities in
germany - I'm allowed to look at them).

Some of these systems use two of the three possible ways - then it's the
easy way only to change the configuration file(s), not to install a part
of the backup system on another machine too.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss


------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

I have been using the approach suggested by Scott for some time - it is much faster than
using NFS directly.

I run a cron job on the PC to be backed up, like:
ssh -i $HOME/.ssh/id_rsa.qnap rsnapshot-script.sh <parameters>
where id_rsa.qnap contains an RSA private key solely used for backups, and the script on
the server (rsnapshot-script.sh) uses rsnapshot to do the backup using a config file I
previously put on the server. You could also copy the config file to the server from the
machine to be backed up before calling the script.

I then use cron on the server to rotate the snapshots daily, weekly etc, as these do not
depend on any data on the client. This also allows rotations to occur if the client is not
running (it is a laptop in my case).

Can anyone explain why NFS is so incredibly slow for rsnapshot? (45 minutes compared to 4
minutes in my case).

Regards,
Peter Barker

On Fri, 13 May 2011 07:35:08 am Scott Hess wrote:
I'm still not sure I understand! I don't think the "one filesystem"
option is relevant (rsync pulling to the remote machine will apply it
just fine). One option you could do would be to define a conf file on
each machine, and then wrap the rsnapshot cron job on the central
server with a script to pull those files down and run rsnapshot on
each. Alternately, the individual machine could have a cron which
pushes the conf data to the central machine and remotely runs
rsnapshot over there (elsewhere in the thread you've already
established that you're willing to let the machine run such scripts
remotely on the backup server).

Regardless - where I'm going, here, is that doing rsnapshot over NFS
works, but isn't terribly efficient. At the point where you're
starting to work around this inefficiency by hacking in scripts to
poke through the NFS abstraction, you might be better off to just poke
_entirely_ through the NFS abstraction and use rsync. It will be
faster and less error prone.

-scott

On Thu, May 12, 2011 at 1:59 PM, Helmut Hullen <Hullen < at > t-online.de> wrote:
Hallo, Scott,

Du meintest am 12.05.11:
I find myself somewhat perplexed as to why you're willing to remote
the cp and rm commands to that box, but _not_ willing to remote the
entire rsnapshot to that box?

Quite simple:

df

shows (on many of my machines) more than 5 mounted partitions, and most
of them are no shares for all other machines in the LAN.

Some machines only have 1 disk, some others have 2 to 5 disks.

I use the "one filesystem" option in the rsnapshot.conf, and I define
which partitions have to be backed up.

And with that configuration I can simply define several targets for the
backup:

The ugliest: a subdirectory on the only disk.
The quickest: a dedicated disk in the same machine
The neurotic: another machine in the LAN or the WAN

(by the way: most of these systems are school servers in other cities in
germany - I'm allowed to look at them).

Some of these systems use two of the three possible ways - then it's the
easy way only to change the configuration file(s), not to install a part
of the backup system on another machine too.

Viele Gruesse!
Helmut

-------------------------------------------------------------------------
----- Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

---------------------------------------------------------------------------
--- Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, May 12, 2011 at 3:56 PM, Peter Barker <peterb < at > zeta.org.au> wrote:
Can anyone explain why NFS is so incredibly slow for rsnapshot? (45 minutes compared to 4
minutes in my case).

NFS is attempting to replicate filesystem semantics over the net,
which means that operations like "what is the metadata for this file?"
can take orders of magnitude longer to complete than they do on a
local file. The first thing rsync does is trawl through all the
existing files to figure out what is there, and on NFS that will take
a _long_ time. Once rsync knows what needs changed, the actual
changes are probably pretty comparable, because the time will be
dominated by streaming bytes over the net. rsync should _still_ be
better for the actual transfer, because I believe it can pipeline the
filesystem operations versus the network operations, and also do some
amount of read-modify-write operations to make things efficient, but
difference in that case shouldn't be as huge in most cases.

Or, more simply put, one of the points of rsync is that it is an
efficient alternative to NFS.

-scott

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Peter,

Du meintest am 13.05.11:

Can anyone explain why NFS is so incredibly slow for rsnapshot? (45
minutes compared to 4 minutes in my case).

That's why I have changed the "cmd_cp" and "cmd_rm" lines.

"cp -al" from the source machine over nfs is slow, but using a script
which tells the target machine to do a local "cp -al" is fast.

Same result with "rm -rf" for the (temporary) "_delete" directory.

And if you have set "link_dest 1" then "rsync ... " over nfs is as slow
as "cp -al".

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss