Backup Central

Hello,

        I'm trying to use rdiff backup to backup from my work server to a hard drive on my home network (needed for audit purposes - I know it sounds crazy but there you go!).  The network is very slow however it seems that rdiff backup starts working and then eventually grinds to a halt about halfway through never making it to the end - I'm backing up about 40Gb.  Obviously once the in ital backup is done the amount of transferred data will be smaller but rdiff-backup never manages to get through to the end it always jams somewhere.  Is there a way to determine if rdiff backup is still transferring the data but very very slowly so I can work out if rdiff-backup has frozen or if it is just on a go slow?


Cheers,


Neil
--
--

Neil Benn Msc
Director
Ziath Ltd
Phone :+44 (0)7508 107942
Website - http://www.ziath.com

IMPORTANT NOTICE:  This message, including any attached documents, is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law.  If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify Ziath Ltd immediately by email at info < at > ziath.com ([email]info < at > ziath.com[/email]). Thank you.

On 08/19/2010 08:57 AM, Neil Benn wrote:
Hello,

I'm trying to use rdiff backup to backup from my work server to
a hard drive on my home network (needed for audit purposes - I know it
sounds crazy but there you go!). The network is very slow however it
seems that rdiff backup starts working and then eventually grinds to a
halt about halfway through never making it to the end - I'm backing up
about 40Gb. Obviously once the in ital backup is done the amount of
transferred data will be smaller but rdiff-backup never manages to get
through to the end it always jams somewhere. Is there a way to
determine if rdiff backup is still transferring the data but very very
slowly so I can work out if rdiff-backup has frozen or if it is just on
a go slow?

You can use -v6 or higher to get a good picture of what's happening. If
several large files are involved then you could use wireshark to see if
there is much activity.

rdiff-backup isn't very good over unreliable links since it doesn't
resume partial transfers. I have solved this in two different ways in
the past:

1. Using openvpn. If you turn on the --ping-restart option then it will
detect a down link and restart it. This should be transparent to the
ssh session. You can also try messing with ssh options like BatchMode.

2. First rsync the data and then run rdiff-backup. The sequence of
commands that I run on the destination server is something like this:


#!/bin/bash
set -e

cp -rpl $dest $temp
rm -r $temp/rdiff-backup-data

while ! rsync -vaz --partial $host:$source/ $temp/
do
echo rsync failed, trying again
sleep 10
done

rdiff-backup --no-hard-links $temp $dest


Since this uses hard links it should not consume too much space during
the rsync process. You also might be able to use rsync's link-dest
option instead of the cp and rm commands.

I have successfully been using this method to back up hundreds of gigs
of data over long periods of time.

Steven

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

great thanks that is looking good

On 19 August 2010 17:27, Steven Willoughby <stevenw < at > nationalfitness.com ([email]stevenw < at > nationalfitness.com[/email])> wrote:
On 08/19/2010 08:57 AM, Neil Benn wrote:
Hello,

        I'm trying to use rdiff backup to backup from my work server to
a hard drive on my home network (needed for audit purposes - I know it
sounds crazy but there you go!).  The network is very slow however it
seems that rdiff backup starts working and then eventually grinds to a
halt about halfway through never making it to the end - I'm backing up
about 40Gb.  Obviously once the in ital backup is done the amount of
transferred data will be smaller but rdiff-backup never manages to get
through to the end it always jams somewhere.  Is there a way to
determine if rdiff backup is still transferring the data but very very
slowly so I can work out if rdiff-backup has frozen or if it is just on
a go slow?


You can use -v6 or higher to get a good picture of what's happening.  If several large files are involved then you could use wireshark to see if there is much activity.

rdiff-backup isn't very good over unreliable links since it doesn't resume partial transfers.  I have solved this in two different ways in the past:

1. Using openvpn.  If you turn on the --ping-restart option then it will detect a down link and restart it.  This should be transparent to the ssh session.  You can also try messing with ssh options like BatchMode.

2. First rsync the data and then run rdiff-backup.  The sequence of commands that I run on the destination server is something like this:


#!/bin/bash
set -e

cp -rpl $dest $temp
rm -r $temp/rdiff-backup-data

while ! rsync -vaz --partial $host:$source/ $temp/
do
       echo rsync failed, trying again
       sleep 10
done

rdiff-backup --no-hard-links $temp $dest


Since this uses hard links it should not consume too much space during the rsync process.  You also might be able to use rsync's link-dest option instead of the cp and rm commands.

I have successfully been using this method to back up hundreds of gigs of data over long periods of time.

Steven



--
--

Neil Benn Msc
Director
Ziath Ltd
Phone :+44 (0)7508 107942
Website - http://www.ziath.com

IMPORTANT NOTICE:  This message, including any attached documents, is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law.  If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify Ziath Ltd immediately by email at info < at > ziath.com ([email]info < at > ziath.com[/email]). Thank you.