Backup Central

BackupPC works good on my LAN but has problems backing-up a remote server of mine. I'm using rsync over ssh and have increased max ping to 500 because the remote server is far away. 


Almost always the transfer is ending after 1h01'or 1h02'. These values in seconds are 3660 and 3720. Maybe they appear in some option...? 
If I manually login to this remote ssh server, and leave my terminal untouched for 1h26" I get disconnected. Looks like this is the reason BackupPC gets disconnected, too. Is there any way to simulate the execution of commands inside BackupPC's session?
What I've tried and didn't work: 
- ServerAliveInterval=300, ServerAliveInterval=60 
- increased PingMaxMsec to 500 
- rsync options --timeout and --contimeout set to 20 and 200 seconds (both values tried on both options) 
- removing block-size option from rsync
Nothing worked. It usually stops in different files, independently of their size.
More info & logs here.


Thank you! 

Chris Mavrakis
cmavrakis.com

On Sun, 2012-03-04 at 01:40 +0200, Chris Mavrakis wrote:
BackupPC works good on my LAN but has problems backing-up a remote
server of mine. I'm using rsync over ssh and have increased max ping
to 500 because the remote server is far away.


Almost always the transfer is ending after 1h01'or 1h02'. These values
in seconds are 3660 and 3720. Maybe they appear in some option...?

If I manually login to this remote ssh server, and leave my terminal
untouched for 1h26" I get disconnected. Looks like this is the reason
BackupPC gets disconnected, too. Is there any way to simulate the
execution of commands inside BackupPC's session?

Firewall timeout somewhere?

Could be a login time limit?

--
Tim Fletcher <tim < at > night-shade.org.uk>


------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Sat, Mar 3, 2012 at 5:40 PM, Chris Mavrakis <cmavr8 < at > gmail.com> wrote:
BackupPC works good on my LAN but has problems backing-up a remote server of
mine. I'm using rsync over ssh and have increased max ping to 500 because
the remote server is far away.

Almost always the transfer is ending after 1h01'or 1h02'. These values in
seconds are 3660 and 3720. Maybe they appear in some option...?

If I manually login to this remote ssh server, and leave my terminal
untouched for 1h26" I get disconnected. Looks like this is the reason
BackupPC gets disconnected, too. Is there any way to simulate the execution
of commands inside BackupPC's session?

What I've tried and didn't work:

- ServerAliveInterval=300, ServerAliveInterval=60

- increased PingMaxMsec to 500

- rsync options --timeout and --contimeout set to 20 and 200 seconds (both
values tried on both options)

- removing block-size option from rsync

Nothing worked. It usually stops in different files, independently of their
size.

I'd expect the ssh ServerAliveInterval to fix the most likely cases of
idle timeout on nat routers or firewalls. Maybe something has an
absolute connection length timeout. Do you have access to the
routers at either or both ends?

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Thanks.. Maybe it's one of the things Tim said (firewall/ssh limit), but I can't influence these settings as the remote server is on a shared hosting plan. (my own firewall is ok) I have no control over sshd_config etc... or the remote routers/firewall.



I also tried to add in the rsync command :  -o ServerAliveInterval=15 -o ServerAliveCountMax=10.


Could we just tell BackupPC to retry a few times even if the connection is terminated? If I manually issue backup commands one after the other (after each failure, every 1h1') I eventually get a full back-up of the remote host. But it would be nice to have Backuppc do it automatically.

Chris Mavrakis
cmavrakis.com



On Sun, Mar 4, 2012 at 5:48 AM, Les Mikesell <lesmikesell < at > gmail.com ([email]lesmikesell < at > gmail.com[/email])> wrote:
On Sat, Mar 3, 2012 at 5:40 PM, Chris Mavrakis <cmavr8 < at > gmail.com ([email]cmavr8 < at > gmail.com[/email])> wrote:
BackupPC works good on my LAN but has problems backing-up a remote server of
mine. I'm using rsync over ssh and have increased max ping to 500 because
the remote server is far away.

Almost always the transfer is ending after 1h01'or 1h02'. These values in
seconds are 3660 and 3720. Maybe they appear in some option...?

If I manually login to this remote ssh server, and leave my terminal
untouched for 1h26" I get disconnected. Looks like this is the reason
BackupPC gets disconnected, too. Is there any way to simulate the execution
of commands inside BackupPC's session?

What I've tried and didn't work:

- ServerAliveInterval=300, ServerAliveInterval=60

- increased PingMaxMsec to 500

- rsync options --timeout and --contimeout set to 20 and 200 seconds (both
values tried on both options)

- removing block-size option from rsync

Nothing worked. It usually stops in different files, independently of their
size.



I'd expect the ssh ServerAliveInterval to fix the most likely cases of
idle timeout on nat routers or firewalls.   Maybe something has an
absolute connection length timeout.   Do you have access to the
routers at either or both ends?

--
   Les Mikesell
     lesmikesell < at > gmail.com ([email]lesmikesell < at > gmail.com[/email])

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net ([email]BackupPC-users < at > lists.sourceforge.net[/email])
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Ok, I verified it, whatever one does, the ssh connection is terminated after 1h1'. So the only "solution" to this would be to have backuppc retry again and again until the sum of the files is transfered, and the final backup is successful .

Maybe I can simulate this behavior by setting 
$Conf{IncrPeriod} = 0.03; (roughly every 40 minutes) for the first copy of the files, and then reset it back to the default 0.97.



Chris Mavrakis
cmavrakis.com



On Sun, Mar 4, 2012 at 11:15 AM, Chris Mavrakis <cmavr8 < at > gmail.com ([email]cmavr8 < at > gmail.com[/email])> wrote:
Thanks.. Maybe it's one of the things Tim said (firewall/ssh limit), but I can't influence these settings as the remote server is on a shared hosting plan. (my own firewall is ok) I have no control over sshd_config etc... or the remote routers/firewall.



I also tried to add in the rsync command :  -o ServerAliveInterval=15 -o ServerAliveCountMax=10.


Could we just tell BackupPC to retry a few times even if the connection is terminated? If I manually issue backup commands one after the other (after each failure, every 1h1') I eventually get a full back-up of the remote host. But it would be nice to have Backuppc do it automatically.

Chris Mavrakis
cmavrakis.com



On Sun, Mar 4, 2012 at 5:48 AM, Les Mikesell <lesmikesell < at > gmail.com ([email]lesmikesell < at > gmail.com[/email])> wrote:
On Sat, Mar 3, 2012 at 5:40 PM, Chris Mavrakis <cmavr8 < at > gmail.com ([email]cmavr8 < at > gmail.com[/email])> wrote:
BackupPC works good on my LAN but has problems backing-up a remote server of
mine. I'm using rsync over ssh and have increased max ping to 500 because
the remote server is far away.

Almost always the transfer is ending after 1h01'or 1h02'. These values in
seconds are 3660 and 3720. Maybe they appear in some option...?

If I manually login to this remote ssh server, and leave my terminal
untouched for 1h26" I get disconnected. Looks like this is the reason
BackupPC gets disconnected, too. Is there any way to simulate the execution
of commands inside BackupPC's session?

What I've tried and didn't work:

- ServerAliveInterval=300, ServerAliveInterval=60

- increased PingMaxMsec to 500

- rsync options --timeout and --contimeout set to 20 and 200 seconds (both
values tried on both options)

- removing block-size option from rsync

Nothing worked. It usually stops in different files, independently of their
size.



I'd expect the ssh ServerAliveInterval to fix the most likely cases of
idle timeout on nat routers or firewalls.   Maybe something has an
absolute connection length timeout.   Do you have access to the
routers at either or both ends?

--
   Les Mikesell
     lesmikesell < at > gmail.com ([email]lesmikesell < at > gmail.com[/email])

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net ([email]BackupPC-users < at > lists.sourceforge.net[/email])
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Sun, Mar 4, 2012 at 3:22 PM, Chris Mavrakis <cmavr8 < at > gmail.com ([email]cmavr8 < at > gmail.com[/email])> wrote:
Ok, I verified it, whatever one does, the ssh connection is terminated after 1h1'. So the only "solution" to this would be to have backuppc retry again and again until the sum of the files is transfered, and the final backup is successful .



You could write a small script which checks the return status of rsync and restart rsync as long as it doesn't return 0. Then, replace the rsync command in backuppc with your script and pass all the arguments to rsync.


This way, whenever it disconnects, the script will simply restart rsync (thus continue the backup) until it is done.

On Sun, Mar 4, 2012 at 8:22 AM, Chris Mavrakis <cmavr8 < at > gmail.com> wrote:
Ok, I verified it, whatever one does, the ssh connection is terminated after
1h1'. So the only "solution" to this would be to have backuppc retry again
and again until the sum of the files is transfered, and the final backup is
successful .

Maybe I can simulate this behavior by setting

$Conf{IncrPeriod} = 0.03;

(roughly every 40 minutes) for the first copy of the files, and then reset
it back to the default 0.97.


Incremental runs will start from the beginning each time. An
incomplete full run should result in a 'partial' backup and pick up
from there on the next attempt, though. So force a full and keep doing
it until it completes. It might help a little to add the '-C' option
to ssh for compression if you haven't already. And perhaps you can
adjust the incrementals to use levels or do fulls more frequently to
not have so much data to copy at once.

If you have enough disk space locally, you might even use a scripted
rsync to do the initial copy, then let backuppc copy that for history.
A native rsync will be somewhat faster, can use the -z option for
compression, and can be restarted without much of a speed penalty.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Thanks again for your answers. I just did subsequent full backups until all 5GB of data were copied (usually getting ~120MB transferred each time). 


Then, an incremental backup takes less than an hour so it works ok!
But if I happen to have big changes I might implement the custom rsync method..


Also, I put -C and I'll see how much more it will download in 1h01'.


Thanks!

Chris Mavrakis
cmavrakis.com



On Sun, Mar 4, 2012 at 6:53 PM, Les Mikesell <lesmikesell < at > gmail.com ([email]lesmikesell < at > gmail.com[/email])> wrote:
On Sun, Mar 4, 2012 at 8:22 AM, Chris Mavrakis <cmavr8 < at > gmail.com ([email]cmavr8 < at > gmail.com[/email])> wrote:
Ok, I verified it, whatever one does, the ssh connection is terminated after
1h1'. So the only "solution" to this would be to have backuppc retry again
and again until the sum of the files is transfered, and the final backup is
successful .

Maybe I can simulate this behavior by setting

$Conf{IncrPeriod} = 0.03;

(roughly every 40 minutes) for the first copy of the files, and then reset
it back to the default 0.97.



Incremental runs will start from the beginning each time.   An
incomplete full run should result in a 'partial' backup and pick up
from there on the next attempt, though. So force a full and keep doing
it until it completes.   It might help a little to add the '-C' option
to ssh for compression if you haven't already.   And perhaps you can
adjust the incrementals to use levels or do fulls more frequently to
not have so much data to  copy at once.

If you have enough disk space locally, you might even use a scripted
rsync to do the initial copy, then let backuppc copy that for history.
 A native rsync will be somewhat faster, can use the -z option for
compression, and can be restarted without much of a speed penalty.

--
 Les Mikesell
    lesmikesell < at > gmail.com ([email]lesmikesell < at > gmail.com[/email])

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net ([email]BackupPC-users < at > lists.sourceforge.net[/email])
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/