Backup Central

I have a DLink DI-604 router, which times out ssh connections unavoidably
after a certain amount of inactivity. I suspect, but don't know for
certain, that this is why my rdiff-backup jobs usually don't
complete--I'm trying to mirror a 20G drive (over a standard cable modem
connection with slow uplink, and a slow CPU on the mirror) so I believe
that there are sometimes long periods of inactivity and thus the ssh
connection is dropped.

The error I get is "Connection reset by peer" and then a python
backtrace.

This was discussed over a year ago on this list:

http://lists.gnu.org/archive/html/rdiff-backup-users/2003-03/msg00119.html

With no real resolution.

Any new ideas about how this problem might be solved?
--
Adam Rosi-Kessel
http://adam.rosi-kessel.org

On 1 Jul 2004, Adam Rosi-Kessel <adam < at > rosi-kessel.org> wrote:
I have a DLink DI-604 router, which times out ssh connections unavoidably
after a certain amount of inactivity. I suspect, but don't know for
certain, that this is why my rdiff-backup jobs usually don't
complete--I'm trying to mirror a 20G drive (over a standard cable modem
connection with slow uplink, and a slow CPU on the mirror) so I believe
that there are sometimes long periods of inactivity and thus the ssh
connection is dropped.

You probably want to set the ServerAliveInterval option in
~/.ssh/config (aka ProtocolKeepAlives, aka something else for SSHv1).
This makes it send a little bit of traffic every while to keep the
connection alive from the router's point of view. Seems to work for
me.

--
Martin

On Fri, Jul 02, 2004 at 02:05:17PM +1000, Martin Pool wrote:
You probably want to set the ServerAliveInterval option in
~/.ssh/config (aka ProtocolKeepAlives, aka something else for SSHv1).
This makes it send a little bit of traffic every while to keep the
connection alive from the router's point of view. Seems to work for
me.

And depending on the timeouts involved, you may need to
increase the frequence of the keepalives. On Linux you
can do that with something like:
echo 900 >/proc/sys/net/ipv4/tcp_keepalive_time

If the TCP keepalives are not enough, investigate how
some SSH clients have the ability to insert an SSH NOOP
periodically on an idle connection. (PuTTY for example.
There are patches that allow you to add it to OpenSSH.
http://www.sc.isc.tohoku.ac.jp/~hgot/sources/openssh-watchdog.html

Although I must admit I'm surprised that your rdiff-backup
session could go silent for so long that the connection looks
idle to the router.

On Fri, Jul 02, 2004 at 02:05:17PM +1000, Martin Pool wrote:
I have a DLink DI-604 router, which times out ssh connections unavoidably
after a certain amount of inactivity. I suspect, but don't know for
certain, that this is why my rdiff-backup jobs usually don't
complete--I'm trying to mirror a 20G drive (over a standard cable modem
connection with slow uplink, and a slow CPU on the mirror) so I believe
that there are sometimes long periods of inactivity and thus the ssh
connection is dropped.
You probably want to set the ServerAliveInterval option in
~/.ssh/config (aka ProtocolKeepAlives, aka something else for SSHv1).
This makes it send a little bit of traffic every while to keep the
connection alive from the router's point of view. Seems to work for
me.

Thanks; that seems to fix the ssh timeout problem. Now I have another
error that arises consistently every time. I don't know that the
UpdateError has anything to do with the crash--I'll try to increase the
verbosity and see if I get any more information. It takes me a few hours
to get to this error each time, so it's hard to troubleshoot.

Any ideas for fixing this? Do I need to wipe out my previous increments
and start fresh?

UpdateError ibm/var/lib/mailman/locks/master-qrunner File changed from regular file before signature
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 259, in Main
take_action(rps)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 229, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 278, in Backup
backup.Mirror_and_increment(rpin, rpout, incdir)
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 226, in patch_and_increment
ITR(diff.index, diff)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 279, in __call__
last_branch.fast_process(*args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 606, in fast_process
if self.patch_to_temp(rp, diff_rorp, tf):
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 487, in patch_to_temp
Hardlink.link_rp(diff_rorp, new, self.basis_root_rp)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Hardlink.py", line 116, in link_rp
dest_rpath.hardlink(dest_link_rpath.path)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 797, in hardlink
self.conn.os.link(linkpath, self.path)
OSError: [Errno 2] No such file or directory
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method GzipFile.__del__ of <gzip open file '/backup/bcn/rdiff-backup-data/file_statistics.2004-07-02T08:30:48-04:00.data.gz', mode 'wb' at 0x404a30a0 0x404e844c>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method GzipFile.__del__ of <gzip open file '/backup/bcn/rdiff-backup-data/error_log.2004-07-02T08:30:48-04:00.data.gz', mode 'wb' at 0x404a3120 0x404dc18c>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method GzipFile.__del__ of <gzip open file '/backup/bcn/rdiff-backup-data/mirror_metadata.2004-07-02T08:30:48-04:00.snapshot.gz', mode 'wb' at 0x404a3220 0x404e87ac>> ignored
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 259, in Main
take_action(rps)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 227, in take_action
connection.PipeConnection(sys.stdin, sys.stdout).Server()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 352, in Server
self.get_response(-1)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 314, in get_response
try: req_num, object = self._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 230, in _get
raise ConnectionReadError("Truncated header string (problem "
rdiff_backup.connection.ConnectionReadError: Truncated header string (problem probably originated remotely)
--
Adam Rosi-Kessel
http://adam.rosi-kessel.org

On Fri, Jul 02, 2004, Jim Tittsler wrote:
Although I must admit I'm surprised that your rdiff-backup session
could go silent for so long that the connection looks idle to the
router.

I'm guessing that the timeout is actually in the regression from the past
failure, which takes an hour or more. The ssh connection is open during
the connection, but I don't know that any data is being sent--presumably
a regression doesn't require any information about the source.

I wonder if rdiff-backup might be able to send null packets or something
during a long silent period? I'm sure other people have this problem as
well.

Or alternatively, could rdiff-backup check first to see if it needs to
regress, and do that before establishing a connection? Is there any
reason to establish the connection first?
--
Adam Rosi-Kessel
http://adam.rosi-kessel.org