Backup Central

Hi everybody,

I'm running rdiff-backup since more than one year on a bunch of machines
without any problem.

This is between 2 debian sarge machines (librsync 0.9.7 and rdiff-backup
0.13.4).

Since a few days (our Internet connection was accidently down), I can't
backup one of the machine (the other seems to be fine), and all I get is
the following error:

Read from remote host server3: Connection reset by peer
Exception 'Truncated header string (problem probably originated remotely)' raised of class 'rdiff_backup.connection.ConnectionReadError':
File "/usr/lib/python2.3/site-packages/rdiff_backup/robust.py", line 32, in check_common_error
try: return function(*args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Rdiff.py", line 89, in patch_local
if outrp: outrp.write_from_fileobj(patchfile)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 947, in write_from_fileobj
copyfileobj(fp, outfp)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 58, in copyfileobj
inbuf = inputfp.read(blocksize)
File "/usr/lib/python2.3/site-packages/rdiff_backup/librsync.py", line 76, in read
self._add_to_outbuf_once()
File "/usr/lib/python2.3/site-packages/rdiff_backup/librsync.py", line 85, in _add_to_outbuf_once
if not self.infile_eof: self._add_to_inbuf()
File "/usr/lib/python2.3/site-packages/rdiff_backup/librsync.py", line 95, in _add_to_inbuf
new_in = self.infile.read(blocksize)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 1155, in read
def read(self, length = -1): return self.file.read(length)
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 117, in read
if not self.addtobuffer(): break
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 132, in addtobuffer
type, data = self.iwf._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 401, in _get
if not self.buf: self.buf += self.file.read()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 513, in read
return self.connection.VirtualFile.readfromid(self.id, length)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 445, in __call__
return apply(self.connection.reval, (self.name,) + args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 365, in reval
result = self.get_response(req_num)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 314, in get_response
try: req_num, object = self._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 230, in _get
raise ConnectionReadError("Truncated header string (problem "

Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 259, in Main
take_action(rps)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 229, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 278, in Backup
backup.Mirror_and_increment(rpin, rpout, incdir)
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 226, in patch_and_increment
ITR(diff.index, diff)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 279, in __call__
last_branch.fast_process(*args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 606, in fast_process
if self.patch_to_temp(rp, diff_rorp, tf):
File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 498, in patch_to_temp
Rdiff.patch_local, (basis_rp, diff_rorp, new)) == 0: return 0
File "/usr/lib/python2.3/site-packages/rdiff_backup/robust.py", line 32, in check_common_error
try: return function(*args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Rdiff.py", line 89, in patch_local
if outrp: outrp.write_from_fileobj(patchfile)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 947, in write_from_fileobj
copyfileobj(fp, outfp)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 58, in copyfileobj
inbuf = inputfp.read(blocksize)
File "/usr/lib/python2.3/site-packages/rdiff_backup/librsync.py", line 76, in read
self._add_to_outbuf_once()
File "/usr/lib/python2.3/site-packages/rdiff_backup/librsync.py", line 85, in _add_to_outbuf_once
if not self.infile_eof: self._add_to_inbuf()
File "/usr/lib/python2.3/site-packages/rdiff_backup/librsync.py", line 95, in _add_to_inbuf
new_in = self.infile.read(blocksize)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 1155, in read
def read(self, length = -1): return self.file.read(length)
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 117, in read
if not self.addtobuffer(): break
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 132, in addtobuffer
type, data = self.iwf._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 401, in _get
if not self.buf: self.buf += self.file.read()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 513, in read
return self.connection.VirtualFile.readfromid(self.id, length)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 445, in __call__
return apply(self.connection.reval, (self.name,) + args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 365, in reval
result = self.get_response(req_num)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 314, in get_response
try: req_num, object = self._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 230, in _get
raise ConnectionReadError("Truncated header string (problem "
rdiff_backup.connection.ConnectionReadError: Truncated header string (problem probably originated remotely)
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method GzipFile.__del__ of <gzip open file '/var/backups/server3/rdiff-backup-data/file_statistics.2005-02-15T12:03:47+01:00.data.gz', mode 'wb' at 0x404f74a0 0x405432ec>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method GzipFile.__del__ of <gzip open file '/var/backups/server3/rdiff-backup-data/error_log.2005-02-15T12:03:47+01:00.data.gz', mode 'wb' at 0x40273360 0x405465ac>> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method GzipFile.__del__ of <gzip open file '/var/backups/server3/rdiff-backup-data/mirror_metadata.2005-02-15T12:03:47+01:00.snapshot.gz', mode 'wb' at 0x404f74e0 0x4054314c>> ignored

The connection dropped while getting the signature of a file if that can help.
On the remote machine, the rdiff-backup script is still running, long after the connection was dropped.

In fact it seems to choke on some files that are quite 'big' (around 50MB).
A tcpdump trace show that the remote server closes the TCP connection (RST), thus the Connection reset by peer.
Anyone have an idea of how I can fix that ?
How can I tell the remote site to verbose log everything it does ?

Running on a virgin directory seems to exhibit the same problem.

Thanks for your help,

Brice

On Tue, 15 Feb 2005, Brice Figureau wrote:

Since a few days (our Internet connection was accidently down), I can't
backup one of the machine (the other seems to be fine), and all I get is
the following error:

Read from remote host server3: Connection reset by peer
...
A tcpdump trace show that the remote server closes the TCP connection (RST), thus the Connection reset by peer.
Anyone have an idea of how I can fix that ?

i have this problem on and off and it typically clears up in a few days.
my wild guess is that it's related to heavy NAT traffic (my NAT is just
another linux box) when there are dozens of bittorrents going... but i've
never proved this wild guess true or false. i think the problem is
*somewhere* in the networking as opposed to in the rdiff-backups though...

i generally just run rdiff-backup again and again until it makes it the
whole way through the backup.

i've tried a bunch of stuff -- including enabling ssh protocol level
keep-alive (i.e. place "BatchMode yes" into your .ssh/config entry if
you're using such a thing).


How can I tell the remote site to verbose log everything it does ?

i've run tcpdumps and strace's on the remote side and still haven't
figured it out...


Running on a virgin directory seems to exhibit the same problem.

i've had success blowing away the rdiff-backup-data subdir, doing an
rsync, then doing an rdiff-backup --force... the rsync can always be made
to complete if you run it enough times (i.e. use -avP --delete ... make
sure you get your path specs right by using -n first).

using volume/fs snapshots can help because you can be sure the image
you're backing up isn't changing...

it can be frustrating.

-dean

Hi Dean,

On Tue, 2005-02-15 at 16:58 -0800, dean gaudet wrote:
On Tue, 15 Feb 2005, Brice Figureau wrote:

Since a few days (our Internet connection was accidently down), I can't
backup one of the machine (the other seems to be fine), and all I get is
the following error:

Read from remote host server3: Connection reset by peer
...
A tcpdump trace show that the remote server closes the TCP connection (RST), thus the Connection reset by peer.
Anyone have an idea of how I can fix that ?

i have this problem on and off and it typically clears up in a few days.
my wild guess is that it's related to heavy NAT traffic (my NAT is just
another linux box) when there are dozens of bittorrents going... but i've
never proved this wild guess true or false. i think the problem is
*somewhere* in the networking as opposed to in the rdiff-backups though...

You are right, this is a networking problem not related to rdiff-backup.
Even downloading with HTTP of a big file exhibits the same behaviour.
I guess I have to RMA the router...

I'm sorry for the false alert.

[snip]
-dean

Thank you for your help!

Brice

On Wed, 16 Feb 2005, Brice Figureau wrote:

You are right, this is a networking problem not related to rdiff-backup.
Even downloading with HTTP of a big file exhibits the same behaviour.
I guess I have to RMA the router...

you might look into something like <http://www.cs.wisc.edu/~zandy/rocks/>
... i briefly looked at it but gave up because i was too lazy to get the
udp it needed through my firewall.

a less sophisticated setup could no doubt provide seamless reconnects
beneath the covers for rdiff-backup ...

it's on my TODO list, but not high priority.

-dean

dean gaudet wrote:
a less sophisticated setup could no doubt provide seamless reconnects
beneath the covers for rdiff-backup ...

we use openvpn for this and it works ok with some ssh tweakings

dave