Backup Central

Hi folks,

I'm very excited about rdiff-backup's value offering. It's a step
forward from Grenville Armitage approach of using rsync and cpio. I'd
like to take it further eventually, by adding permissions metadata so
that it can run as an unpriveleged user, and support NTFS ownership
information.

Unfortuntely, I can't even get it to restore from remote backup yet.

I created my backup as follows:

rdiff-backup ~/backup-test sheldonh < at > urchin.clue.co.za::backup-test
# Make some changes to ~/backup-test on the local host
rdiff-backup ~/backup-test sheldonh < at > urchin.clue.co.za::backup-test
# Repeat, and be amazed at the elegance of rdiff-backup-data on
# the remote host.

Then I waited 20 minutes and tried to restore as follows:

rdiff-backup --restore-as-of 10m \
sheldonh < at > urchin.clue.co.za::backup-test \
/home/sheldonh/restore-test

And lo, I got this error:

Truncated header string (problem probably originated remotely)

The trace is attached.

The two hosts are connected on mostly idle 100BaseTX ethernet, and I
don't get connection drops between them.

Does anyone have any ideas? I'm using rdiff-backup-0.13.4 on Gentoo
Linux (2.6.10) with glibc-2.3.4.20040808. I've checked the FAQ and
searched the mailing list archive. This question has been asked before,
but I couldn't find any appropriate answers. For some people, they
really did have a flakey network situation. But I don't,

Thanks,
Sheldon.

Warning: ownership cannot be changed on filesystem at /home/sheldonh/restore-test
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
Characters needing quoting ''
Ownership changing Off
Hard linking On
fsync() directories On
Directory inc permissions On
Access control lists Off
Extended attributes Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
Access control lists Off
Extended attributes Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 259, in Main
take_action(rps)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 239, in take_action
elif action == "restore-as-of": Restore(rps[0], rps[1], 1)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 465, in Restore
inc_rpath, dest_rp, time)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 45, in Restore
TargetS.patch(target, diff_iter)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 309, in patch
ITR(diff.index, diff)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 279, in __call__
last_branch.fast_process(*args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 625, in fast_process
self.patch_to_temp(rp, diff_rorp, tf)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 633, in patch_to_temp
rpath.copy(diff_rorp, new)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 96, in copy
if rpin.isreg(): copy_reg_file(rpin, rpout, compress)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 118, in copy_reg_file
rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 947, in write_from_fileobj
copyfileobj(fp, outfp)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 58, in copyfileobj
inbuf = inputfp.read(blocksize)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 1155, in read
def read(self, length = -1): return self.file.read(length)
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 117, in read
if not self.addtobuffer(): break
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 132, in addtobuffer
type, data = self.iwf._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 401, in _get
if not self.buf: self.buf += self.file.read()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 513, in read
return self.connection.VirtualFile.readfromid(self.id, length)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 445, in __call__
return apply(self.connection.reval, (self.name,) + args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 367, in reval
if isinstance(result, Exception): raise result
AssertionError: (('diskimages',), ('diskimages', 'dell', 'ED5049A0.tar.gz'))
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 259, in Main
take_action(rps)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 227, in take_action
connection.PipeConnection(sys.stdin, sys.stdout).Server()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 352, in Server
self.get_response(-1)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 314, in get_response
try: req_num, object = self._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 230, in _get
raise ConnectionReadError("Truncated header string (problem "
rdiff_backup.connection.ConnectionReadError: Truncated header string (problem probably originated remotely)

Any takers on this? I'm surprised that this apparent show stopper has
no useful advice in the archives or the FAQ. Is it a case of "maybe it
works for you, if not, you're on your own"?

FWIW, I got one private reply containing a suggestion that made no
difference. :-)

Ciao,
Sheldon.

On Wed, 2005-03-09 at 16:20 +0200, Sheldon Hearn wrote:

Quote:

Hi folks,

I'm very excited about rdiff-backup's value offering. It's a step
forward from Grenville Armitage approach of using rsync and cpio. I'd
like to take it further eventually, by adding permissions metadata so
that it can run as an unpriveleged user, and support NTFS ownership
information.

Unfortuntely, I can't even get it to restore from remote backup yet.

I created my backup as follows:

rdiff-backup ~/backup-test sheldonh < at > urchin.clue.co.za::backup-test
# Make some changes to ~/backup-test on the local host
rdiff-backup ~/backup-test sheldonh < at > urchin.clue.co.za::backup-test
# Repeat, and be amazed at the elegance of rdiff-backup-data on
# the remote host.

Then I waited 20 minutes and tried to restore as follows:

rdiff-backup --restore-as-of 10m \
sheldonh < at > urchin.clue.co.za::backup-test \
/home/sheldonh/restore-test

And lo, I got this error:

Truncated header string (problem probably originated remotely)

The trace is attached.

The two hosts are connected on mostly idle 100BaseTX ethernet, and I
don't get connection drops between them.

Does anyone have any ideas? I'm using rdiff-backup-0.13.4 on Gentoo
Linux (2.6.10) with glibc-2.3.4.20040808. I've checked the FAQ and
searched the mailing list archive. This question has been asked before,
but I couldn't find any appropriate answers. For some people, they
really did have a flakey network situation. But I don't,

Thanks,
Sheldon.
plain text document attachment (trace.txt)
Warning: ownership cannot be changed on filesystem at /home/sheldonh/restore-test
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
Characters needing quoting ''
Ownership changing Off
Hard linking On
fsync() directories On
Directory inc permissions On
Access control lists Off
Extended attributes Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
Access control lists Off
Extended attributes Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 259, in Main
take_action(rps)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 239, in take_action
elif action == "restore-as-of": Restore(rps[0], rps[1], 1)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 465, in Restore
inc_rpath, dest_rp, time)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 45, in Restore
TargetS.patch(target, diff_iter)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 309, in patch
ITR(diff.index, diff)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 279, in __call__
last_branch.fast_process(*args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 625, in fast_process
self.patch_to_temp(rp, diff_rorp, tf)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 633, in patch_to_temp
rpath.copy(diff_rorp, new)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 96, in copy
if rpin.isreg(): copy_reg_file(rpin, rpout, compress)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 118, in copy_reg_file
rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 947, in write_from_fileobj
copyfileobj(fp, outfp)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 58, in copyfileobj
inbuf = inputfp.read(blocksize)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 1155, in read
def read(self, length = -1): return self.file.read(length)
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 117, in read
if not self.addtobuffer(): break
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 132, in addtobuffer
type, data = self.iwf._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 401, in _get
if not self.buf: self.buf += self.file.read()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 513, in read
return self.connection.VirtualFile.readfromid(self.id, length)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 445, in __call__
return apply(self.connection.reval, (self.name,) + args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 367, in reval
if isinstance(result, Exception): raise result
AssertionError: (('diskimages',), ('diskimages', 'dell', 'ED5049A0.tar.gz'))
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 259, in Main
take_action(rps)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 227, in take_action
connection.PipeConnection(sys.stdin, sys.stdout).Server()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 352, in Server
self.get_response(-1)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 314, in get_response
try: req_num, object = self._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 230, in _get
raise ConnectionReadError("Truncated header string (problem "
rdiff_backup.connection.ConnectionReadError: Truncated header string (problem probably originated remotely)
_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

On Fri, 2005-03-11 at 10:06 +0200, Sheldon Hearn wrote:

Quote:

Any takers on this? I'm surprised that this apparent show stopper has
no useful advice in the archives or the FAQ. Is it a case of "maybe it
works for you, if not, you're on your own"?

FWIW, I got one private reply containing a suggestion that made no
difference.

The out-of-band feedback I'm getting on this suggests that remote
restores are not a common use case.

Can anyone confirm that they actually work in real-world usage?

Ciao,
Sheldon.

Someone suggested I try a local restore (on the backup server), to
narrow down the problem.

A local restore on the backup server works. It's only remote restores
that don't work.

So, are remote restores working for anyone?

Ciao,
Sheldon.

On Fri, 11 Mar 2005 12:00:25 +0200
Sheldon Hearn <sheldonh < at > clue.co.za> wrote:

Quote:

So, are remote restores working for anyone?

Yes, although they are slow. That doesn't help you, I realise, but if you
can do local restores you can always tar them up or whatever and restore
that way.

Keith

--
----------------------------------------------------------------------
Business computing support: http://www.tiger-computing.co.uk
Linux consultancy: http://www.TheLinuxConsultancy.co.uk
----------------------------------------------------------------------

On Fri, 2005-03-11 at 12:12 +0000, Keith Edmunds wrote:

Quote:

Quote:

So, are remote restores working for anyone?

Yes, although they are slow. That doesn't help you, I realise, but if you
can do local restores you can always tar them up or whatever and restore
that way.

Actually, it helps enormously, because it means there's something
locally wrong, rather than that it's a complete waste of time. Thank
you :-)

In fact, I've now managed to narrow it down to this:

1) If I restore to a not-yet-existing directory, I get the dreaded
"Truncated header string" error.

2) If I restore to an empty directory, I still get the error.

3) If I restore to a fully populated directory, I get no error. This
just means that the problem is likely to exist around the retrieval,
transmission and application of an rdiff.

4) If I restore to a populated directory from which I've removed the
file that causes the error (ED5049A0.tar.gz), I get no error.

5) If I create a new backup test containing only the file that causes
the error, and test that instead, I get no error.

This smells like stale state in an iterator. Unfortunately, it means I
can't really submit a useful "how to repeat".

What's interesting is that this has been reported many times, by many
people, and doesn't seem to have been addressed. Is rdiff-backup
unmaintained, or is it just that folks with the clue to fix it haven't
seen the problem themselves?

Ciao,
Sheldon.

Perhaps it is an issue with directory creation. Your 5 cases below don't
exclude this case.

You could try doing case #3 after removing all the files (but not the
directories), which you can do by running:

find . -type f -exec rm {} \;

Gary

Sheldon Hearn wrote:

Quote:

In fact, I've now managed to narrow it down to this:

1) If I restore to a not-yet-existing directory, I get the dreaded
"Truncated header string" error.

2) If I restore to an empty directory, I still get the error.

3) If I restore to a fully populated directory, I get no error. This
just means that the problem is likely to exist around the retrieval,
transmission and application of an rdiff.

4) If I restore to a populated directory from which I've removed the
file that causes the error (ED5049A0.tar.gz), I get no error.

5) If I create a new backup test containing only the file that causes
the error, and test that instead, I get no error.

On Fri, 11 Mar 2005 14:50:49 +0200
Sheldon Hearn <sheldonh < at > clue.co.za> wrote:

Quote:

What's interesting is that this has been reported many times, by many
people, and doesn't seem to have been addressed. Is rdiff-backup
unmaintained, or is it just that folks with the clue to fix it haven't
seen the problem themselves?

I'd hesitate to say it's unmaintained - while he was around, Ben (the
author) was extremely helpful. As I understand it he's changed his working
day (I think he was at university and how now left), and doesn't seem to
have time for rdiff-backup anymore. So yes, I suppose it is unmaintained.
There are a few patches around, but it would be good to collate them.

Is anyone in touch with Ben?

--
----------------------------------------------------------------------
Business computing support: http://www.tiger-computing.co.uk
Linux consultancy: http://www.TheLinuxConsultancy.co.uk
----------------------------------------------------------------------

Quote:

So, are remote restores working for anyone?

Like you I've just started evaluating rdiff-backup, but I'm not using it
in anger yet. I built 0.12.7 and librsync 0.9.7 from source. Clients are
SuSE Linux (various versions from 7.3 to 9.0) using Python 2.4. Server
is OpenBSD 3.6 using Python 2.3.4.

I've just done some remote restores without any problems. Slow, but
reliable, both to populated and empty directories. I'll be testing more
thoroughly over the next few days.

Maybe a significant difference between your setup and mine is that I'm
using the latest stable version and you're using the development
version? Maybe it's worth giving 0.12.7 a try?


Peter Hamilton

--- Keith Edmunds <keith < at > midnighthax.com> wrote:

Quote:

On Fri, 11 Mar 2005 14:50:49 +0200
Sheldon Hearn <sheldonh < at > clue.co.za> wrote:

Quote:

What's interesting is that this has been reported

many times, by many

Quote:

people, and doesn't seem to have been addressed.

Is rdiff-backup

Quote:

unmaintained, or is it just that folks with the

clue to fix it haven't

Quote:

seen the problem themselves?

I'd hesitate to say it's unmaintained - while he was
around, Ben (the
author) was extremely helpful. As I understand it
he's changed his working
day (I think he was at university and how now left),
and doesn't seem to
have time for rdiff-backup anymore. So yes, I
suppose it is unmaintained.
There are a few patches around, but it would be good
to collate them.

saradiya has already suggested this with an earlier
email but there weren't any takers. How about taking
him up on the offer. It would be a shame to see such a
great project bite it.

Quote:

Is anyone in touch with Ben?

I have tried emailing him several times using the
several emails that were available, but to no avail.

Quote:

--

----------------------------------------------------------------------

Quote:

Business computing support:
http://www.tiger-computing.co.uk
Linux consultancy:
http://www.TheLinuxConsultancy.co.uk

----------------------------------------------------------------------

Quote:

_______________________________________________
rdiff-backup-users mailing list at
rdiff-backup-users < at > nongnu.org

http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users

Quote:

Wiki URL:

http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki



__________________________________
Do you Yahoo!?
Yahoo! Small Business - Try our new resources site!
http://smallbusiness.yahoo.com/resources/

On Fri, 2005-03-11 at 08:29 -0500, Gary Mulder wrote:

Quote:

Perhaps it is an issue with directory creation. Your 5 cases below don't
exclude this case.

You could try doing case #3 after removing all the files (but not the
directories), which you can do by running:

find . -type f -exec rm {} \;

Worth trying, but not enlightening; same "Truncated header string"
response.

Next up: trying 0.12.7 instead of 0.13.4, as per Peter Hamilton's
suggestion. I was only using the latter because that's what "emerge
rdiff-backup" scored me and I wasn't looking closely. :-)

Next
Ciao,
Sheldon.

On Fri, 2005-03-11 at 13:41 +0000, Peter Hamilton wrote:

Quote:

Maybe a significant difference between your setup and mine is that I'm
using the latest stable version and you're using the development
version? Maybe it's worth giving 0.12.7 a try?

Peter, you're the man!

That's it. I can't reproduce the problem with 0.12.7.

I'll file a Gentoo bug tomorrow, recommending that 0.13.4 be marked
unstable and 0.12.7 be restored.

Thanks!
Sheldon.

Hello!

Quote:

That's it. I can't reproduce the problem with 0.12.7.

I'll file a Gentoo bug tomorrow, recommending that 0.13.4 be marked
unstable and 0.12.7 be restored.

Is it possible to downgrade and re-use the already existing rdiff
metadata files?

Bjoern

Quote:

That's it. I can't reproduce the problem with 0.12.7.

Interesting. So that answers the question I was about to ask the list:
whether I should consider "upgrading" to the development version.

I've been giving 0.12.7 a really hard time, with no problems. I feel
quite confident about using it for serious backups now.

Peter Hamilton

On Tue, 2005-03-15 at 18:10 +0200, Sheldon Hearn wrote:

Quote:

I'll file a Gentoo bug tomorrow, recommending that 0.13.4 be marked
unstable and 0.12.7 be restored.

Gentoo Linux users can track the bug at:

http://bugs.gentoo.org/show_bug.cgi?id=85462

Ciao,
Sheldon.