Backup Central

Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

At first I thought it might be a gzip issue, so I tried the following:

rdiff-backup --no-compression root < at > gw::test test

But it failed with exactly the same error.
Both systems use reiserfs as the file system, rdiff-backup version
0.13.4 and python version 2.3.3. I have no idea what could be causing
this.

Ben <ben < at > ukotic.net>
wrote the following on Tue, 01 Jun 2004 14:41:15 +1000
Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

This looks like a librsync problem. (When backing up initially,
rdiff-backup doesn't use librsync, just when updating.) Make sure you
are running the latest version of librsync. You can test it in
isolation by running "rdiff" (an binary included in librsync) in a
sig/diff/patch cycle.

Check also the librsync mailing lists; there has been much discussion
there of largefile issues.


--
Ben Escoto

rsync actually works:

rsync -auvroge ssh root < at > gw:test test

It first copies over the file initially, then I add some data to it at
the remote end, do another transfer and its fine. Pretty weird.

On Tue, 2004-06-01 at 15:29, Ben Escoto wrote:
Ben <ben < at > ukotic.net>
wrote the following on Tue, 01 Jun 2004 14:41:15 +1000
Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

This looks like a librsync problem. (When backing up initially,
rdiff-backup doesn't use librsync, just when updating.) Make sure you
are running the latest version of librsync. You can test it in
isolation by running "rdiff" (an binary included in librsync) in a
sig/diff/patch cycle.

Check also the librsync mailing lists; there has been much discussion
there of largefile issues.

I'm trying an rdiff now, I will let you know how it goes.

On Tue, 2004-06-01 at 15:29, Ben Escoto wrote:
Ben <ben < at > ukotic.net>
wrote the following on Tue, 01 Jun 2004 14:41:15 +1000
Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

This looks like a librsync problem. (When backing up initially,
rdiff-backup doesn't use librsync, just when updating.) Make sure you
are running the latest version of librsync. You can test it in
isolation by running "rdiff" (an binary included in librsync) in a
sig/diff/patch cycle.

Check also the librsync mailing lists; there has been much discussion
there of largefile issues.

Replying to Ben:
rsync actually works:

rsync -auvroge ssh root < at > gw:test test

rsync isn't dependent on librsync.

If you are compiling and installing librsync yourself, from sources,
you can do 'make test' or 'make check' (trouble remembering exact
command) in librsync build directory. It should pass all tests.

--
Paul P 'Stingray' Komkoff Jr // http://stingr.net/key <- my pgp key
This message represents the official view of the voices in my head

On 1 Jun 2004, Paul P Komkoff Jr <i < at > stingr.net> wrote:
Replying to Ben:
rsync actually works:

rsync -auvroge ssh root < at > gw:test test

rsync isn't dependent on librsync.

If you are compiling and installing librsync yourself, from sources,
you can do 'make test' or 'make check' (trouble remembering exact
command) in librsync build directory. It should pass all tests.

Yes, it should, but I don't think the 'make check' in 0.9.6 tests that
large files work. It should probably have that as an optional test,
given how much trouble it has caused.

You can try './rdiff --version' which will tell you if it thinks it
supports large files, but that might not be correct.

--
Martin

Ok I checked the version:
Capabilities: 64 bit files, gzip, bzip2, trace disabled

So thats good.
I ran an rdiff test:

ls -lh
total 2.3G
-rw-r--r-- 1 root root 2.2G Jun 1 15:25 bigfile
-rw-r--r-- 1 root root 92M Jun 1 17:34 bigfile2

rdiff signature bigfile2 bigfile2.sig
rdiff delta bigfile2.sig bigfile bigfile1-2.delta
rdiff patch bigfile2 bigfile1-2.delta newbigfile

ls -lh
total 4.9G
-rw-r--r-- 1 root root 2.2G Jun 1 15:25 bigfile
-rw-r--r-- 1 root root 546M Jun 1 17:45 bigfile1-2.delta
-rw-r--r-- 1 root root 92M Jun 1 17:34 bigfile2
-rw-r--r-- 1 root root 548K Jun 1 17:39 bigfile2.sig
-rw-r--r-- 1 root root 2.2G Jun 1 17:50 newbigfile

So it looks like this actually worked. I will try it on the remote host
and see if the same thing happens. I read that people were getting these
errors when running those commands.

Also, I recompiled librsync as a shared library. I also noticed that it
puts the libraries in /usr/local/lib and rdiff-backup was looking in
/usr/lib. I re-installed rdiff-backup to use /usr/local/lib rsync
libraries and re-ran rdiff-backup (done this on both machines). This
didn't seem to make a difference though.



On Tue, 2004-06-01 at 17:42, Martin Pool wrote:
On 1 Jun 2004, Paul P Komkoff Jr <i < at > stingr.net> wrote:
Replying to Ben:
rsync actually works:

rsync -auvroge ssh root < at > gw:test test

rsync isn't dependent on librsync.

If you are compiling and installing librsync yourself, from sources,
you can do 'make test' or 'make check' (trouble remembering exact
command) in librsync build directory. It should pass all tests.

Yes, it should, but I don't think the 'make check' in 0.9.6 tests that
large files work. It should probably have that as an optional test,
given how much trouble it has caused.

You can try './rdiff --version' which will tell you if it thinks it
supports large files, but that might not be correct.

I saw this as well. The fix is simple. In buf.c in the librsync source
move the

#include <config.h>

before

#include <sys/types.h>

If the order is reversed things get all messed up in the types definitions
and you get some 32bit ints where they do not belong.


--

2 3 5 7 B D 11 13 17 1D 1F 25 29 2B 2F 35 3B Robert Weber
3D 43 47 49 4F 53 59 61 65 67 6B 6D 71 7F 83 UnixOps/ITS UCB
89 8B 95 97 9D A3 A7 AD B3 B5 BF C1 C5 C7 D3
DF E3 E5 E9 EF F1 FB ... still searching ...

--------

Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

At first I thought it might be a gzip issue, so I tried the following:

rdiff-backup --no-compression root < at > gw::test test

But it failed with exactly the same error.
Both systems use reiserfs as the file system, rdiff-backup version
0.13.4 and python version 2.3.3. I have no idea what could be causing
this.



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

--------

BTW, just read on Slashdot that you won an Open Source Merit Award for
distcc. Congrats!


--
Ben Escoto

Nope this still didn't work. Recompiled rsync and rdiff-backup on both
machines and still got the same error.

On Wed, 2004-06-02 at 01:53, Robert < at > Colorado.EDU wrote:
I saw this as well. The fix is simple. In buf.c in the librsync source
move the

#include <config.h>

before

#include <sys/types.h>

If the order is reversed things get all messed up in the types definitions
and you get some 32bit ints where they do not belong.


--

2 3 5 7 B D 11 13 17 1D 1F 25 29 2B 2F 35 3B Robert Weber
3D 43 47 49 4F 53 59 61 65 67 6B 6D 71 7F 83 UnixOps/ITS UCB
89 8B 95 97 9D A3 A7 AD B3 B5 BF C1 C5 C7 D3
DF E3 E5 E9 EF F1 FB ... still searching ...

--------

Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

At first I thought it might be a gzip issue, so I tried the following:

rdiff-backup --no-compression root < at > gw::test test

But it failed with exactly the same error.
Both systems use reiserfs as the file system, rdiff-backup version
0.13.4 and python version 2.3.3. I have no idea what could be causing
this.



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

--------

Btw, before I meant I recompiled librsync. I can't seem to reproduce the problem with rdiff via signatures/deltas/pach. I think that might be because one of the files I was working with wasn't all that large. I need to do a proper test (and find some more hard disk space) and then I'm pretty sure I will be able to get that error with rdiff as well.

________________________________

From: Robert.Weber < at > Colorado.EDU on behalf of Robert < at > Colorado.EDU
Sent: Wed 6/2/2004 1:53 AM
To: Ben Zampatti
Cc: Rdiff-Backup Users
Subject: Re: [rdiff-backup-users] Large files



I saw this as well. The fix is simple. In buf.c in the librsync source
move the

#include <config.h>

before

#include <sys/types.h>

If the order is reversed things get all messed up in the types definitions
and you get some 32bit ints where they do not belong.


--

2 3 5 7 B D 11 13 17 1D 1F 25 29 2B 2F 35 3B Robert Weber
3D 43 47 49 4F 53 59 61 65 67 6B 6D 71 7F 83 UnixOps/ITS UCB
89 8B 95 97 9D A3 A7 AD B3 B5 BF C1 C5 C7 D3
DF E3 E5 E9 EF F1 FB ... still searching ...

--------

Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

At first I thought it might be a gzip issue, so I tried the following:

rdiff-backup --no-compression root < at > gw::test test

But it failed with exactly the same error.
Both systems use reiserfs as the file system, rdiff-backup version
0.13.4 and python version 2.3.3. I have no idea what could be causing
this.



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

--------

Do you get any compile time errors? What is your os/hardware? I got it
working on linux, freebsd and solaris with the same fix as below. I found
the error from compiler warnings about refined types(rs_something). You
should also check the configure to make sure largefiles were enabled
correctly. You can also see that in the config.h near the end. Also check
that your libz has largefiles support.

--
Robert Weber


--------

Nope this still didn't work. Recompiled rsync and rdiff-backup on both
machines and still got the same error.

On Wed, 2004-06-02 at 01:53, Robert < at > Colorado.EDU wrote:
I saw this as well. The fix is simple. In buf.c in the librsync source
move the

#include <config.h>

before

#include <sys/types.h>

If the order is reversed things get all messed up in the types definitions
and you get some 32bit ints where they do not belong.


--

2 3 5 7 B D 11 13 17 1D 1F 25 29 2B 2F 35 3B Robert Weber
3D 43 47 49 4F 53 59 61 65 67 6B 6D 71 7F 83 UnixOps/ITS UCB
89 8B 95 97 9D A3 A7 AD B3 B5 BF C1 C5 C7 D3
DF E3 E5 E9 EF F1 FB ... still searching ...

--------

Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

At first I thought it might be a gzip issue, so I tried the following:

rdiff-backup --no-compression root < at > gw::test test

But it failed with exactly the same error.
Both systems use reiserfs as the file system, rdiff-backup version
0.13.4 and python version 2.3.3. I have no idea what could be causing
this.



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackup
Wiki

--------



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

--------

----------
Status: by weberr Tue Jun 1 19:49:25 2004
----------

On 1 Jun 2004, Ben Escoto <bescoto < at > stanford.edu> wrote:
BTW, just read on Slashdot that you won an Open Source Merit Award for
distcc. Congrats!

Thanks Ben.


--
Martin

I think this email you sent was at the same time I sent my last email.
Anyway you are right, I fixed the problem. It must of been compiled
against the old rsync library some how. Anyway I completely deleted
librsync and re-compiled then re-compiled rdiff-backup and it works
fine. It even backed up a 7gb file from a windows PC running
rdiff-backup on cygwin, so thankyou for this fix.

On Wed, 2004-06-02 at 11:59, Robert Weber wrote:
PLEASE read the archives. You are on the same road I was on in November of
last year. You can see that you have already taken many of the steps I
did, and possibly there were some intermediate solutions that I took that
were part of the solution.

--
Robert Weber


--------

Btw, before I meant I recompiled librsync. I can't seem to reproduce the prob
lem with rdiff via signatures/deltas/pach. I think that might be because one of
the files I was working with wasn't all that large. I need to do a proper test
(and find some more hard disk space) and then I'm pretty sure I will be able t
o get that error with rdiff as well.

________________________________

From: Robert.Weber < at > Colorado.EDU on behalf of Robert < at > Colorado.EDU
Sent: Wed 6/2/2004 1:53 AM
To: Ben Zampatti
Cc: Rdiff-Backup Users
Subject: Re: [rdiff-backup-users] Large files



I saw this as well. The fix is simple. In buf.c in the librsync source
move the

#include <config.h>

before

#include <sys/types.h>

If the order is reversed things get all messed up in the types definitions
and you get some 32bit ints where they do not belong.


--

2 3 5 7 B D 11 13 17 1D 1F 25 29 2B 2F 35 3B Robert Weber
3D 43 47 49 4F 53 59 61 65 67 6B 6D 71 7F 83 UnixOps/ITS UCB
89 8B 95 97 9D A3 A7 AD B3 B5 BF C1 C5 C7 D3
DF E3 E5 E9 EF F1 FB ... still searching ...

--------

Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

At first I thought it might be a gzip issue, so I tried the following:

rdiff-backup --no-compression root < at > gw::test test

But it failed with exactly the same error.
Both systems use reiserfs as the file system, rdiff-backup version
0.13.4 and python version 2.3.3. I have no idea what could be causing
this.



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWi
ki

--------





--------

----------
Status: by weberr Tue Jun 1 19:57:43 2004
----------

I now can backup files larger than 2gb but not 7gb for some reason. I
tested 3, 4, 5 and 6gb files and they were ok but as soon as it gets to
7gb (or over 6gb) I get an error:
UpdateError bigfile Updated mirror temp file test2/rdiff-backup.tmp.1
does not match source

I know what this error means; However I am 100% sure that the source
file does not change as it's just a test file that I created. I know
it's not a gzip issue as I get the same error when doing it without
compression.

On Wed, 2004-06-02 at 11:53, Robert Weber wrote:
Do you get any compile time errors? What is your os/hardware? I got it
working on linux, freebsd and solaris with the same fix as below. I found
the error from compiler warnings about refined types(rs_something). You
should also check the configure to make sure largefiles were enabled
correctly. You can also see that in the config.h near the end. Also check
that your libz has largefiles support.

--
Robert Weber


--------

Nope this still didn't work. Recompiled rsync and rdiff-backup on both
machines and still got the same error.

On Wed, 2004-06-02 at 01:53, Robert < at > Colorado.EDU wrote:
I saw this as well. The fix is simple. In buf.c in the librsync source
move the

#include <config.h>

before

#include <sys/types.h>

If the order is reversed things get all messed up in the types definitions
and you get some 32bit ints where they do not belong.


--

2 3 5 7 B D 11 13 17 1D 1F 25 29 2B 2F 35 3B Robert Weber
3D 43 47 49 4F 53 59 61 65 67 6B 6D 71 7F 83 UnixOps/ITS UCB
89 8B 95 97 9D A3 A7 AD B3 B5 BF C1 C5 C7 D3
DF E3 E5 E9 EF F1 FB ... still searching ...

--------

Hello, I'm having trouble backing up files > 2gb. I can back up the
large file for the first time, but any subsequent attempts result in an
error. This is the error I get:

python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
UpdateError bigfile librsync error 100 while in patch cycle

At first I thought it might be a gzip issue, so I tried the following:

rdiff-backup --no-compression root < at > gw::test test

But it failed with exactly the same error.
Both systems use reiserfs as the file system, rdiff-backup version
0.13.4 and python version 2.3.3. I have no idea what could be causing
this.



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackup
Wiki

--------



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

--------

----------
Status: by weberr Tue Jun 1 19:49:25 2004
----------