Hi all,
a while ago I reported seeing librsync error 107 when updating large
files. The answer was to use the command rdiff to see if this was indeed
rdiff-backup related. At the time I was mighty busy, but now I have
taken the time to look into this more thorough and found out that it is
indeed a problem in rdiff-backup. I would like to flag this as a bug.
The following was carried out with rdiff-backup 1.0.3 with Debian. I
know this is not the latest and greatest, but according to the
changelist nothing even remotely resembling this problem has been fixed.
Here goes:
To start I created some files in a temporary directory:
$ mkdir /bigdisk/tmp
$ cd /bigdisk/tmp
$ for i in 512 1024 2048 4096 5120; do echo $i; dd if=/dev/urandom
of=largefile_${i}_MB.dat bs=1024K count=$i; done
This creates files with random data sized 512Mb, 1, 2 , 4 and 5 Gb. I
proceeded by making a backup of the temporary directory:
$ cd ..
$ rdiff-backup tmp/ tmp2
This takes a while and succeeds. Everything is ok up to this point. The
mirror is created, no errors reported, the files are identical in both
directories. Now I changed the big files somewhat by adding some bytes:
$ cd tmp
$ for i in *.dat; do echo bladiebla >> $i; done
Now restart the backup process:
$ cd ..
$ rdiff-backup tmp/ tmp2
This again takes a while and then produces the following output:
python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=23, orig_out=65536, final_in=23, final_out=65536]
UpdateError largefile_4096_MB.dat librsync error 107 while in patch cycle
python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=17533, orig_out=65536, final_in=17533, final_out=65536]
UpdateError largefile_5120_MB.dat librsync error 107 while in patch cycle
sure enough, the files mentioned in the output are no longer in sync:
$ md5sum tmp/*.dat tmp2/*.dat
cb79763db0567150a3962299d568ba81 tmp/largefile_1024_MB.dat
ea52ed724707b474208ff08baabd2020 tmp/largefile_2048_MB.dat
e2948b4fdbdd140392f166718e19f1fd tmp/largefile_4096_MB.dat --> *
73aec8d4c7d0a57c3ecfe5e6be0755ae tmp/largefile_5120_MB.dat --> *
b86db4919db290010b8610b3a234035a tmp/largefile_512_MB.dat
cb79763db0567150a3962299d568ba81 tmp2/largefile_1024_MB.dat
ea52ed724707b474208ff08baabd2020 tmp2/largefile_2048_MB.dat
38b463c451c667c91430e82cfb611ed3 tmp2/largefile_4096_MB.dat -->*
4f864e56beac1ce384fe66608ee98e85 tmp2/largefile_5120_MB.dat --> *
b86db4919db290010b8610b3a234035a tmp2/largefile_512_MB.dat
I have seen this behaviour on my servers on several occasion, and it
seems consistent. Also, this has nothing to do with ssh, because we are
making a local backup. However, the behaviour over ssh is exactly the
same. Now for good measure I used rdiff to see if this would generate a
similar error:
$ rdiff signature tmp2/largefile_4096_MB.dat signature.dat
$ rdiff delta signature.dat tmp/largefile_4096_MB.dat delta.dat
$ rdiff patch tmp2/largefile_4096_MB.dat delta.dat largefile_4096_MB.dat
rdiff: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=23, orig_out=16000, final_in=23, final_out=16000]
rdiff: ERROR: library internal error
It is notable that the error is given immediately after the 3rd command.
The output file is 0 bytes long. It may be worth to note that when I
tried this for files of the same size, but consisting of all zeros, no
problem occured. Perhaps the files are compressed during the rdiff
process and this has influence by making the datastream smaller?
Hopefully you can patch this. I dont have a clue where to start!
Sincerely
Gerard van Dijnsen
