Backup Central

Hi,

I seem to have found a rather serious bug somewhere in ridff-backup.
I have been using it for almost three months now to do twice daily
(unattended) incremental backups from one server onto a backup server
(using the command "rdiff-backup -v5 --exclude /proc [remote
server]::/ /backup/[remote server name]"). This was working fine up
until a few days ago when I logged in to find /usr/bin, /usr/share and
/usr/lib to be missing (deleted) from the system. At first I did not
suspect rdiff-backup, however reinstalling it and listing the
increments indicated a failed backup, so I ran "rdiff-backup -v5
--check-destination-dir /backup/[remote server name]". This completed
successfully, however after this /usr/bin and /usr/lib (which had some
reinstalled programs in them) were again missing. Doing this several
time had the same result.

As the system was such a mess I decided to reinstall everything.
Previously we had been using Debian-3.0, however since then our client
has had several new servers put in, all running Gentoo, thus I
installed Gentoo. After this had been completed I again had to check
the destination directory again as a remote backup failed. Again
/usr/bin, /usr/lib and /usr/share had been deleted, this time using a
completely different distribution and in a completely different
environment (ie. no longer on the same network). This was all done by
manually executing rdiff-backup (ie. not using automated scripts).
Any help would be much appreciated as this seems to be a rather
serious problem.

Regards,
Alistair

Alistair Popple wrote:

environment (ie. no longer on the same network). This was all done by
manually executing rdiff-backup (ie. not using automated scripts).
Any help would be much appreciated as this seems to be a rather
serious problem.

does -v6 or -v7 tell you its deleting the files - it should describe
every operation. You can check in the backup.log in rdiff-backup-data

dave

Dave,

There is no reference to any files being deleted from /usr/* in the
backup.log file (ie. running grep "Deleting" backup.log | grep
"/usr" does not produce anything). Also I forgot to mention I'm using
version 0.13.4. Also when I ran rdiff-backup --check-destination-dir
there where quite a few errors about files being deleted from the
rdiff-backup directory, which I did not delete - here is the error:

A regular file was indicated by the metadata, but could not be
constructed from existing increments because last increment had type
None. Instead of the actual file's data, an empty length file will be
created. This error is probably caused by data loss in the
rdiff-backup destination directory, or a bug in rdiff-backup
Warning: Could not restore file [filename removed]!

Alistair

On Sat, 18 Sep 2004 17:34:09 +1000, David Kempe
<dave < at > solutionsfirst.com.au> wrote:
Alistair Popple wrote:

environment (ie. no longer on the same network). This was all done by
manually executing rdiff-backup (ie. not using automated scripts).
Any help would be much appreciated as this seems to be a rather
serious problem.

does -v6 or -v7 tell you its deleting the files - it should describe
every operation. You can check in the backup.log in rdiff-backup-data

dave

begin quotation of Alistair Popple on 2004-09-18 17:30:45 +1000:

As the system was such a mess I decided to reinstall everything.
Previously we had been using Debian-3.0,

This problem first occured with woody's 0.6.0-1 or with
testing/unstable's 0.13.4-3?

however since then our client has had several new servers put in,
all running Gentoo, thus I installed Gentoo. After this had been
completed I again had to check the destination directory again as a
remote backup failed. Again /usr/bin, /usr/lib and /usr/share had
been deleted, this time using a completely different distribution
and in a completely different environment (ie. no longer on the same
network).

After you installed Gentoo, did you continue using the same backup
set? That is, did you continue to send rdiffs to the backup host, or
did you start fresh with a new set of backups?

This problem first occured with woody's 0.6.0-1 or with
testing/unstable's 0.13.4-3?

The problem first occured using testing/unstable's 0.13.4-3. Version
0.6.0-1 was never used as it seemed too out of date and was not
available for some of our other servers.

After you installed Gentoo, did you continue using the same backup
set? That is, did you continue to send rdiffs to the backup host, or
did you start fresh with a new set of backups?

This was on the same backup set, as I didn't want to loose older
revisions of files, again using version 0.13.4. I'm pretty sure the
problem is related to the data-set, as I have been using rdiff-backup
for months and have not encountered any problems similar to this.

begin quotation of Alistair Popple on 2004-09-19 14:55:49 +1000:

After you installed Gentoo, did you continue using the same backup
set? That is, did you continue to send rdiffs to the backup host,
or did you start fresh with a new set of backups?

This was on the same backup set, as I didn't want to loose older
revisions of files, again using version 0.13.4. I'm pretty sure the
problem is related to the data-set, as I have been using
rdiff-backup for months and have not encountered any problems
similar to this.

If you're interested in pursuing this, would you track down the
increment that started deleting stuff? You should find it in the
rdiff-backup-data/increments/{dir}. Perhaps that will shed some light
on this.

I think I have located the reason files in /usr were being deleted.
Within the backup directory there is a symlink to /usr (called
./export/journyx/journyx/pd/Linux/pgres), which must somehow be
related to the deleting as the problem first occured after I added
this link on the main server. I found this out by running the
destination-check on the rebuilt system, which for obvious reasons was
setup so that /usr and / were mounted read-only. The destination
check then failed with the following error:

OSError: [Errno 30] Read-only file system:
'./export/journyx/journyx/pd/Linux/pgres/bin/.keep'

However the destination directory was mounted read-write and the file
mentioned does not exist on the server being backed up. Also the
destination-check succeded once this link was removed from the backup
set (both the destination directory and the rdiff-backup-data
directory)

On Sun, 19 Sep 2004 22:48:00 -0400, Alec Berryman <alec < at > thened.net> wrote:
begin quotation of Alistair Popple on 2004-09-19 14:55:49 +1000:

After you installed Gentoo, did you continue using the same backup
set? That is, did you continue to send rdiffs to the backup host,
or did you start fresh with a new set of backups?

This was on the same backup set, as I didn't want to loose older
revisions of files, again using version 0.13.4. I'm pretty sure the
problem is related to the data-set, as I have been using
rdiff-backup for months and have not encountered any problems
similar to this.

If you're interested in pursuing this, would you track down the
increment that started deleting stuff? You should find it in the
rdiff-backup-data/increments/{dir}. Perhaps that will shed some light
on this.



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

begin quotation of Alistair Popple on 2004-09-22 21:10:17 +1000:

I think I have located the reason files in /usr were being deleted.
Within the backup directory there is a symlink to /usr (called
./export/journyx/journyx/pd/Linux/pgres), which must somehow be
related to the deleting as the problem first occured after I added
this link on the main server. I found this out by running the
destination-check on the rebuilt system, which for obvious reasons was
setup so that /usr and / were mounted read-only. The destination
check then failed with the following error:

OSError: [Errno 30] Read-only file system:
'./export/journyx/journyx/pd/Linux/pgres/bin/.keep'

However the destination directory was mounted read-write and the file
mentioned does not exist on the server being backed up. Also the
destination-check succeded once this link was removed from the backup
set (both the destination directory and the rdiff-backup-data
directory)

Hmm... so do you think when the symlink to /usr was deleted, /usr was
deleted along with it?

Hmm... so do you think when the symlink to /usr was deleted, /usr was
deleted along with it?

This is probably best explained with an example I used to replicate
the problem on a test system at home. Hopefully this is
understandable (I can clarify anything if it isn't):

I created the following directory structure in /tmp/rdiff-backup-test/src:

.
|-- test1
| |-- test1.1
| | |-- a
| | |-- b
| | `-- c
| `-- test1.2
| |-- d
| |-- e
| `-- f
`-- test2
|-- test2.1
| `-- test1.1
| |-- g
| |-- h
| `-- i
`-- test2.2
|-- j
|-- k
`-- l

The following steps were then performed:
* rdiff-backup -v7 alistair < at > 127.0.0.1::/tmp/rdiff-backup-test/src/
/tmp/rdiff-backup-test/dest/
* mv /tmp/rdiff-backup-test/src/test2/test2.1
/tmp/rdiff-backup-test/src/test2/test2.1.old
* ln -s /tmp/rdiff-backup-test/src/test1
/tmp/rdiff-backup-test/src/test2/test2.1
* rdiff-backup -v7 alistair < at > 127.0.0.1::/tmp/rdiff-backup-test/src/
/tmp/rdiff-backup-test/dest/

At this point everything should be fine and working as expected. The
following directory structure exists in the desination directory:

|-- rdiff-backup-data
| |-- ......................
|-- test1
| |-- test1.1
| | |-- a
| | |-- b
| | `-- c
| `-- test1.2
| |-- d
| |-- e
| `-- f
`-- test2
|-- test2.1 -> /tmp/rdiff-backup-test/src/test1/
|-- test2.1.old
| `-- test1.1
| |-- g
| |-- h
| `-- i
`-- test2.2
|-- j
|-- k
`-- l

Note the symlink from test2.1 to a file outside the destination path.

However if the backup fails (eg. it is interrupted halfway through)
then the desintation directory needs checking:
* rdiff-backup -v7 --check-desintation /tmp/rdiff-backup-test/dest

It is at this point the problems begin. If you check
/tmp/rdiff-backup-test/src/test1 (a directory outside of the
destination) you will find that the test1.1 directory has been
deleted. This only happens for the directories mentioned in the
original (non-linked) version of test2.1, which is why test1.2
remains. The problem here seems to be that the check-destination-dir
option follows the symlinks rather than treating them as files. If
the link is changed to a relative one then sure enough no directories
outside of the destination seem to be modified, however after the
destination check test1.1 is missing from the destination/backup - ie.
it has _not_ been backed up. Performing subsequent backups will also
not back that information up, even if the relative symlink is removed,
thus the backups remain incomplete for what I assume is ever more.