Backup Central

I have been a very happy user of rdiff-backup for about a year now.
Recently though I noticed that certain directories are not in my backup.
I did a 'diff -rq' on some of the source/backup directories and
(although I haven't checked extensively) all the missing ones seem to be
empty directory trees.

I am currently running rdiff-backup 1.0.1-r1 on a Gentoo Linux box
(2.6.12-gentoo-r10 on AMD64).

The source data itself was restored after a disk crash, some time
towards the end of August, so I can only assume this problem emerged
some time after that (since I know that some of the empty directory
trees are more than a year old and otherwise I don't see how they would
have been restored in the first place).

I tried running a small backup of just a single directory (one that
contained some of the missing directories) and the odd thing is that
worked as expected, i.e. it backed up everything including the empty
sub-directory trees. Could this have something to do with volumes of data?

Looking at my Gentoo package installation log I see the history is as
follows:
rdiff-backup-0.13.6 -> installed 8 September 2005
rdiff-backup-1.0.1-r1 -> installed 22 September 2005

So I'm guessing this is a problem that came with upgrading to 1.0.1.
Perhaps it would be better for me to back out to 0.13.6? (Unfortunately
the most recent versions are not yet available on 64-bit Gentoo.) I
tried searching Google & the rdiff-backup-user archives but didn't come
up with anything. If it would help, I can post my rdiff-backup command
or any other relevant information.

Thanks
Toni

Toni Price <tonip < at > mail.com>
wrote the following on Sun, 01 Jan 2006 09:02:02 +0000
I have been a very happy user of rdiff-backup for about a year now.
Recently though I noticed that certain directories are not in my backup.
I did a 'diff -rq' on some of the source/backup directories and
(although I haven't checked extensively) all the missing ones seem to be
empty directory trees.

All I can say is I don't see why rdiff-backup wouldn't back up empty
directories. I've never personally noticed this problem. I'm not
sure what to do unless you can isolate or reproduce the problem.

Are you sure its a bug? Maybe it's just some unexpected
--include/--exclude interaction, or a permission problem?

So I'm guessing this is a problem that came with upgrading to
1.0.1. Perhaps it would be better for me to back out to 0.13.6?
(Unfortunately the most recent versions are not yet available on
64-bit Gentoo.) I tried searching Google & the rdiff-backup-user
archives but didn't come up with anything. If it would help, I can
post my rdiff-backup command or any other relevant information.

0.13.6 was the last development release before the stable 1.0.x
series, so 1.0.1 is just 0.13.6 with bugfixes. So it probably won't
help if you downgrade.


--
Ben Escoto

Ben Escoto wrote:

Toni Price <tonip < at > mail.com>
wrote the following on Sun, 01 Jan 2006 09:02:02 +0000


I have been a very happy user of rdiff-backup for about a year now.
Recently though I noticed that certain directories are not in my backup.
I did a 'diff -rq' on some of the source/backup directories and
(although I haven't checked extensively) all the missing ones seem to be
empty directory trees.



All I can say is I don't see why rdiff-backup wouldn't back up empty
directories. I've never personally noticed this problem. I'm not
sure what to do unless you can isolate or reproduce the problem.

Are you sure its a bug? Maybe it's just some unexpected
--include/--exclude interaction, or a permission problem?



No - I'm not at all sure it's a bug! Though it seems very unlikely to me
that it could be a permissions problem. The permissions on some of the
missing directories look like this:

(...)
drwx------ 2 tprice users 4096 Jan 9 2005 <...>
drwxr-xr-x 3 tprice users 4096 Jan 9 2005 <...>
drwxrwxrwx 2 tprice users 4096 May 28 2004 <...>
drwx------ 5 tprice users 4096 Jan 27 2005 <...>
drwxr-xr-x 2 tprice users 4096 Jan 30 2005 <...>
drwxr-xr-x 2 tprice users 4096 Jan 30 2005 <...>
(...)

As far as includes and excludes are concerned, I wouldn't have guessed
they were causing this but I'll try and do some testing to see if I can
pinpoint anything.



So I'm guessing this is a problem that came with upgrading to
1.0.1. Perhaps it would be better for me to back out to 0.13.6?
(Unfortunately the most recent versions are not yet available on
64-bit Gentoo.) I tried searching Google & the rdiff-backup-user
archives but didn't come up with anything. If it would help, I can
post my rdiff-backup command or any other relevant information.



0.13.6 was the last development release before the stable 1.0.x
series, so 1.0.1 is just 0.13.6 with bugfixes. So it probably won't
help if you downgrade.

Ben Escoto wrote:



Are you sure its a bug? Maybe it's just some unexpected
--include/--exclude interaction, or a permission problem?



Ok, I have managed to repeat the problem on some dummy data I've created.

I've attached a copy of the data with a bash script to illustrate what's
happening ... this data (sort of) mimics my own data structure, which is
why I've done it like this. (If you untar it and run
test-rdiff-backup.sh from the directory where it's been untarred, it
should copy some data to /tmp/rdiff-data and run two backups to
/tmp/rdiff-test/bak).

It's entirely possible that I simply don't understand properly how the
--includes and --excludes work, but this behaviour seems odd to me.

My data structure looks as follows:

rdiff-data/
|-- docs
| |-- technotes
| | |-- anotherdir
| | | |-- empty_tree_2
| | | | |-- dir1
| | | | | `-- structure
| | | | | |-- sub1
| | | | | `-- sub2
| | | | | `-- subsub
| | | | `-- dir2
| | | | |-- lower1
| | | | `-- lower2
| | | `-- somefile.txt
| | |-- empty_tree_1
| | | `-- sub1
| | | |-- sibling1
| | | | `-- nextdir
| | | `-- sibling2
| | `-- misc
| | |-- dir1
| | | `-- fox-and-dogs.txt
| | `-- dir2
| | `-- sub1
| | `-- stuff.txt
| |-- tmp
| | |-- tempdir1
| | |-- tempfile1.txt
| | `-- tempfile2.txt
| `-- various
| |-- another
| | |-- empty_tree_3
| | | `-- testdir1
| | | `-- testdir2
| | | |-- testsub1
| | | `-- testsub2
| | `-- moreStuff.txt
| |-- file1.txt
| `-- file2.txt
|-- mediafiles
| `-- audio
| |-- Classical
| | |-- audio1
| | `-- audio2
| |-- Jazz
| | `-- audio
| `-- Playlists
| |-- playlist1.txt
| `-- playlist2.txt
|-- ref
| `-- ref.txt
`-- software
`-- sofware-stuff.txt

37 directories, 15 files

Within this there are three empty directory trees,
rdiff-data/docs/technotes/empty_tree_1/
rdiff-data/docs/technotes/anotherdir/empty_tree_2/
rdiff-data/docs/various/another/empty_tree_3

Basically, if I run the following command:

/usr/bin/rdiff-backup \
--exclude /tmp/rdiff-data/docs/tmp/'*' \
--include /tmp/rdiff-data/mediafiles/audio/Playlists \
--exclude /tmp/rdiff-data/mediafiles/audio/'*' \
--exclude /tmp/rdiff-data/ref/'*' \
--exclude /tmp/rdiff-data/software/'*' \
/tmp/rdiff-data /tmp/rdiff-test/bak/data1

it works as expected, whilst if I change it as follows (the change I've
made here is from /tmp/rdiff-data to '**' in the --include)

/usr/bin/rdiff-backup \
--exclude /tmp/rdiff-data/docs/tmp/'*' \
--include '**'/mediafiles/audio/Playlists \
--exclude /tmp/rdiff-data/mediafiles/audio/'*' \
--exclude /tmp/rdiff-data/ref/'*' \
--exclude /tmp/rdiff-data/software/'*' \
/tmp/rdiff-data /tmp/rdiff-test/bak/data2

then it all goes a bit weird, and excludes empty_tree_1, empty_tree_2 &
empty_tree_3 from the backup (as well as excluding tmp, ref and software
altogether, which is not my intention).

Here's the output I get when running test-rdiff-backup.sh. (Hopefully it
will produce the same results when run in a different environment):

===== START output =================================================

Files only in /tmp/rdiff-data after [correct] backup:

Only in /tmp/rdiff-data/docs/tmp: tempdir1
Only in /tmp/rdiff-data/docs/tmp: tempfile1.txt
Only in /tmp/rdiff-data/docs/tmp: tempfile2.txt
Only in /tmp/rdiff-data/mediafiles/audio: Classical
Only in /tmp/rdiff-data/mediafiles/audio: Jazz
Only in /tmp/rdiff-data/ref: ref.txt
Only in /tmp/rdiff-data/software: sofware-stuff.txt


Files only in /tmp/rdiff-test/bak/data1 after [correct] backup:

Only in /tmp/rdiff-test/bak/data1: rdiff-backup-data

------------------------------------------------

Files only in /tmp/rdiff-data after [incorrect] backup:

Only in /tmp/rdiff-data/docs/technotes/anotherdir: empty_tree_2
Only in /tmp/rdiff-data/docs/technotes: empty_tree_1
Only in /tmp/rdiff-data/docs: tmp
Only in /tmp/rdiff-data/docs/various/another: empty_tree_3
Only in /tmp/rdiff-data/mediafiles/audio: Classical
Only in /tmp/rdiff-data/mediafiles/audio: Jazz
Only in /tmp/rdiff-data: ref
Only in /tmp/rdiff-data: software


Files only in /tmp/rdiff-test/bak/data2 after [incorrect] backup:

Only in /tmp/rdiff-test/bak/data2: rdiff-backup-data

===== END output ==================================================

What I don't understand is, why would the following include:
--include '**'/mediafiles/audio/Playlists
cause rdiff-backup to exclude something like
"/tmp/rdiff-data/docs/technotes/anotherdir/empty_tree_2"? Am I simply
missing something about how --include should work?

Thanks ...
Toni

Toni Price <tonip < at > mail.com>
wrote the following on Tue, 03 Jan 2006 21:00:06 +0000

Ok, I have managed to repeat the problem on some dummy data I've created.
...
It's entirely possible that I simply don't understand properly how the
--includes and --excludes work, but this behaviour seems odd to me.

You were absolutely right, under some conditions when you use an
include like "--include **XXX", empty directories may be skipped, and
this bug may have first appeared in version 1.0.0 (which fixed another
selection bug).

I think this patch will fix the problem, but you may want to test it
on your directory structure:

http://cvs.savannah.nongnu.org/viewcvs/rdiff-backup/rdiff_backup/selection.py?root=rdiff-backup&r1=1.41&r2=1.42&makepatch=1&diff_format=u


--
Ben Escoto