Backup Central

How do you think rdiff-backup should be improved? Help me prioritize
rdiff-backup development by taking the rdiff-backup poll! See:

http://www.misterpoll.com/3662297208.html

For current feature suggestions see:

http://savannah.nongnu.org/support/?group=rdiff-backup
http://rdiff-backup.solutionsfirst.com.au/?SuggestedFeatures

And of course more in-depth responses to the mailing list are very
welcome.

Oh, and please answer relative to the development branch (1.1.x). If
you are using the stable branch, the development versions basically
adds longname support, some Mac-specific fixes, SHA1 hashs, some more
compare options, and cleaner error messages. (It's also slower.)


--
Ben Escoto

Also feel free to post if you think this poll has the wrong options
listed. Or if you feel the whole poll idea is stupid and have a
better way to get user feedback...


--
Ben Escoto

Ben Escoto wrote:
How do you think rdiff-backup should be improved? Help me prioritize
rdiff-backup development by taking the rdiff-backup poll! See:

http://www.misterpoll.com/3662297208.html

For current feature suggestions see:

http://savannah.nongnu.org/support/?group=rdiff-backup
http://rdiff-backup.solutionsfirst.com.au/?SuggestedFeatures

And of course more in-depth responses to the mailing list are very
welcome.

Oh, and please answer relative to the development branch (1.1.x). If
you are using the stable branch, the development versions basically
adds longname support, some Mac-specific fixes, SHA1 hashs, some more
compare options, and cleaner error messages. (It's also slower.)

Thanks! I just voted.

The one feature I think would be handy would be reporting of what was added or
deleted that caused a huge backup. My normal nightly backups are around 20
Mbytes, but last night I got one that was 220 Mbytes and I'd like to figure out
what changed that caused this. Some report by directory, say like a du output,
of changed file size would be nice. Directories with no changed wouldn't be
printed.

Out of this list

http://rdiff-backup.solutionsfirst.com.au/?SuggestedFeatures

I would vote for the following features in descending order of importance:

* RemoveSpecifiedFiles - --remove Removes specified files from rdiff-backup archive
* DeleteIntermediate - Remove intermediate backups, retaining older ones
* RemoveOlderThanAllowsSubdirectories - argument to --remove-older-than can
specify a subdirectory
* UpdateErrorRetry - Retry to copy files with update error later
* BackupFromRsyncd - Allow rdiff-backup to get snapshot from an rsyncd daemon
* SparseFiles - handle sparse files well

Nice to have, but not necessary:
* DetectMoves - Detect file moves
* Psyco - use the Psyco JIT compiler to speed things up
I run these jobs at night.

In the "How buggy do you find rdiff-backup?", you might want to add a level
between "Very buggy" and "Found a few, but nothing serious". I'd hate to call
rdiff-backup very buggy, otherwise I wouldn't use it and I love this program.
On the other hand, I did run into that KeyError bug in 1.1.3 which prevented a
backup from working and the Mac OS X issues, so it hasn't been un-serious. I
didn't vote for that section.

Regards,
Blair

--
Blair Zajac, Ph.D.
<blair < at > orcaware.com>
Subversion and Orca training and consulting
http://www.orcaware.com/svn/

On Mon, 19 Dec 2005, Blair Zajac wrote:

The one feature I think would be handy would be reporting of what was added or
deleted that caused a huge backup. My normal nightly backups are around 20
Mbytes, but last night I got one that was 220 Mbytes and I'd like to figure
out what changed that caused this. Some report by directory, say like a du
output, of changed file size would be nice. Directories with no changed
wouldn't be printed.

try <http://arctic.org/~dean/rdiff-backup/backup-statistics> ... give it
the path to your mirror.

-dean

Blair Zajac <blair < at > orcaware.com>
wrote the following on Mon, 19 Dec 2005 22:37:32 -0800

The one feature I think would be handy would be reporting of what
was added or deleted that caused a huge backup. My normal nightly
backups are around 20 Mbytes, but last night I got one that was 220
Mbytes and I'd like to figure out what changed that caused this.
Some report by directory, say like a du output, of changed file size
would be nice. Directories with no changed wouldn't be printed.

Yes, in theory there is the file_statistics files, which have detailed
by-directory statistics, but this file requires some computer
processing (like Dean's script) to be easily human readable. You can
also use 'du' itself, along with find and regular expressions and
such, but yeah I can see this isn't optimally easy.

In the "How buggy do you find rdiff-backup?", you might want to add
a level between "Very buggy" and "Found a few, but nothing serious".
I'd hate to call rdiff-backup very buggy, otherwise I wouldn't use
it and I love this program. On the other hand, I did run into that
KeyError bug in 1.1.3 which prevented a backup from working and the
Mac OS X issues, so it hasn't been un-serious. I didn't vote for
that section.

Well, that poll has accomplished its purpose, in that I know that the
majority of people taking the poll run into bugs in rdiff-backup. I
think part of this is I need to take the development versions a bit
more seriously. I've been a bit hypocritical in that I sometimes
advertise development features to, for instance, Mac OS X people, but
then I also sometimes have the "release early and often, the dev
version may eat your files" mentality. So one thing I'll start doing
is using exclusively the development version personally, instead of
being lazy and using whatever version of rdiff-backup Fedora
packages.

(This is all assuming the bugs people find are mostly in the devel
version. If not then I needed a new poll option like you said.)


--
Ben Escoto

dean gaudet <dean-list-rdiff-backup < at > arctic.org>
wrote the following on Tue, 20 Dec 2005 00:08:26 -0800 (PST)

try <http://arctic.org/~dean/rdiff-backup/backup-statistics>
... give it the path to your mirror.

What do you think about adding this to the rdiff-backup distribution?
It might be nice to rewrite it in python, and maybe add some bloat.

For instance I'd like to see a list of any files/directories that
contribute to more than 5% of total size (there could be at most 20 of
these). Any subdirectories mentioned could be subtracted from the
total. Like if /var contributes 17% total, and /var/spool contributes
15%, only /var/spool would be listed.


--
Ben Escoto

Ben Escoto <ben < at > emerose.org>
wrote the following on Tue, 20 Dec 2005 13:54:59 -0600

What do you think about adding this to the rdiff-backup distribution?
It might be nice to rewrite it in python, and maybe add some bloat.

For instance I'd like to see a list of any files/directories that
contribute to more than 5% of total size (there could be at most 20 of
these). Any subdirectories mentioned could be subtracted from the
total. Like if /var contributes 17% total, and /var/spool contributes
15%, only /var/spool would be listed.

And we could do the same thing with time if a time field were added to
the file_statistics file.


--
Ben Escoto

Ben Escoto wrote:
dean gaudet <dean-list-rdiff-backup < at > arctic.org>
wrote the following on Tue, 20 Dec 2005 00:08:26 -0800 (PST)

try <http://arctic.org/~dean/rdiff-backup/backup-statistics>
... give it the path to your mirror.


What do you think about adding this to the rdiff-backup distribution?
It might be nice to rewrite it in python, and maybe add some bloat.

For instance I'd like to see a list of any files/directories that
contribute to more than 5% of total size (there could be at most 20 of
these). Any subdirectories mentioned could be subtracted from the
total. Like if /var contributes 17% total, and /var/spool contributes
15%, only /var/spool would be listed.

Yes, I would like to see that also.

Some way of specifying the depth of reporting and which directories to report on
would be useful. Something like --max-depth N would aggregate everything to
that level, not reporting anything deeper. This would be used to find the first
place to look for. Then when one finds the offending directory, one can ask for
a report on that, increasing --max-depth.

One note regarding the Perl script. I had to add a line

$\ = "\0";

since I always pass the --null-separator to all my rdiff-backups, so the normal
EOL matching wasn't working.

Regards,
Blair

--
Blair Zajac, Ph.D.
<blair < at > orcaware.com>
Subversion and Orca training and consulting
http://www.orcaware.com/svn/

dean gaudet wrote:
On Mon, 19 Dec 2005, Blair Zajac wrote:


The one feature I think would be handy would be reporting of what was added or
deleted that caused a huge backup. My normal nightly backups are around 20
Mbytes, but last night I got one that was 220 Mbytes and I'd like to figure
out what changed that caused this. Some report by directory, say like a du
output, of changed file size would be nice. Directories with no changed
wouldn't be printed.


try <http://arctic.org/~dean/rdiff-backup/backup-statistics> ... give it
the path to your mirror.

Dean,

Thanks for the script. It's very useful.

Regards,
Blair

Hi Ben and all,

How do you think rdiff-backup should be improved?

I voted in the poll, thanks. However, a few things have come to mind
since:

I normally back up to an unprivileged account on the remote server, and
I'd like a way to suppress these warning messages:

Warning: ownership cannot be changed on filesystem at
/mnt/backup/.../rdiff-backup-data
Unable to import module xattr.
Extended attributes not supported on filesystem at
/mnt/backup/.../rdiff-backup-data/rdiff-backup.tmp.0
Unable to import module posix1e from pylibacl package.
ACLs not supported on filesystem at
/mnt/backup/.../rdiff-backup-data/rdiff-backup.tmp.0

Like others on the list, I would like a way to remove old versions of
certain (large) files from the increments, and to remove (merge)
older increments.

I'm also considering developing a GUI tool for running and managing
rdiff-backup, and I'd be interested to know what others think. I'm already
developing a similar tool for Box Backup, using C++. Ideally I would like
this tool, Boxi [http://boxi.sourceforge.net], to support all the
following backup systems:

* Box Backup
* Rdiff-backup
* Duplicity
* Bacula
* any others that the community are interested in

The main issue with supporting Rdiff-backup is the interface to Python
(from C++). I was wondering if I could/should embed a Python interpreter
in my code and use that to call rdiff-backup's functions? I would probably
also need to implement callbacks to report backup progress and errors in
an appropriate manner for a GUI.

I'd like to know whether anyone else is interested in such support,
whether there are any objections or suggestions from rdiff-backup users,
and whether there is any likelihood that the changes I will need to make
to the core rdiff-backup code might be merged back into the core
distribution.

I will also need to learn Python, and would appreciate pointers to good
resources for this.

Has anyone tested rdiff-backup on Windows platforms, with or without
Cygwin?

By the way, in case you didn't know, Sourceforget uses rdiff-backup for
their backup strategy. Well done!

Deployed an rdiff-backup based backup solution to strengthen our our
tape backup coverage; centralized filesystem-based backups archived to
tape. Launched 2005-09-21.

[https://sourceforge.net/docman/display_doc.php?docid=19242&group_id=1]

Cheers, Chris.
--
_ ___ __ _
/ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |

On Tue, 20 Dec 2005, Ben Escoto wrote:

dean gaudet <dean-list-rdiff-backup < at > arctic.org>
wrote the following on Tue, 20 Dec 2005 00:08:26 -0800 (PST)

try <http://arctic.org/~dean/rdiff-backup/backup-statistics>
... give it the path to your mirror.

What do you think about adding this to the rdiff-backup distribution?
It might be nice to rewrite it in python, and maybe add some bloat.

including it is cool with me... but my python skills are still in the
"look at other code and mimic it" stage of development :)

at the moment i've got a bunch of other coding on my plate though... so if
you or someone else wants to run with it, i'd be happy.


For instance I'd like to see a list of any files/directories that
contribute to more than 5% of total size (there could be at most 20 of
these). Any subdirectories mentioned could be subtracted from the
total. Like if /var contributes 17% total, and /var/spool contributes
15%, only /var/spool would be listed.

yeah that'd probably be a good change... i tend to just eyeball the daily
results i get from my ~100 user server and usually the peak "new" or peak
"inc" lines just pop out and i find someone's new mp3 collection. but
cutting back on the extra fluff in the output would be helpful.

-dean

On Tue, 20 Dec 2005, Ben Escoto wrote:

Ben Escoto <ben < at > emerose.org>
wrote the following on Tue, 20 Dec 2005 13:54:59 -0600

What do you think about adding this to the rdiff-backup distribution?
It might be nice to rewrite it in python, and maybe add some bloat.

For instance I'd like to see a list of any files/directories that
contribute to more than 5% of total size (there could be at most 20 of
these). Any subdirectories mentioned could be subtracted from the
total. Like if /var contributes 17% total, and /var/spool contributes
15%, only /var/spool would be listed.

And we could do the same thing with time if a time field were added to
the file_statistics file.

oh yeah time would be interesting...

and my script doesn't consider inode count under a tree -- time is going
to be proportional to inode count most likely ... that'd probably be
useful to display.

i have to exclude a ~2M inode subtree on my server because its inode churn
is way too high for my dsl + home /backup disk to accomodate. (it's the
http://electricsheep.org/ screensaver central server... the co-ordination
of the animations is unfortunately inode intensive.)

-dean

On Tue, 20 Dec 2005, Chris Wilson wrote:

The main issue with supporting Rdiff-backup is the interface to Python (from
C++). I was wondering if I could/should embed a Python interpreter in my code
and use that to call rdiff-backup's functions? I would probably also need to
implement callbacks to report backup progress and errors in an appropriate
manner for a GUI.

why wouldn't you just spawn the rdiff-backup program? otherwise you'd be
dependent on the internal apis remaining stable... which doesn't seem like
a good choice for your gui or for rdiff-backup development...

progress meters kind of depend on knowing how much there is to be done,
and rdiff-backup doesn't know that initially at least... because it starts
transferring data even before it has scanned all the inodes needing
backup. but rdiff-backup could probably be easily mod'd to have an option
to display how many inodes/bytes it had processed so far... just hook in
where the file-statistics file is written...

-dean

Hi Dean,

why wouldn't you just spawn the rdiff-backup program? otherwise you'd
be dependent on the internal apis remaining stable... which doesn't seem
like a good choice for your gui or for rdiff-backup development...

Well, I'm assuming that sooner or later rdiff-backup will have some kind
of stable API that I can use. Also, the compiler (and python bytecode
interpreter) will check that I'm using the correct API and give me a
warning if not. Trying to parse the output of a process is a nightmare
that I really don't want to get involved with (again).

progress meters kind of depend on knowing how much there is to be done,
and rdiff-backup doesn't know that initially at least... because it starts
transferring data even before it has scanned all the inodes needing
backup.

Well, I guess I would have to change that :-)

but rdiff-backup could probably be easily mod'd to have an option to
display how many inodes/bytes it had processed so far... just hook in
where the file-statistics file is written...

Yeah.

Cheers, Chris.
--
_ ___ __ _
/ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |

dean gaudet <dean-list-rdiff-backup < at > arctic.org>
wrote the following on Tue, 20 Dec 2005 14:06:16 -0800 (PST)

i have to exclude a ~2M inode subtree on my server because its inode
churn is way too high for my dsl + home /backup disk to accomodate.
(it's the http://electricsheep.org/ screensaver central
server... the co-ordination of the animations is unfortunately inode
intensive.)

Wow, it needs 2 million files? That seems incredibly excessive for a
screensaver.


--
Ben Escoto