Backup Central

Hello,

I've just started using rdiff-backup. I've had a few difficulties, but
nothing major. I thought I would make a few comments about these.

I've being trying to set up remote unattended backups using Dean Gaudet's
writeup, including use of LVM snapshots. This seems to work Ok for the
most part. However, I did some stress testing by having rdiff-backup do
backups of big changes at 5 min intervals.

This predictably means that (since the previous job has not completed) at
some point two different instances of rdiff-backup will be running and
trying to write to the same directory. Perhaps problems were to be
expected and in fact rdiff-backup produced errors and appeared to lose
data.

I managed to work around this by having the script check whether the LVM
snapshot were mounted first, since this would mean the previous backups
were still in progress. However, I wonder whether it would not be possible
for rdiff-backup to detect that another rdiff-backup process was writing
to the same directory and exit gracefully? Perhaps this would be difficult
to do in a portable way.

Secondly, I noticed that the --remove-older-than option removes files in
the increments directory, but does not touch any of the log and statistics
files that are generated, eg session_statistics, error_log,
mirror_metadata etc.

Would it not be reasonable for this command to remove all these files for
the relevant dates as well? If not, what is the recommended way to remove
these files?

Thirdly, when using --remove-older-than, if there are no increments that
need to be removed, I see a message, apparently on standard error, saying
something like

"No increments older than Tue Nov 22 11:46:48 2005 found, exiting."

Isn't this somewhat unnecessary? This seems hardly to merit a message at
all, let alone one on standard error.

Thanks for reading. I look forward to hearing back. Please cc me on any
reply. Thanks.

Faheem.

Faheem Mitha wrote:
This predictably means that (since the previous job has not completed) at
some point two different instances of rdiff-backup will be running and
trying to write to the same directory
.
.
I wonder whether it would not be possible
for rdiff-backup to detect that another rdiff-backup process was writing
to the same directory and exit gracefully?

I see this as down to the system administrator to manage, and much
easier too. For example, suppose I have two backup jobs, one backing up
user xxx and a second backing up user yyy. Those two backup jobs might
writing, respectively, to:

/backups/home/xxx/...
/backups/home/yyy...

That won't cause a problem at all. Now suppose on your system you backup
the whole of /home. A second invocation is started, so at one point
there are two rdiff-backup processes, and they are writing to the
directories shown above. How should rdiff-backup decide that the former
pair of processes are safe to run, but the latter pair are not? Yes, it
can be done by parsing command lines of other processes - but even that
may not be possible if each of user xxx and yyy are running the backup
process under their own UIDs.

What I do:

#!/bin/bash
# Start of script
ACTIVE_FLAG=/var/run/backup_in_progress
[ -e $ACTIVE_FLAG] && exit 1 || touch $ACTIVE_FLAG
.
.
# End of script
rm $ACTIVE_FLAG

Actually it's a little more sophisticated than that, but you get the idea.

Keith

--
Keith Edmunds

+---------------------------------------------------------------------+
| Tiger Computing Ltd | Helping businesses make the most of Linux |
| "The Linux Company" | http://www.TheLinuxConsultancy.co.uk |
+---------------------------------------------------------------------+

On Tue, 22 Nov 2005, Keith Edmunds wrote:

I see this as down to the system administrator to manage, and much easier
too. For example, suppose I have two backup jobs, one backing up user xxx and
a second backing up user yyy. Those two backup jobs might writing,
respectively, to:

/backups/home/xxx/...
/backups/home/yyy...

That won't cause a problem at all. Now suppose on your system you backup the
whole of /home. A second invocation is started, so at one point there are two
rdiff-backup processes, and they are writing to the directories shown above.
How should rdiff-backup decide that the former pair of processes are safe to
run, but the latter pair are not? Yes, it can be done by parsing command
lines of other processes - but even that may not be possible if each of user
xxx and yyy are running the backup process under their own UIDs.

Actually, one could use some kind of locking mechanism, since the
directory being written to by rdiff-backup is well-defined by the
existence of the rdiff-backup-data directory at the top level.

This is hardly an ideal solution though, since one problem with locking is
guaranteeing that it be released correctly.

What I do:

#!/bin/bash
# Start of script
ACTIVE_FLAG=/var/run/backup_in_progress
[ -e $ACTIVE_FLAG] && exit 1 || touch $ACTIVE_FLAG
.
.
# End of script
rm $ACTIVE_FLAG

Actually it's a little more sophisticated than that, but you get the idea.

Yes, fair enough. I'm doing something similar in concept, but using mounts
of LVM volumes instead. The trouble with what you are doing it that if the
script exits unexpectedly, the ACTIVE_FLAG file might still remain, no?

I was just wondering if this issue had been considered, and what the
concensus thinking about it was.

Thanks for your comments.
Faheem.

Faheem Mitha <faheem < at > email.unc.edu>
wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)

I managed to work around this by having the script check whether the
LVM snapshot were mounted first, since this would mean the previous
backups were still in progress. However, I wonder whether it would
not be possible for rdiff-backup to detect that another rdiff-backup
process was writing to the same directory and exit gracefully?
Perhaps this would be difficult to do in a portable way.

The development version writes its PID to the current_mirror file in
the backup repository. Some future version should check that PID and
abort if the process is still running (unless you run --force or
something)

Secondly, I noticed that the --remove-older-than option removes files in
the increments directory, but does not touch any of the log and statistics
files that are generated, eg session_statistics, error_log,
mirror_metadata etc.

Are you sure it doesn't remove those? It seems to from a few quick
tests I did. In fact, I just found a bug, if you pick a time later
than the current mirror time it will delete those too :-(

Thirdly, when using --remove-older-than, if there are no increments that
need to be removed, I see a message, apparently on standard error, saying
something like

"No increments older than Tue Nov 22 11:46:48 2005 found, exiting."

Isn't this somewhat unnecessary? This seems hardly to merit a message at
all, let alone one on standard error.

Ok, I'll remove that message.


--
Ben Escoto

Ben Escoto wrote:
Faheem Mitha <faheem < at > email.unc.edu>
wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)
Thirdly, when using --remove-older-than, if there are no increments that
need to be removed, I see a message, apparently on standard error, saying
something like

"No increments older than Tue Nov 22 11:46:48 2005 found, exiting."

Isn't this somewhat unnecessary? This seems hardly to merit a message at
all, let alone one on standard error.


Ok, I'll remove that message.

Hold on! I like that message. In my nightly crontab, I delete any backups
older than 60 days, so I would like to see if there were any deleted.

Regards,
Blair

--
Blair Zajac, Ph.D.
<blair < at > orcaware.com>
Subversion and Orca training and consulting
http://www.orcaware.com/svn/

On Tue, 22 Nov 2005, Ben Escoto wrote:

Faheem Mitha <faheem < at > email.unc.edu>
wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)

I managed to work around this by having the script check whether the
LVM snapshot were mounted first, since this would mean the previous
backups were still in progress. However, I wonder whether it would
not be possible for rdiff-backup to detect that another rdiff-backup
process was writing to the same directory and exit gracefully?
Perhaps this would be difficult to do in a portable way.

The development version writes its PID to the current_mirror file in
the backup repository. Some future version should check that PID and
abort if the process is still running (unless you run --force or
something)

Yes, that sounds like a good idea.

Secondly, I noticed that the --remove-older-than option removes files in
the increments directory, but does not touch any of the log and statistics
files that are generated, eg session_statistics, error_log,
mirror_metadata etc.

Are you sure it doesn't remove those? It seems to from a few quick
tests I did. In fact, I just found a bug, if you pick a time later
than the current mirror time it will delete those too :-(

Pretty sure. I've been doing extensive testing. However, I'm running the
current Debian version in Sarge, namely 1.0.1. Is it possible this is
something that has been fixed recently?

In any case, if you can point me to the code that removes the
log/statistics files in the code (I looked in 1.0.1 but could not find
anything) then that would be very helpful.

Hmm, now looking in 1.1.2, but can't see anything there either.

Thanks. Faheem.

On Tue, 22 Nov 2005, Blair Zajac wrote:

Ben Escoto wrote:
Faheem Mitha <faheem < at > email.unc.edu>
wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)
Thirdly, when using --remove-older-than, if there are no increments that
need to be removed, I see a message, apparently on standard error, saying
something like

"No increments older than Tue Nov 22 11:46:48 2005 found, exiting."

Isn't this somewhat unnecessary? This seems hardly to merit a message at
all, let alone one on standard error.


Ok, I'll remove that message.

Hold on! I like that message. In my nightly crontab, I delete any backups
older than 60 days, so I would like to see if there were any deleted.

Ok, but couldn't this go to standard output rather than standard error
instead? It is not an error message.

Faheem.

Faheem Mitha wrote:


On Tue, 22 Nov 2005, Blair Zajac wrote:

Ben Escoto wrote:

Faheem Mitha <faheem < at > email.unc.edu>
wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)

Thirdly, when using --remove-older-than, if there are no increments
that need to be removed, I see a message, apparently on standard
error, saying something like

"No increments older than Tue Nov 22 11:46:48 2005 found, exiting."

Isn't this somewhat unnecessary? This seems hardly to merit a
message at all, let alone one on standard error.



Ok, I'll remove that message.


Hold on! I like that message. In my nightly crontab, I delete any
backups older than 60 days, so I would like to see if there were any
deleted.


Ok, but couldn't this go to standard output rather than standard error
instead? It is not an error message.

Fine with me.

Regards,
Blair

On Tue, 22 Nov 2005, Faheem Mitha wrote:

Secondly, I noticed that the --remove-older-than option removes files in
the increments directory, but does not touch any of the log and statistics
files that are generated, eg session_statistics, error_log,
mirror_metadata etc.

Are you sure it doesn't remove those? It seems to from a few quick
tests I did. In fact, I just found a bug, if you pick a time later
than the current mirror time it will delete those too :-(

Pretty sure. I've been doing extensive testing. However, I'm running the
current Debian version in Sarge, namely 1.0.1. Is it possible this is
something that has been fixed recently?

In any case, if you can point me to the code that removes the log/statistics
files in the code (I looked in 1.0.1 but could not find anything) then that
would be very helpful.

Hmm, now looking in 1.1.2, but can't see anything there either.

Ok, further info.

I have two test backups running every 5 minutes. I have it set to clear
out everything more than an hr old.

I enabled the one hr removal after the backups had been running for a
while, and it does not clear the log and statistics files in that case.

I am not sure whether it cleared the increments.

However, I deleted one on the backups and restarted it with the 1 hr
removal enabled, and now it is deleting the log/statistics files
correctly.

It looks to me like there is a bug somewhere.

Faheem.