Backup Central

Hi!

I'm trying to restore a backup that has already been written while an
rdiff-backup session is running, and this fails:

machine:/home/user# rdiff-backup -r 1D /some/rdiff-backup/repo/foo .

Fatal Error: It appears that a previous rdiff-backup session with process
id 353 is still running. If two different rdiff-backup processes write
the same repository simultaneously, data corruption will probably
result. To proceed with regress anyway, rerun rdiff-backup with the
--force option.

If I add '--force', it tells me that the previous backup has failed
(which is wrong)

Is this intended behavior, or am I doing something wrong? Shouldn't it
be possible to do a read operation on the repository without affecting
the process that's currently making a new increment?
If it is intended, are there any workarounds for this? We need to be
able to restore a backup within minutes, and a backup process can last
for 6+ hours (and we do them twice a day).


--
Helge Milde, 69701808 / 41747242
www.monsternett.no



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Helge Milde wrote:

Quote:

If it is intended, are there any workarounds for this? We need to be
able to restore a backup within minutes, and a backup process can last
for 6+ hours (and we do them twice a day).

I believe this is intended behaviour. The backups are mean to be atomic.
Thus its not possible to restore increments inside the time it takes to
backup (ie while the backup is running). However, if you are not trying
to restore an increment, you could just copy the file out of the
repository you wanted (assuming the running backup process has
progressed past that file.
Say you have directories A, B, C, D, and the backup takes ages on C, but
has already done the files in A, then you can just copy them out of the
repository. You only need to involve rdiff-backup for increments.
Other than that, you will probably just have to backup less data in a
job if your time to restore falls inside your backup window.
IE, have more, fewer backup jobs.

dave


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Dave Kempe wrote:

Quote:

Helge Milde wrote:

Quote:

If it is intended, are there any workarounds for this? We need to be
able to restore a backup within minutes, and a backup process can
last for 6+ hours (and we do them twice a day).

I believe this is intended behaviour. The backups are mean to be
atomic. Thus its not possible to restore increments inside the time it
takes to backup (ie while the backup is running).

Can you elaborate on this, please? I don't understand what's technically
preventing rdiff-backup from letting me restore a backup from (for
instance) a given date. As far as I understand, rdiff already has all
the data it needs to do a restore, and since it should be a read-only
operation, I don't understand how that would impact or corrupt the
ongoing backup process.



--
Helge Milde



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Helge Milde wrote:

Quote:

Can you elaborate on this, please? I don't understand what's technically
preventing rdiff-backup from letting me restore a backup from (for
instance) a given date. As far as I understand, rdiff already has all
the data it needs to do a restore, and since it should be a read-only
operation, I don't understand how that would impact or corrupt the
ongoing backup process.

Well you may be right logically, but I am not sure rdiff-backup could
allow this to happen in the way the meta-data and other backup processes
work. I believe the checks in place for restoring are put in place to
ensure the logical consistency of the files in the backup repository.
Otherwise you may have some files from one age and some from other. Not
to mention the mess that could make with reverse diffs etc.
To me, it seems like a dangerous operation, and one that rdiff-backup is
designed to avoid. However, that being said, if you can design and
implement a way to do it, I think the list would be happy to test out
your patches.
The other developers may have a better and more in depth opinion than me
(certainly more qualified than me!). The answers are in the source of
course.

thanks

dave


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Dave Kempe wrote:

Quote:

Helge Milde wrote:

Quote:

If it is intended, are there any workarounds for this? We need to be able to restore a backup within minutes, and a backup process can last for 6+ hours (and we do them twice a day).

I believe this is intended behaviour. The backups are mean to be atomic. Thus its not possible to restore increments inside the time it takes to backup (ie while the backup is running). However, if you are not trying to restore an increment, you could just copy the file out of the repository you wanted (assuming the running backup process has progressed past that file.
Say you have directories A, B, C, D, and the backup takes ages on C, but has already done the files in A, then you can just copy them out of the repository. You only need to involve rdiff-backup for increments.
Other than that, you will probably just have to backup less data in a job if your time to restore falls inside your backup window.
IE, have more, fewer backup jobs.

It may be intended behaviour but IMHO the --force override ought to be allowed. Helge's case seems a good example of why one would want to use this.

Does --force work - or is it even necessary - if accessing a different repository while an rdiff-backup process is ongoing?

Can one have multiple rdiff-backup sessions running from different clients simultaneously to one server but to different repositories (I hope and have assumed so, because this is bound to happen with push backups, which is the solution I am working towards)?

Dominic

Dominic wrote:

Quote:

Can one have multiple rdiff-backup sessions running from different
clients simultaneously to one server but to different repositories (I
hope and have assumed so, because this is bound to happen with push
backups, which is the solution I am working towards)?

all the locking and magic is limited to a repository. hence my
suggestion of having a smaller repository will lead to faster backups
and quicker restores. rdiff-backup retrieves the PID of the rdiff-backup
process if there are two current_mirror markers in the rdiff-backup data
directory and throws this error if there is.

dave


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Helge Milde wrote:

Quote:

Can you elaborate on this, please? I don't understand what's technically
preventing rdiff-backup from letting me restore a backup from (for
instance) a given date. As far as I understand, rdiff already has all
the data it needs to do a restore, and since it should be a read-only
operation, I don't understand how that would impact or corrupt the
ongoing backup process.

OK, say you want to restore file A from some time ago.

rdiff-backup must take the current version of the file and apply all
the rdiffs back to the version you need. So it starts with the
current version of File A.

Two scenarios :

1) The backup process tries to update File A and finds it locked -
backup fails.

2) The backup process doesn't find a locked file, but it managed to
swap out the file for a new version just before the restore accesses
it - but after it's accessed the metadata. The restore process now
computes the old version of the file to restore whilst missing out
the latest rdiff that it doesn't know about - you get a corrupt
restored file.


Both of these need very precise timing to 'fail', but it's far easier
to apply locking and be 100% sure it won't happen. The alternative
would be file by file locking which would complicate things somewhat.

--
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Simon Hobson wrote:

Quote:

Helge Milde wrote:

Quote:

Can you elaborate on this, please?

OK, say you want to restore file A from some time ago...

Without fully understanding this, would it be possible, if the
repository is inside an LVM logical volume (mine will be, as it so
happens ), to create a snapshot and then restore from that, while
the original backup proceeds on the original volume? Or would the
snapshot also be locked/unusable, whether for an rdiff-backup restore or
for other processes? (I want to replicate my rdiff-backup server
repositories to an offsite machine, and my plan is to have a cron job
which creates an LVM snapshot and then uses rsync to synchronise the
remote data, but again this activity might overlap with ongoing backups.)

Dominic


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Dave Kempe wrote:

Quote:

all the locking and magic is limited to a repository. hence my
suggestion of having a smaller repository will lead to faster backups
and quicker restores. rdiff-backup retrieves the PID of the
rdiff-backup process if there are two current_mirror markers in the
rdiff-backup data directory and throws this error if there is.

Thanks Dave that is a relief!

Dominic



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Dominic wrote:

Quote:

Without fully understanding this, would it be possible, if the
repository is inside an LVM logical volume (mine will be, as it so
happens ), to create a snapshot and then restore from that,
while the original backup proceeds on the original volume? Or would
the snapshot also be locked/unusable, whether for an rdiff-backup
restore or for other processes? (I want to replicate my rdiff-backup
server repositories to an offsite machine, and my plan is to have a
cron job which creates an LVM snapshot and then uses rsync to
synchronise the remote data, but again this activity might overlap
with ongoing backups.)

The repository would still be locked, as the snapshot would simply
show your the state of the disk at a set point in time. What you
could do is to create a snapshot before starting the backup, and then
destroy it later. I'd guess the timing you'd need would be :

Create snapshot (if not already in existance) before starting backup
Do backup
Kill snapshot IF there is no copy in progress.

Create snapshot (if not already in existance) before starting copy
Do copy from snapshot
Kill snapshot IF there is no backup in progress.


--
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

On Tue, 2008-11-25 at 16:26 +0000, Simon Hobson wrote:

Quote:

The repository would still be locked, as the snapshot would simply
show your the state of the disk at a set point in time. What you
could do is to create a snapshot before starting the backup, and then
destroy it later. I'd guess the timing you'd need would be :

Create snapshot (if not already in existance) before starting backup
Do backup
Kill snapshot IF there is no copy in progress.

Create snapshot (if not already in existance) before starting copy
Do copy from snapshot
Kill snapshot IF there is no backup in progress.

<plug>
BTW, this is one of the features built into safekeep already:
http://safekeep.sourceforge.net/
</plug>

--
Dimi Paun <dimi < at > lattica.com>
Lattica, Inc.



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Simon Hobson wrote:

Quote:

Dominic wrote:

Quote:

Without fully understanding this, would it be possible, if the
repository is inside an LVM logical volume (mine will be, as it so
happens ), to create a snapshot and then restore from that,
while the original backup proceeds on the original volume? Or would
the snapshot also be locked/unusable, whether for an rdiff-backup
restore or for other processes? (I want to replicate my rdiff-backup
server repositories to an offsite machine, and my plan is to have a
cron job which creates an LVM snapshot and then uses rsync to
synchronise the remote data, but again this activity might overlap
with ongoing backups.)

The repository would still be locked, as the snapshot would simply
show your the state of the disk at a set point in time. What you could
do is to create a snapshot before starting the backup, and then
destroy it later. I'd guess the timing you'd need would be :

Create snapshot (if not already in existance) before starting backup
Do backup
Kill snapshot IF there is no copy in progress.

Create snapshot (if not already in existance) before starting copy
Do copy from snapshot
Kill snapshot IF there is no backup in progress.

Neat! A further complication is that this would have to be initiated
from the client computers as it is a push backup situation, and most of
these run Windows, but I guess it can be done with plink. And there
might be some permissions issues since client computers connect as
restricted users. But thanks for the suggestion.

Dominic


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Dimi Paun wrote:

Quote:

Quote:

On Tue, 2008-11-25 at 16:26 +0000, Simon Hobson wrote:

Quote:

The repository would still be locked, as the snapshot would simply
show your the state of the disk at a set point in time. What you
could do is to create a snapshot before starting the backup, and then
destroy it later. I'd guess the timing you'd need would be :

Create snapshot (if not already in existance) before starting backup
Do backup
Kill snapshot IF there is no copy in progress.

Create snapshot (if not already in existance) before starting copy
Do copy from snapshot
Kill snapshot IF there is no backup in progress.

<plug>
BTW, this is one of the features built into safekeep already:
http://safekeep.sourceforge.net/
</plug>

Well it may be a plug, but it's justified IMHO for an open-source product that I had not heard of and that is based on rdiff-backup. Safekeep is not mentioned at http://www.nongnu.org/rdiff-backup/related.html. Looks good enough for me to wish I had seen it earlier. But is it only for Linux clients?

Dominic

Dominic wrote:

Quote:

Well it may be a plug, but it's justified IMHO for an open-source product that I had not heard of and that is based on rdiff-backup. Safekeep is not mentioned at http://www.nongnu.org/rdiff-backup/related.html. Looks good enough for me to wish I had seen it earlier. But is it only for Linux clients?

thats a good point - we should also add backupninja to that list as well.

dave

Dave Kempe wrote:

Quote:

thats a good point - we should also add backupninja to that list as well.

Although, they are on the wiki:
http://wiki.rdiff-backup.org/wiki/index.php/ContribScripts

dave


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki