SearchFAQMemberlist Log in
Reply to topic Page 1 of 1
Safe to mirror backup dir with rsync?
Author Message
Post Safe to mirror backup dir with rsync? 
Hello. I tried searching for this in the mailing list, but could not find a definitive answer to my question, so... I am using rdiff-backup to backup to a local directory. I would like to have this backup mirrored to another server (and potentially mirrored a third time to another server with a portable hard drive attached). Is it safe (and recommended) to just use rsync for the mirror of the backup directory? What if rsync fails to finish a session? Will the mirror of the backup directory end up in a consistent state such that I can restore from it if necessary? If this is not feasible, then I might try to simply run rdiff-backup from the remote server from the original files and not the local backup. But I would prefer to mirror the first backup if possible. Thanks in advance for any help/advice.

Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net






_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
Hi Engel,

Yes rsync would work for this but if you are connecting to a server that
has or can have rdiff-backup on it you may as well run
rdiff-backup to that server and then again for the local copy. All from
the original files.

I rdiff-backup all my home and server boxes to a /backup partition on
one machine.
Then rsync /backup to an encrypted portable hard drive.
I also run with box A keeps box B's backups while box B keeps box A's
backups.

Naturally rsync must complete successfully for the mirror to be a mirror
but rsync can recover from interruptions.
This is why rsync is often used to start an rdiff-backup repository.
Check out backupninja for setting up such backup systems.

Cheers
Gavin

Engel Sanchez wrote:
Hello. I tried searching for this in the mailing list, but could not find a definitive answer to my question, so... I am using rdiff-backup to backup to a local directory. I would like to have this backup mirrored to another server (and potentially mirrored a third time to another server with a portable hard drive attached). Is it safe (and recommended) to just use rsync for the mirror of the backup directory? What if rsync fails to finish a session? Will the mirror of the backup directory end up in a consistent state such that I can restore from it if necessary? If this is not feasible, then I might try to simply run rdiff-backup from the remote server from the original files and not the local backup. But I would prefer to mirror the first backup if possible. Thanks in advance for any help/advice.

Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net






_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki




_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
Hello,

I'm using rdiff-backup to create the backups of my directories. It all
goes to /backup, which is a external harddrive. Then I'm using rsync to
mirror everything to /backup_sync, which is a second external harddrive.
This one is encrypted and I move it to an external location once a week
(and the disk from the external location then becomes the sync disk Wink
). I'm using this strategy because when I swap the sync disks, they are
disconnected from my computer for a day. Some data I'm backing up every
hour, so I need a external drive at my computer at all times.
This system is working perfectly for me. The only thing I would
recommend is to prevent running rdiff-backup at the same time as rsync.
To prevent that your mirror is accidentally destroyed, because you would
by rsyncing incomplete rdiff-backup data. I haven't build in this
precaution in my backupscripts yet. For the time being I'm preventing it
from running at the same moment by choosing the correct times in my
cronjobs.

Cheers,
Cybertinus

On 23/12/09 00:03, Gavin wrote:
Hi Engel,

Yes rsync would work for this but if you are connecting to a server that
has or can have rdiff-backup on it you may as well run
rdiff-backup to that server and then again for the local copy. All from
the original files.

I rdiff-backup all my home and server boxes to a /backup partition on
one machine.
Then rsync /backup to an encrypted portable hard drive.
I also run with box A keeps box B's backups while box B keeps box A's
backups.

Naturally rsync must complete successfully for the mirror to be a mirror
but rsync can recover from interruptions.
This is why rsync is often used to start an rdiff-backup repository.
Check out backupninja for setting up such backup systems.

Cheers
Gavin

Engel Sanchez wrote:

Hello. I tried searching for this in the mailing list, but could not find a definitive answer to my question, so... I am using rdiff-backup to backup to a local directory. I would like to have this backup mirrored to another server (and potentially mirrored a third time to another server with a portable hard drive attached). Is it safe (and recommended) to just use rsync for the mirror of the backup directory? What if rsync fails to finish a session? Will the mirror of the backup directory end up in a consistent state such that I can restore from it if necessary? If this is not feasible, then I might try to simply run rdiff-backup from the remote server from the original files and not the local backup. But I would prefer to mirror the first backup if possible. Thanks in advance for any help/advice.

Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net






_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki




_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
On Wed, Dec 23, 2009 at 12:52:03AM +0100, Cybertinus wrote:
Hello,

I'm using rdiff-backup to create the backups of my directories. It all
goes to /backup, which is a external harddrive. Then I'm using rsync to
mirror everything to /backup_sync, which is a second external harddrive.
This one is encrypted and I move it to an external location once a week
(and the disk from the external location then becomes the sync disk Wink
). I'm using this strategy because when I swap the sync disks, they are
disconnected from my computer for a day. Some data I'm backing up every
hour, so I need a external drive at my computer at all times.
This system is working perfectly for me. The only thing I would
recommend is to prevent running rdiff-backup at the same time as rsync.
To prevent that your mirror is accidentally destroyed, because you would
by rsyncing incomplete rdiff-backup data. I haven't build in this
precaution in my backupscripts yet. For the time being I'm preventing it
from running at the same moment by choosing the correct times in my
cronjobs.

Hi

I am doing something similar.

local machine rdiff-backup to /backup (lvm partition).
then a rsync to the local backup server and at the same time a rsync to
an offsite location - via adsl.

I do this because my initial rdiff-backup can happen quickly and cause
the lest disruption - if the source is on a lvm I can take a snapshot
and use that but not all the sources are.

I have come across one problem, if the over adsl takes too long the
rdiff-backup destination on the local machine can change thus giving you
a non in sync backup offsite.

There is another thread asking for a tool to lock the destination to
prevent this

Alex


Cheers,
Cybertinus

On 23/12/09 00:03, Gavin wrote:
Hi Engel,
[snip]

--
Darth Vader:
Your powers are weak, old man.
Ben (Obi-Wan) Kenobi:
You can't win, Darth. If you strike me down, I shall
become more powerful than you could possibly
imagine.

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
Thank you all for your suggestions! I'm going to go with the solution of doing cascaded rdiff-backups both pointing at the source directories and without rsync and simply spreading the cron jobs to avoid touching the sources while they are updated (the sources are hot copies of a couple of databases and a subversion repo). Using rsync and risking leaving the remote backup in a bad state didn't seem like a great idea to me. (Say rsync's connection breaks and original server fails before next backup). At least rdiff-backup can deal with a partially updated backup if also done by rdiff-backup.

Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net



----- Original Message ----
From: Alex Samad <alex < at > samad.com.au>
To: rdiff-backup-users < at > nongnu.org
Sent: Wed, December 23, 2009 4:13:16 PM
Subject: Re: [rdiff-backup-users] Safe to mirror backup dir with rsync?

On Wed, Dec 23, 2009 at 12:52:03AM +0100, Cybertinus wrote:
Hello,

I'm using rdiff-backup to create the backups of my directories. It all
goes to /backup, which is a external harddrive. Then I'm using rsync to
mirror everything to /backup_sync, which is a second external harddrive.
This one is encrypted and I move it to an external location once a week
(and the disk from the external location then becomes the sync disk Wink
). I'm using this strategy because when I swap the sync disks, they are
disconnected from my computer for a day. Some data I'm backing up every
hour, so I need a external drive at my computer at all times.
This system is working perfectly for me. The only thing I would
recommend is to prevent running rdiff-backup at the same time as rsync.
To prevent that your mirror is accidentally destroyed, because you would
by rsyncing incomplete rdiff-backup data. I haven't build in this
precaution in my backupscripts yet. For the time being I'm preventing it
from running at the same moment by choosing the correct times in my
cronjobs.

Hi

I am doing something similar.

local machine rdiff-backup to /backup (lvm partition).
then a rsync to the local backup server and at the same time a rsync to
an offsite location - via adsl.

I do this because my initial rdiff-backup can happen quickly and cause
the lest disruption - if the source is on a lvm I can take a snapshot
and use that but not all the sources are.

I have come across one problem, if the over adsl takes too long the
rdiff-backup destination on the local machine can change thus giving you
a non in sync backup offsite.

There is another thread asking for a tool to lock the destination to
prevent this

Alex


Cheers,
Cybertinus

On 23/12/09 00:03, Gavin wrote:
Hi Engel,
[snip]

--
Darth Vader:
Your powers are weak, old man.
Ben (Obi-Wan) Kenobi:
You can't win, Darth. If you strike me down, I shall
become more powerful than you could possibly
imagine.






_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
On Wed, Dec 23, 2009 at 01:58:30PM -0800, Engel Sanchez wrote:
Thank you all for your suggestions! I'm going to go with the solution of doing cascaded rdiff-backups both pointing at the source directories and without rsync and simply spreading the cron jobs to avoid touching the sources while they are updated (the sources are hot copies of a couple of databases and a subversion repo). Using rsync and risking leaving the remote backup in a bad state didn't seem like a great idea to me. (Say rsync's connection breaks and original server fails before next backup). At least rdiff-backup can deal with a partially updated backup if also done by rdiff-backup.

strange for the same reasons I have used rsync - I can capture when it
fails and restart and it doesn't need to roll back it continues where it
left off. rdiff-backup has to unroll and then restart so for a 2 hour
backup that fails in the last 10 min, you will need to do another 2
hours of backup with rdiff-backup and only 10 min with rsync....

to stop the corruption problem right now I use semaphores around my
backup scripts to stop certain things happening at the same time Smile

Alex


Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net



----- Original Message ----
From: Alex Samad <alex < at > samad.com.au>
To: rdiff-backup-users < at > nongnu.org
Sent: Wed, December 23, 2009 4:13:16 PM
Subject: Re: [rdiff-backup-users] Safe to mirror backup dir with rsync?

On Wed, Dec 23, 2009 at 12:52:03AM +0100, Cybertinus wrote:
Hello,

I'm using rdiff-backup to create the backups of my directories. It all
goes to /backup, which is a external harddrive. Then I'm using rsync to
mirror everything to /backup_sync, which is a second external harddrive.
This one is encrypted and I move it to an external location once a week
(and the disk from the external location then becomes the sync disk Wink
). I'm using this strategy because when I swap the sync disks, they are
disconnected from my computer for a day. Some data I'm backing up every
hour, so I need a external drive at my computer at all times.
This system is working perfectly for me. The only thing I would
recommend is to prevent running rdiff-backup at the same time as rsync.
To prevent that your mirror is accidentally destroyed, because you would
by rsyncing incomplete rdiff-backup data. I haven't build in this
precaution in my backupscripts yet. For the time being I'm preventing it
from running at the same moment by choosing the correct times in my
cronjobs.

Hi

I am doing something similar.

local machine rdiff-backup to /backup (lvm partition).
then a rsync to the local backup server and at the same time a rsync to
an offsite location - via adsl.

I do this because my initial rdiff-backup can happen quickly and cause
the lest disruption - if the source is on a lvm I can take a snapshot
and use that but not all the sources are.

I have come across one problem, if the over adsl takes too long the
rdiff-backup destination on the local machine can change thus giving you
a non in sync backup offsite.

There is another thread asking for a tool to lock the destination to
prevent this

Alex


Cheers,
Cybertinus

On 23/12/09 00:03, Gavin wrote:
Hi Engel,
[snip]


--
"Bureaucracy is the enemy of innovation."
-- Mark Shepherd, former President and CEO of Texas Instruments

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
Hi Alex. My understanding could be wrong, but this is the scenario that I'm trying to avoid:

1) Machine 1 has the original data and backs it up locally to /backup
2) Machine 2 rsyncs /backup to local /backup every day
3) Machine 2 is rsyncing /backup, and right then machine 1 dies for good (fire, flood, apocalypse)

At that point in time, machine 1 is dead and /backup in machine 2 is not a valid rdiff-backup directory, so you can't restore your data from it anymore. If Machine 2 was instead using rdiff-backup on the original sources, /backup would have an incomplete backup, but previous data would not be lost, so you can restore from it even if it was interrupted.

Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net



----- Original Message ----
From: Alex Samad <alex < at > samad.com.au>
To: rdiff-backup-users < at > nongnu.org
Sent: Wed, December 23, 2009 6:32:37 PM
Subject: Re: [rdiff-backup-users] Safe to mirror backup dir with rsync?

On Wed, Dec 23, 2009 at 01:58:30PM -0800, Engel Sanchez wrote:
Thank you all for your suggestions! I'm going to go with the solution of doing cascaded rdiff-backups both pointing at the source directories and without rsync and simply spreading the cron jobs to avoid touching the sources while they are updated (the sources are hot copies of a couple of databases and a subversion repo). Using rsync and risking leaving the remote backup in a bad state didn't seem like a great idea to me. (Say rsync's connection breaks and original server fails before next backup). At least rdiff-backup can deal with a partially updated backup if also done by rdiff-backup.

strange for the same reasons I have used rsync - I can capture when it
fails and restart and it doesn't need to roll back it continues where it
left off. rdiff-backup has to unroll and then restart so for a 2 hour
backup that fails in the last 10 min, you will need to do another 2
hours of backup with rdiff-backup and only 10 min with rsync....

to stop the corruption problem right now I use semaphores around my
backup scripts to stop certain things happening at the same time :)

Alex


Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net



----- Original Message ----
From: Alex Samad <alex < at > samad.com.au>
To: rdiff-backup-users < at > nongnu.org
Sent: Wed, December 23, 2009 4:13:16 PM
Subject: Re: [rdiff-backup-users] Safe to mirror backup dir with rsync?

On Wed, Dec 23, 2009 at 12:52:03AM +0100, Cybertinus wrote:
Hello,

I'm using rdiff-backup to create the backups of my directories. It all
goes to /backup, which is a external harddrive. Then I'm using rsync to
mirror everything to /backup_sync, which is a second external harddrive.
This one is encrypted and I move it to an external location once a week
(and the disk from the external location then becomes the sync disk Wink
). I'm using this strategy because when I swap the sync disks, they are
disconnected from my computer for a day. Some data I'm backing up every
hour, so I need a external drive at my computer at all times.
This system is working perfectly for me. The only thing I would
recommend is to prevent running rdiff-backup at the same time as rsync.
To prevent that your mirror is accidentally destroyed, because you would
by rsyncing incomplete rdiff-backup data. I haven't build in this
precaution in my backupscripts yet. For the time being I'm preventing it
from running at the same moment by choosing the correct times in my
cronjobs.

Hi

I am doing something similar.

local machine rdiff-backup to /backup (lvm partition).
then a rsync to the local backup server and at the same time a rsync to
an offsite location - via adsl.

I do this because my initial rdiff-backup can happen quickly and cause
the lest disruption - if the source is on a lvm I can take a snapshot
and use that but not all the sources are.

I have come across one problem, if the over adsl takes too long the
rdiff-backup destination on the local machine can change thus giving you
a non in sync backup offsite.

There is another thread asking for a tool to lock the destination to
prevent this

Alex


Cheers,
Cybertinus

On 23/12/09 00:03, Gavin wrote:
Hi Engel,
[snip]


--
"Bureaucracy is the enemy of innovation."
-- Mark Shepherd, former President and CEO of Texas Instruments






_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
Am 2009-12-24 01:55, schrieb Engel Sanchez:
Hi Alex. My understanding could be wrong, but this is the scenario that I'm trying to avoid:

1) Machine 1 has the original data and backs it up locally to /backup
2) Machine 2 rsyncs /backup to local /backup every day
3) Machine 2 is rsyncing /backup, and right then machine 1 dies for good (fire, flood, apocalypse)

At that point in time, machine 1 is dead and /backup in machine 2 is not a valid rdiff-backup directory, so you can't restore your data from it anymore.

You can still restore from machine 2.

The history will probably be broken and the latest backup will at the
filesystem level not be a perfect snapshot in time (if you aren't using
LVM snapshots you don't get that snapshot in time anyway).

But every file in the latest backup on it's own will be consistent
(rsync does not modify files in-place).

So, you will have some files that are from the latest backup, some that
are from the one before.

Acceptable IMO.


Jakob


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
Hi Jakob. That depends on the application using those files. If we are taking about a source code repository like subversion, having some files new, some files old like that might make the repo unusable anyway.

Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net



----- Original Message ----
From: Jakob Unterwurzacher <jakobunt < at > gmail.com>
To: rdiff-backup-users < at > nongnu.org
Sent: Wed, December 23, 2009 8:25:33 PM
Subject: Re: [rdiff-backup-users] Safe to mirror backup dir with rsync?

Am 2009-12-24 01:55, schrieb Engel Sanchez:
Hi Alex. My understanding could be wrong, but this is the scenario that I'm trying to avoid:

1) Machine 1 has the original data and backs it up locally to /backup
2) Machine 2 rsyncs /backup to local /backup every day
3) Machine 2 is rsyncing /backup, and right then machine 1 dies for good (fire, flood, apocalypse)

At that point in time, machine 1 is dead and /backup in machine 2 is not a valid rdiff-backup directory, so you can't restore your data from it anymore.

You can still restore from machine 2.

The history will probably be broken and the latest backup will at the filesystem level not be a perfect snapshot in time (if you aren't using LVM snapshots you don't get that snapshot in time anyway).

But every file in the latest backup on it's own will be consistent (rsync does not modify files in-place).

So, you will have some files that are from the latest backup, some that are from the one before.

Acceptable IMO.


Jakob


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki






_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
On 12/23/2009 05:55 PM, Engel Sanchez wrote:
Hi Alex. My understanding could be wrong, but this is the scenario that I'm trying to avoid:

1) Machine 1 has the original data and backs it up locally to /backup
2) Machine 2 rsyncs /backup to local /backup every day
3) Machine 2 is rsyncing /backup, and right then machine 1 dies for good (fire, flood, apocalypse)

At that point in time, machine 1 is dead and /backup in machine 2 is not a valid rdiff-backup directory, so you can't restore your data from it anymore. If Machine 2 was instead using rdiff-backup on the original sources, /backup would have an incomplete backup, but previous data would not be lost, so you can restore from it even if it was interrupted.

Use rsync's --delay-updates. That will narrow the window of failure
significantly.

If that's not enough, create a temporary hard-linked copy of /backup on
machine 2:

set -e

cp -al /backup /backup.new
rsync -a --del machine1:/backup/ /backup.new/
mv /backup /backup.old
mv /backup.new /backup
rm -r /backup.old

Steven


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
On Wed, Dec 23, 2009 at 04:55:11PM -0800, Engel Sanchez wrote:
Hi Alex. My understanding could be wrong, but this is the scenario that I'm trying to avoid:

1) Machine 1 has the original data and backs it up locally to /backup
2) Machine 2 rsyncs /backup to local /backup every day
3) Machine 2 is rsyncing /backup, and right then machine 1 dies for good (fire, flood, apocalypse)

At that point in time, machine 1 is dead and /backup in machine 2 is not a valid rdiff-backup directory, so you can't restore your data from it anymore. If Machine 2 was instead using rdiff-backup on the original sources, /backup would have an incomplete backup, but previous data would not be lost, so you can restore from it even if it was interrupted.


Very true, another example of different horses for different courses. At
any one time I have 3 rdiff backup repos. 1 local machine, 1 local lan
another machine and 1 different location.

the rsync's are done in sequence, so if the source dies the off site one will have a complete backup.


All this does is show how DR / Backups / HA (not all the same) have to
be considered for each case.

A


Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net



----- Original Message ----
From: Alex Samad <alex < at > samad.com.au>
To: rdiff-backup-users < at > nongnu.org
Sent: Wed, December 23, 2009 6:32:37 PM
Subject: Re: [rdiff-backup-users] Safe to mirror backup dir with rsync?

On Wed, Dec 23, 2009 at 01:58:30PM -0800, Engel Sanchez wrote:
Thank you all for your suggestions! I'm going to go with the solution of doing cascaded rdiff-backups both pointing at the source directories and without rsync and simply spreading the cron jobs to avoid touching the sources while they are updated (the sources are hot copies of a couple of databases and a subversion repo). Using rsync and risking leaving the remote backup in a bad state didn't seem like a great idea to me. (Say rsync's connection breaks and original server fails before next backup). At least rdiff-backup can deal with a partially updated backup if also done by rdiff-backup.

strange for the same reasons I have used rsync - I can capture when it
fails and restart and it doesn't need to roll back it continues where it
left off. rdiff-backup has to unroll and then restart so for a 2 hour
backup that fails in the last 10 min, you will need to do another 2
hours of backup with rdiff-backup and only 10 min with rsync....

to stop the corruption problem right now I use semaphores around my
backup scripts to stop certain things happening at the same time Smile

Alex


Engel A. Sanchez


“Do what you love to do and give it your very best. Whether it's business or baseball, or the theater, or any field. If you don't love what you're doing and you can't give it your best, get out of it. Life is too short. You'll be an old man before you know it.”


http://engelsanchez.net



----- Original Message ----
From: Alex Samad <alex < at > samad.com.au>
To: rdiff-backup-users < at > nongnu.org
Sent: Wed, December 23, 2009 4:13:16 PM
Subject: Re: [rdiff-backup-users] Safe to mirror backup dir with rsync?

On Wed, Dec 23, 2009 at 12:52:03AM +0100, Cybertinus wrote:
Hello,

I'm using rdiff-backup to create the backups of my directories. It all
goes to /backup, which is a external harddrive. Then I'm using rsync to
mirror everything to /backup_sync, which is a second external harddrive.
This one is encrypted and I move it to an external location once a week
(and the disk from the external location then becomes the sync disk Wink
). I'm using this strategy because when I swap the sync disks, they are
disconnected from my computer for a day. Some data I'm backing up every
hour, so I need a external drive at my computer at all times.
This system is working perfectly for me. The only thing I would
recommend is to prevent running rdiff-backup at the same time as rsync.
To prevent that your mirror is accidentally destroyed, because you would
by rsyncing incomplete rdiff-backup data. I haven't build in this
precaution in my backupscripts yet. For the time being I'm preventing it
from running at the same moment by choosing the correct times in my
cronjobs.

Hi

I am doing something similar.

local machine rdiff-backup to /backup (lvm partition).
then a rsync to the local backup server and at the same time a rsync to
an offsite location - via adsl.

I do this because my initial rdiff-backup can happen quickly and cause
the lest disruption - if the source is on a lvm I can take a snapshot
and use that but not all the sources are.

I have come across one problem, if the over adsl takes too long the
rdiff-backup destination on the local machine can change thus giving you
a non in sync backup offsite.

There is another thread asking for a tool to lock the destination to
prevent this

Alex


Cheers,
Cybertinus

On 23/12/09 00:03, Gavin wrote:
Hi Engel,
[snip]



--
Boucher's Observation:
He who blows his own horn always plays the music
several octaves higher than originally written.

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Safe to mirror backup dir with rsync? 
On Wed, Dec 23, 2009 at 07:39:05PM -0700, Steven Willoughby wrote:
On 12/23/2009 05:55 PM, Engel Sanchez wrote:
Hi Alex. My understanding could be wrong, but this is the scenario that I'm trying to avoid:

1) Machine 1 has the original data and backs it up locally to /backup
2) Machine 2 rsyncs /backup to local /backup every day
3) Machine 2 is rsyncing /backup, and right then machine 1 dies for good (fire, flood, apocalypse)

At that point in time, machine 1 is dead and /backup in machine 2 is not a valid rdiff-backup directory, so you can't restore your data from it anymore. If Machine 2 was instead using rdiff-backup on the original sources, /backup would have an incomplete backup, but previous data would not be lost, so you can restore from it even if it was interrupted.

Use rsync's --delay-updates. That will narrow the window of failure
significantly.

If that's not enough, create a temporary hard-linked copy of /backup
on machine 2:

set -e

cp -al /backup /backup.new
rsync -a --del machine1:/backup/ /backup.new/
mv /backup /backup.old
mv /backup.new /backup
rm -r /backup.old

before using rdiff-backup I used to use rolling rsync directories with
the --link-dest option and renaming directories T T-1 T-2. The same
principle could be used here as well.


Or just changing the schedule of my original operation.

1) rdiff-backup to local lan different machine
2) rsync from local lan different machine to remote site.

My choice for local machine is purely for speed.
one of my backup sets is a VirtualBox directory ~80G

Alex


Steven


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Display posts from previous:
Reply to topic Page 1 of 1
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
  


Magic SEO URL for phpBB