Welcome! » Log In » Create A New Profile

rsnapshot without syncing

Posted by Anonymous 
rsnapshot without syncing
August 06, 2016 03:40PM
Summary: I'd like to use rsnapshot only for rotation, is that
possible? Can sync_first do rotation only (including hard link
copying) WITHOUT ever running sync?

Detailed Explanation:

I have a server with a larg-ish hard disk attached. Backups are done
from multiple machines (Linux using ssh+rsync, windows using cygwin
ssh+rsync, Android using sftp). The resulting folders for 2 machines
would look something like:-
/mnt/HDD/A
/mnt/HDD/B

I'm already using rsnapshot for some months, so I have:-
/mnt/HDD/snapshots/daily.0/A
/mnt/HDD/snapshots/daily.0/B
/mnt/HDD/snapshots/daily.1/A
/mnt/HDD/snapshots/daily.1/B
/mnt/HDD/snapshots/daily.2/A
/mnt/HDD/snapshots/daily.2/B
and so on, including weekly/monthly intervals.

There's actually two full copies of the data on this server. For
machine A, this is a copy at /mnt/HDD/A and another copy at
/mnt/HDD/snapshots/daily.0/A. I'd like to save some space, and imagine
I could perhaps do something like this:-

1. Tell rsnapshot to only rotate (mv daily.5->daily.6, mv daily.4->daily.5 ....)
2. Tell rsnapshot to do hard linking from daily.0 to daily.1

My intention is that machine A would run a backup rsync to
/mnt/HDD/snapshots/daily.0 instead of /mnt/HDD/A. In my understanding,
that would mean there's only one full copy of the data, and if the
data never changed, the backup machine would only take up as much size
as the sum of all machines which are sending backups.

Please advise if:-

1. The above is flawed and impossible as a concept.
2. rsyncs during the rsnapshot run would corrupt the backup
3. The sync_first option can do what I want

Thank you for your time.

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot without syncing
August 06, 2016 05:02PM
On Sat, Aug 6, 2016 at 6:37 PM, Oon-Ee Ng <ngoonee.talk < at > gmail.com> wrote:
[quote]Summary: I'd like to use rsnapshot only for rotation, is that
possible? Can sync_first do rotation only (including hard link
copying) WITHOUT ever running sync?
[/quote]
It's a modest perl script or two well designed to be extremely
flexible per user requirements.

[quote]Detailed Explanation:

I have a server with a larg-ish hard disk attached. Backups are done
from multiple machines (Linux using ssh+rsync, windows using cygwin
ssh+rsync, Android using sftp). The resulting folders for 2 machines
would look something like:-
/mnt/HDD/A
/mnt/HDD/B

I'm already using rsnapshot for some months, so I have:-
/mnt/HDD/snapshots/daily.0/A
/mnt/HDD/snapshots/daily.0/B
/mnt/HDD/snapshots/daily.1/A
/mnt/HDD/snapshots/daily.1/B
/mnt/HDD/snapshots/daily.2/A
/mnt/HDD/snapshots/daily.2/B
and so on, including weekly/monthly intervals.

There's actually two full copies of the data on this server. For
machine A, this is a copy at /mnt/HDD/A and another copy at
/mnt/HDD/snapshots/daily.0/A. I'd like to save some space, and imagine
I could perhaps do something like this:-

1. Tell rsnapshot to only rotate (mv daily.5->daily.6, mv daily.4->daily.5 ....)
2. Tell rsnapshot to do hard linking from daily.0 to daily.1
[/quote]
daily.0 and daily.1 should already be hardlinked. Any files that did
not differ between daily.0 and daily.1 should have been hardlinked
when daily.0 got generated.

[quote]My intention is that machine A would run a backup rsync to
/mnt/HDD/snapshots/daily.0 instead of /mnt/HDD/A. In my understanding,
[/quote]
What? Why? Why not do a cp -al from daily.0 first, then ran a
well-formed rsync command on top of that? This is asically wnat
daily.1 and daily.0 do anyway.

Do be careful with this approach. If someone then SCP's on top of
/mnt/HDD/A, they'll be copying on top of the "static" backups. This is
the risk of *any* approach writes on top of the rsnapshot hardlinked
backups, for any reason.

That risk is why few people do this. Anything that might write on top
of that staging area, even accidentally, could corrupt all the
backups.

[quote]that would mean there's only one full copy of the data, and if the
data never changed, the backup machine would only take up as much size
as the sum of all machines which are sending backups.
[/quote]
Plus inodes for all rhe subdirectories. I've seen people get bitten by that one!

[quote]Please advise if:-

1. The above is flawed and impossible as a concept.
[/quote]
Possible? Yes, it's a scripting environment and you can get away with
anything you can script. Good idea? I don't think so.

[quote]2. rsyncs during the rsnapshot run would corrupt the backup
[/quote]
[quote]3. The sync_first option can do what I want
[/quote]
I think this is actually your best bet.

[quote]
Thank you for your time.

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
[/quote]
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot without syncing
August 10, 2016 07:10PM
Thanks for your reply.

On Sun, Aug 7, 2016 at 8:00 AM, Nico Kadel-Garcia <nkadel < at > gmail.com> wrote:
[quote]On Sat, Aug 6, 2016 at 6:37 PM, Oon-Ee Ng <ngoonee.talk < at > gmail.com> wrote:
[quote]My intention is that machine A would run a backup rsync to
/mnt/HDD/snapshots/daily.0 instead of /mnt/HDD/A. In my understanding,
[/quote]
What? Why? Why not do a cp -al from daily.0 first, then ran a
well-formed rsync command on top of that? This is asically wnat
daily.1 and daily.0 do anyway.

Do be careful with this approach. If someone then SCP's on top of
/mnt/HDD/A, they'll be copying on top of the "static" backups. This is
the risk of *any* approach writes on top of the rsnapshot hardlinked
backups, for any reason.

That risk is why few people do this. Anything that might write on top
of that staging area, even accidentally, could corrupt all the
backups.
[/quote]
The 'why' is that I would like to save space, basically. More below.

[quote][quote]3. The sync_first option can do what I want
[/quote]
I think this is actually your best bet.
[/quote]
As I understand it, sync_first uses a .sync folder which would use up
as much space as the folder being backed up, something I'm trying to
avoid.

Here was my idea on how things should work:-

1. daily.1 is a hard-linked identical copy of daily.0
2. copying files into daily.0 disassociates that link, meaning daily.1
is not changed at all, while daily.0 has the 'latest' copy
3. running `rsnapshot daily` just rotates daily.1 to daily.2 and then
cp -al daily.0 daily.1

It appears, if I'm deducing from your replies correctly, that my '2'
is mistaken as changes to daily.0 would also affect daily.1 (and hence
all subsequent backups). Is this only a concern with scp? Can I avoid
this problem with a properly formed rsync, for example? I have full
control over all possible writes to the backup hard disc.

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. http://sdm.link/zohodev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot without syncing
August 11, 2016 02:58AM
On 11 August 2016 at 03:08, Oon-Ee Ng <ngoonee.talk < at > gmail.com ([email]ngoonee.talk < at > gmail.com[/email])> wrote:
[quote]> That risk is why few people do this. Anything that might write on top
[quote]of that staging area, even accidentally, could corrupt all the
backups.
[/quote]
The &#39;why&#39; is that I would like to save space, basically. More below.

[quote][quote]3. The sync_first option can do what I want
[/quote]
I think this is actually your best bet.
[/quote]
As I understand it, sync_first uses a .sync folder which would use up
as much space as the folder being backed up, something I&#39;m trying to
avoid.
[/quote]

Wrong. The .sync folder becomes the backup folder when the process is finished, and only files which have changed since the last backup will be physically copied, the rest being links. This is exactly what you want.

poc
rsnapshot without syncing
August 11, 2016 02:04PM
On Thu, Aug 11, 2016 at 5:56 PM, Patrick O'Callaghan
<pocallaghan < at > gmail.com> wrote:
[quote]On 11 August 2016 at 03:08, Oon-Ee Ng <ngoonee.talk < at > gmail.com> wrote:
[quote]As I understand it, sync_first uses a .sync folder which would use up
as much space as the folder being backed up, something I'm trying to
avoid.
[/quote]Wrong. The .sync folder becomes the backup folder when the process is
finished, and only files which have changed since the last backup will be
physically copied, the rest being links. This is exactly what you want.
[/quote]
Thanks! It appears then that I misunderstand the man page, which
states regarding sync_first that:-
This benefit comes at the cost of one more snapshot worth of disk
space. The default is 0 (off).

I understood that to mean .sync would take up as much disk space as a
full backup, but I guess the 'snapshot worth' is important there,
meaning it would be an additional number of hard links rather than
additional space taken up.

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. http://sdm.link/zohodev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot without syncing
August 11, 2016 03:52PM
On 11 August 2016 at 22:02, Oon-Ee Ng <ngoonee.talk < at > gmail.com ([email]ngoonee.talk < at > gmail.com[/email])> wrote:
[quote]On Thu, Aug 11, 2016 at 5:56 PM, Patrick O&#39;Callaghan
<pocallaghan < at > gmail.com ([email]pocallaghan < at > gmail.com[/email])> wrote:
[quote]On 11 August 2016 at 03:08, Oon-Ee Ng <ngoonee.talk < at > gmail.com ([email]ngoonee.talk < at > gmail.com[/email])> wrote:
[quote]As I understand it, sync_first uses a .sync folder which would use up
as much space as the folder being backed up, something I&#39;m trying to
avoid.
[/quote]Wrong. The .sync folder becomes the backup folder when the process is
finished, and only files which have changed since the last backup will be
physically copied, the rest being links. This is exactly what you want.
[/quote]
Thanks! It appears then that I misunderstand the man page, which
states regarding sync_first that:-
This benefit comes at the cost of one more snapshot worth of disk
space. The default is 0 (off).

I understood that to mean .sync would take up as much disk space as a
full backup, but I guess the &#39;snapshot worth&#39; is important there,
meaning it would be an additional number of hard links rather than
additional space taken up.

[/quote]

Plus any changed files of course, but essentially yes.

poc
rsnapshot without syncing
August 11, 2016 06:52PM
On Fri, Aug 12, 2016 at 6:49 AM, Patrick O'Callaghan
<pocallaghan < at > gmail.com> wrote:
[quote]On 11 August 2016 at 22:02, Oon-Ee Ng <ngoonee.talk < at > gmail.com> wrote:
[quote]I understood that to mean .sync would take up as much disk space as a
full backup, but I guess the 'snapshot worth' is important there,
meaning it would be an additional number of hard links rather than
additional space taken up.

[/quote]
Plus any changed files of course, but essentially yes.
[/quote]
Thanks, I'm going to test replacing the 'sync' step with remotely run
rsyncs and see how well that works.

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. http://sdm.link/zohodev2dev
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Sorry, only registered users may post in this forum.

Click here to login