Welcome! » Log In » Create A New Profile

rsync_long_args --delete alternative

Posted by r1100gspd 
rsync_long_args --delete alternative
June 12, 2015 05:46AM
A backup to an external hard drive recently filled up, and wasn't detected for some days.
I found that even after pregressively deleting ALL the Daily and Weekly backups I still couldn't run a sync because the disk was too full.  I suppose this is because with --delete set, files were attempting to add to the sync folder prior to any directories with deletable files were processed.

The backup data size was 3.3TB and the backup drive was 3.8TB, so there should be enough room for a normal backup.
I could have just deleted the sync folder and started again, but instead I set rsync_long_args
  from:  --delete
  to:      --delete-before. 

This then allowed the backup to complete successfully.

I think it is sensible to leave the rsync_long_args parameter as --delete-before, as it should help avoid out of disk space conditions.  But I am sure there is a reason why it is not the default.  How do others set the --delete paramter, and is there any harm in leaving it set to --delete-before?

- Gunter

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 12, 2015 08:53PM
On Fri, Jun 12, 2015 at 05:44:33AM -0700, r1100gspd < at > yahoo.com.au wrote:
[quote]A backup to an external hard drive recently filled up, and wasn't detected for some days.
[/quote]
[quote]The backup data size was 3.3TB and the backup drive was 3.8TB
[/quote]
My rule of thumb is the backup drive (containing snapshot_root) should
be at least twice as big as the data you are backing up.

I don't think rsnapshot will keep working for very long given the
ratio 3.8:3.3 (unless you are lucky with a combination like you have
very few files that change over time, no new large files added, and
not too many files).

The options I would suggest are
(1) double the size of your backup drive
(2) reduce your backup data set to about 2GB (eg backup a subset)
(3) monitor rsnapshot closely to see when it runs out of space, then
do things like deleting old backups, tweaking rsnapshot config)
(4) only try to keep a single snapshot/backup of your data (in which
case, you don't need rsnapshot).

You seem to be on the track of option (3). I would encourage you to
investigate (1) or (2) instead.

--
___________________________________________________________________________
David Keegel <djk < at > cyber.com.au> Cyber IT Solutions Pty. Ltd.
http://www.cyber.com.au/~djk/ Linux & Unix Systems Administration

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 14, 2015 01:47AM
--------------------------------------------
On Sat, 13/6/15, David Keegel <djk < at > cyber.com.au> wrote:

Subject: Re: [rsnapshot-discuss] rsync_long_args --delete alternative
To: r1100gspd < at > yahoo.com.au
Cc: rsnapshot-discuss < at > lists.sourceforge.net
Received: Saturday, 13 June, 2015, 11:34 AM

On Fri, Jun 12, 2015 at
05:44:33AM -0700, r1100gspd < at > yahoo.com.au
wrote:
[quote]A backup to an external hard drive
[/quote]recently filled up, and wasn't detected for some
days.

[quote]The backup data
[/quote]size was 3.3TB and the backup drive was 3.8TB

My rule of thumb is the backup
drive (containing snapshot_root) should
be
at least twice as big as the data you are backing up.

I agree, but in this case the boss is holding off spending for a few more months.

I don't think rsnapshot
will keep working for very long given the
ratio 3.8:3.3 (unless you are lucky with a
combination like you have
very few files
that change over time, no new large files added, and
not too many files).

+1

The options I would suggest are
(1) double the size of your backup drive
(2) reduce your backup data set to about 2GB
(eg backup a subset)
I have configured some exclusions already, but I may have to push for more.

My particular concern was of using the --delete-before switch, because to me it seem a more sensible option, but I though there must be a good reason why it's not the default, especially with this set also:
sync_first 1

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 14, 2015 01:59AM
--------------------------------------------
On Fri, 12/6/15, Nico Kadel-Garcia <nkadel < at > gmail.com> wrote:

[quote]I think it is sensible to leave the rsync_long_args parameter as
[/quote]--delete-before, as it should help avoid out of disk space
conditions.  But I am sure there is a reason why it is not
the default.  How do others set the --delete paramter, and
is there any harm in leaving it set to --delete-before?

I believe that rsync option came after rsnapshot was written. I don't see any issue with using it for rsnapshot,
which is active in a new directory anyway.

-----------------------------------------------

By "active in a new directory" do you mean the backup destination?

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 14, 2015 05:51AM
On Sun, Jun 14, 2015 at 4:57 AM, <r1100gspd < at > yahoo.com.au> wrote:
[quote]
--------------------------------------------
On Fri, 12/6/15, Nico Kadel-Garcia <nkadel < at > gmail.com> wrote:

[quote]I think it is sensible to leave the rsync_long_args parameter as
[/quote]--delete-before, as it should help avoid out of disk space
conditions. But I am sure there is a reason why it is not
the default. How do others set the --delete paramter, and
is there any harm in leaving it set to --delete-before?

I believe that rsync option came after rsnapshot was written. I don't see any issue with using it for rsnapshot,
which is active in a new directory anyway.
[/quote]
Yeah, I had to think about this one. Since, in most configurations,
all the files in the new rsnapshot backup directory are all still
hardlinked to the last snapshot, any benefit from a "--delete-before"
seems quite modest. The files will remain available, occupying space
on the backup filesystem, until the oldest snapshot is expired.

But it has me wondering: is there a lot of *directory namespace* churn
in the new snapshots? Directories take one inode each, whether or not
they have any files and whether the files are hardlinked. I'm now
wondering if the "out of space" errors are actually due to running out
of inodes, inodes which are accumulated in a cluttered directory space
until the '--delete' part of rsync occurs?

A "df -i' on the backup filesystem might be enlightening.

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 14, 2015 09:07AM
--------------------------------------------
On Sun, 14/6/15, Nico Kadel-Garcia <nkadel < at > gmail.com> wrote:

Subject: Re: [rsnapshot-discuss] rsync_long_args --delete alternative
To: r1100gspd < at > yahoo.com.au
Cc: "rsnapshot discuss" <rsnapshot-discuss < at > lists.sourceforge.net>
Received: Sunday, 14 June, 2015, 8:48 PM

On Sun, Jun 14, 2015 at
4:57 AM,  <r1100gspd < at > yahoo.com.au>
wrote:
--------------------------------------------
[quote]On Fri, 12/6/15, Nico Kadel-Garcia <nkadel < at > gmail.com>
[/quote]wrote:
[quote]
  > I
[/quote]think it is sensible to leave the rsync_long_args parameter
as
[quote]  --delete-before, as it should help avoid out of disk space
  conditions. 
How do others set the --delete paramter, and
  is there any harm in leaving it set to --delete-before?
  >

  I believe that
[/quote]rsync option came after rsnapshot was written. I don't
see any issue with using it for rsnapshot,
[quote]  which is active in a new directory
[/quote]anyway.

Yeah, I had
to think about this one. Since, in most configurations,
all the files in the new rsnapshot backup
directory are all still
hardlinked to the
last snapshot, any benefit from a
"--delete-before"
seems quite
modest. The files will remain available, occupying space
on the backup filesystem, until the oldest
snapshot is expired.

--------------------------------------------
I think you are right. The churn in between backups is the amount that would be deleted first if --delete-before was enabled. In my case we use 3 USB backup drives in rotation, which amplifies the churn, especially because they are not strictly rotated on a daily basis. Often it has been a week or more before Disk #1 is used again. In this case, if the churn is say 20GB per day, 7 backups could be easily > 140GB (thought I have seen some of around >300GB) So --delete-before can only be a good thing to prevent disk full issues when we are pushing the upper limits of the backup disk capacity, therefore I'll stick with it for now.

For backups with very low churn, especially with a single backup drive, then it's use would be far less noticeable.

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 15, 2015 05:58AM
On Fri, Jun 12, 2015 at 05:44:33AM -0700, r1100gspd < at > yahoo.com.au wrote:

[quote]I set rsync_long_args
  from:  --delete
  to:      --delete-before. 

This then allowed the backup to complete successfully.
[/quote]
--delete-before is rsync's default if you're --deleteing at all.

--
David Cantrell | Nth greatest programmer in the world

We decided that in a world of 8 string guitars, 12 string basses
and drumkits from hell "shredding" in weird scat-jazz time
signatures that we wanted a stripped down song with lots of
shouting, and where the verse and chorus are the same chords.

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 16, 2015 12:40AM
--------------------------------------------
On Mon, 15/6/15, David Cantrell <david < at > cantrell.org.uk> wrote:

Subject: Re: [rsnapshot-discuss] rsync_long_args --delete alternative
To: rsnapshot-discuss < at > lists.sourceforge.net
Received: Monday, 15 June, 2015, 8:42 PM

On Fri, Jun 12, 2015 at
05:44:33AM -0700, r1100gspd < at > yahoo.com.au
wrote:

[quote]I set rsync_long_args
  from:  --delete
  to:      --delete-before. 

This then allowed the backup to complete successfully.
[/quote]
--delete-before is rsync's default if you're --deleteing at all.

--
David Cantrell | Nth
greatest programmer in the world

--------------------------------------------

That may be the case for plain rsync, but for my rsnapshot setup, my backup would not complete until I instigated the --delete-before setting.
--
Gunter

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 16, 2015 07:44AM
On 06/16/2015 12:39 AM, r1100gspd < at > yahoo.com.au wrote:
[quote]On Mon, 15/6/15, David Cantrell <david < at > cantrell.org.uk> wrote:
[quote]--delete-before is rsync's default if you're --deleteing at all.
[/quote]
That may be the case for plain rsync, but for my rsnapshot setup, my
backup would not complete until I instigated the --delete-before
setting.
[/quote]
--delete-during is actually the default.

--delete-before requires rsync to walk the entire filesystem before
transferring data, like it did before 3.0. That requires a lot more memory.

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 17, 2015 02:13AM
On Tue, 16 Jun 2015 07:42:22 Gordon Messmer wrote:
[quote]On 06/16/2015 12:39 AM, r1100gspd < at > yahoo.com.au wrote:
[quote]On Mon, 15/6/15, David Cantrell <david < at > cantrell.org.uk> wrote:
[quote]--delete-before is rsync's default if you're --deleteing at all.
[/quote]
That may be the case for plain rsync, but for my rsnapshot setup, my
backup would not complete until I instigated the --delete-before
setting.
[/quote]
--delete-during is actually the default.

--delete-before requires rsync to walk the entire filesystem before
transferring data, like it did before 3.0. That requires a lot more memory.

------------------------------------------------------------------------------
[/quote]

Good point Gordon! For us we have around 3.4TB of files to backup. Do you know of a way to measure the maximum memory usage of rsnapshot?

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 17, 2015 04:35AM
On Tue, Jun 16, 2015 at 12:39:04AM -0700, r1100gspd < at > yahoo.com.au wrote:
[quote]On Fri, Jun 12, 2015 at
05:44:33AM -0700, r1100gspd < at > yahoo.com.au
wrote:
[quote]I set rsync_long_args
  from:  --delete
  to:      --delete-before. 

This then allowed the backup to complete successfully.
[/quote]--delete-before is rsync's default if you're --deleteing at all.
That may be the case for plain rsync, but for my rsnapshot setup, my backup would not complete until I instigated the --delete-before setting.
[/quote]
rsnapshot is really just a fancy wrapper around rsync, so I can only
assume that you've got a --delete-after or similar either in your
rsnapshot config or in some system-wide rsync config thingy.

--
David Cantrell | Reality Engineer, Ministry of Information

The Law of Daves: in any gathering of technical people, the
number of Daves will be greater than the number of women.

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 17, 2015 04:48AM
On Tue, Jun 16, 2015 at 07:42:22AM -0700, Gordon Messmer wrote:
[quote]On 06/16/2015 12:39 AM, r1100gspd < at > yahoo.com.au wrote:
[quote]On Mon, 15/6/15, David Cantrell <david < at > cantrell.org.uk> wrote:
[quote]--delete-before is rsync's default if you're --deleteing at all.
[/quote]
That may be the case for plain rsync, but for my rsnapshot setup, my
backup would not complete until I instigated the --delete-before
setting.
[/quote]
--delete-during is actually the default.
[/quote]
$ tree foo bar
foo
└── baz
bar
└── barf

$ rsync -av --delete foo/ bar/
building file list ... done
deleting barf
./
baz

[quote]--delete-before requires rsync to walk the entire filesystem before
transferring data, like it did before 3.0. That requires a lot more memory.
[/quote]
Ahh, maybe the default changed in version 3.

Perhaps in the interests of consistency we should put --delete-during in
the example rsnapshot config.

--
David Cantrell | Minister for Arbitrary Justice

I think the most difficult moment that anyone could face is seeing
their domestic servants, whether maid or drivers, run away
-- Abdul Rahman Al-Sheikh, writing on 25 Jan 2004 at
http://archive.arabnews.com/?article=38558

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 17, 2015 05:03AM
On Wed, Jun 17, 2015 at 11:51:26AM +0800, gordonm < at > zanotech.com.au wrote:

[quote]we have around 3.4TB of files to backup. Do you know of a way to measure the maximum memory usage of rsnapshot?
[/quote]
/usr/bin/time -v

rsnapshot itself won't use much memory. I ASSume that that will also
include memory used by child processes (which is what you're really
interested in), but I haven't checked.

valgrind can also do it, but at the cost of, well, being valgrind :-)

It's the number of files that matters, BTW, not the amount of data. 3TB
of videos will make rsync consume very little memory. 3TB of maildirs on
the other hand ... ugh.

--
David Cantrell | Enforcer, South London Linguistic Massive

Irregular English:
you have anecdotes; they have data; I have proof

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsync_long_args --delete alternative
June 17, 2015 07:51AM
On 06/16/2015 08:51 PM, gordonm < at > zanotech.com.au wrote:
[quote]Good point Gordon! For us we have around 3.4TB of files to backup.
Do you know of a way to measure the maximum memory usage of
rsnapshot?
[/quote]
You could always run 'top' and watch memory use. I'm pretty sure that a
dry-run (-n) will use nearly the same amount of memory.

The FAQ says that rsync used to use about 100 bytes per file, plus some
vague "more" with -H and --delete. If you get a file count, you can
probably get a rough estimate that way.

https://rsync.samba.org/FAQ.html

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Sorry, only registered users may post in this forum.

Click here to login