Backup Central

Hi everyone,


I'm running rsnapshot under cygwin, scheduled by cron... Now, my
computer is not always on and our backups seem to take a quite long (
3 hours ) time: we are backing up a NAS (approx 500GB worth of data)
to a USB disk (approx 2TB) attached to my machine.

Is there any easy way to prevent (weekly/monthly/daily) backups from
overlapping without reverting to a 'handwaving' approach like "space
them in time as far apart as possible"?

Thx a lot for any hints!


- Bram

--
Bram de Jong - CTO
SampleSumo BVBA, Wiedauwkaai 23 G, B-9000 Ghent, Belgium
Web: http://www.samplesumo.com
Twitter: http://twitter.com/SampleSumo
Facebook: http://facebook.com/SampleSumo
Phone: +32 9 3355925 - Mobile: +32 484 154730

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

My rsnapshot.conf sets a lockfile, like this:


# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile /var/run/rsnapshot.pid



On Thu, Mar 29, 2012 at 11:30:53AM +0200, Bram de Jong wrote:
Hi everyone,


I'm running rsnapshot under cygwin, scheduled by cron... Now, my
computer is not always on and our backups seem to take a quite long (
3 hours ) time: we are backing up a NAS (approx 500GB worth of data)
to a USB disk (approx 2TB) attached to my machine.

Is there any easy way to prevent (weekly/monthly/daily) backups from
overlapping without reverting to a 'handwaving' approach like "space
them in time as far apart as possible"?

Thx a lot for any hints!


- Bram

--
Bram de Jong - CTO
SampleSumo BVBA, Wiedauwkaai 23 G, B-9000 Ghent, Belgium
Web: http://www.samplesumo.com
Twitter: http://twitter.com/SampleSumo
Facebook: http://facebook.com/SampleSumo
Phone: +32 9 3355925 - Mobile: +32 484 154730

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

--
___________________________________________________________________________
David Keegel <djk < at > cyber.com.au> Cyber IT Solutions Pty. Ltd.
http://www.cyber.com.au/~djk/ Linux & Unix Systems Administration


------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, Mar 29, 2012 at 12:52 PM, David Keegel <djk < at > cyber.com.au> wrote:
My rsnapshot.conf sets a lockfile, like this:

# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile        /var/run/rsnapshot.pid

But say I have a daily scheduled at 14:00 and a weekly scheduled at
16:00, this could mean that -of the backup takes longer than 2 hours-
the weekly might never be executed.

So, that's not really a good option in my case...


- bram

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Bram,

Du meintest am 29.03.12:

I'm running rsnapshot under cygwin, scheduled by cron... Now, my
computer is not always on and our backups seem to take a quite long (
3 hours ) time: we are backing up a NAS (approx 500GB worth of data)
to a USB disk (approx 2TB) attached to my machine.

Is there any easy way to prevent (weekly/monthly/daily) backups from
overlapping without reverting to a 'handwaving' approach like "space
them in time as far apart as possible"?

Only the very new backup needs much time. On your system that may be
"daily.0".

All other activities are much faster, most times it's only renaming.

Ok - deleting may also need some time, but it doesn't lock a new backup
run.

On my machines I sometimes run a special job in the backup directory:

rm -rf _delete.*

It may be necessary when the machine which runs rsnapshot has been shut
down too early.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Bram,

Du meintest am 29.03.12:

lockfile        /var/run/rsnapshot.pid

But say I have a daily scheduled at 14:00 and a weekly scheduled at
16:00, this could mean that -of the backup takes longer than 2 hours-
the weekly might never be executed.

It's a very good idea to run (if it exists) first the "yearly" job, then
the "monthly", then the "weekly", then the "daily" and then (perhaps)
the "hourly" job.

Study it using the log file ...

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, Mar 29, 2012 at 1:07 PM, Helmut Hullen <Hullen < at > t-online.de> wrote:
Only the very new backup needs much time. On your system that may be
"daily.0".

All other activities are much faster, most times it's only renaming.

I know the theory, but in practice all my snapshots seem to take a
LONG time. I suppose this could have to do with the fact that I am
backing up from a NAS to a USB disk. I.e. relatively slow NAS ->
(network) -> my computer running cygwin -> USB disk. Also, many if not
all the files on the nas are large ( > 500MB ), I don't know if this
changes anything...

As an example: the current one was started at 12:00 and it's now 13:30
and it's still running. It's not running verbose it's hard to see
what's going on right now.
I siwtched on the loggin in the rsnapshot log, let's see what that
gives tomorrow...


- bram

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo Bram de Jong,

Am Donnerstag, 29. März 2012 13:09 schrieb Bram de Jong:
On Thu, Mar 29, 2012 at 12:52 PM, David Keegel <djk < at > cyber.com.au>
wrote:
My rsnapshot.conf sets a lockfile, like this:

# If enabled, rsnapshot will write a lockfile to prevent two
instances # from running simultaneously (and messing up the
snapshot_root). # If you enable this, make sure the lockfile
directory is not world # writable. Otherwise anyone can prevent
the program from running. #
lockfile        /var/run/rsnapshot.pid

But say I have a daily scheduled at 14:00 and a weekly scheduled
at 16:00, this could mean that -of the backup takes longer than 2
hours- the weekly might never be executed.


You schould do it the other way round...

man rsnapshot says:

It is usually a good idea to schedule the larger backup
levels to run a bit before the lower ones. For example, in
the crontab above, notice that "daily" runs 10 minutes
before "hourly". The main reason for this is that the daily
rotate will pull out the oldest hourly and make that the
youngest daily (which means that the next hourly rotate
will not need to delete the oldest hourly), which is more
efficient. A secondary reason is that it is harder to
predict how long the lowest backup level will take, since
it needs to actually do an rsync of the source as well as
the rotate that all backups do.



--
Herzliche Grüße!
Rolf Muth
Meine Adressen duerfen nicht fuer Werbung verwendet werden!
PGP Public Key:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xF8DC41935544C89A
fingerprint: C025 3071 8E56 F8F1 250A 5624 F8DC 4193 5544 C89A

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, Mar 29, 2012 at 1:29 PM, Rolf Muth <rolf.muth < at > web.de> wrote:

You schould do it the other way round...
<snip>

OK, but this really isn't the issue yet because right now I'm ONLY
running dailies right now...

My question is: if a daily takes an unknown large amount of timeis it
better to for example run ONLY one type of backup per day if you have
limited time?

Example: if my daily takes 5 hours to complete and I only have
(approx) 8 hours per day I can run stuff on, imagine trying to run a
daily, weekly and monthly on the same day (if the first of the week
happens to be the same day as the first of the month)...

Or does running the higher levels first "fix" this by assuming: ok, if
I only have time for one backup, let's perform the one with the
highest "level" (i.e. lowest rate - preferring monthly to daily).


- bram

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, Mar 29, 2012 at 01:07:00PM +0200, Helmut Hullen wrote:

Only the very new backup needs much time. On your system that may be
"daily.0".

All other activities are much faster, most times it's only renaming.

Ok - deleting may also need some time, but it doesn't lock a new backup
run.

Deletes are always done as part of rotation, and they're always done
last. Rotation and backup need an exclusive lock, maintained by the
lockfile that DK mentioned. Deletion however does not need a lock and
can be deferred to run later. See the use_lazy_deletes option.

If you turn that on, then directories to be deleted are renamed so as to
not be in the way, the rotation carries on as normal, the lock is
released, and then the delete happens. If you turn this on, then
rotation is very fast.

Here's some snippets from my logfile. The D or M at the beggining
indicates which rsnapshot job the line comes from ...

M [01/Jan/2012:23:30:02] /usr/local/bin/rsnapshot monthly: started
M [01/Jan/2012:23:30:02] echo 83228 > /var/run/rsnapshot.pid # get lock
M [ a bunch of 'mv's ]
M [01/Jan/2012:23:30:02] rm -f /var/run/rsnapshot.pid # release lock
M [01/Jan/2012:23:30:02] /bin/rm -rf ... # start the delete
D [02/Jan/2012:00:00:04] /usr/local/bin/rsnapshot daily: started
D [02/Jan/2012:00:00:04] echo 83352 > /var/run/rsnapshot.pid # get lock
D [ a bunch of 'mv's, rsyncs etc ]
M [02/Jan/2012:00:07:34] /usr/local/bin/rsnapshot monthly: completed successfully
D [ some more rsyncs from 'rsnapshot daily' ]
D ...
D [02/Jan/2012:04:16:54] /usr/local/bin/rsnapshot daily: completed successfully

As you can see, 'rsnapshot monthly', which just handles rotates,
maintains a lock for under a second. Even though it's still deleting
the oldest monthly backup when the 'rsnapshot daily' starts, that's just
fine.

In summary, your cron jobs should run backup levels starting with the
least frequent going through to the most frequent, with short times
between them, and provided you use_lazy_deletes you'll be fine. My
crontab looks like this ...

0 0 * * * /usr/local/bin/rsnapshot daily
45 23 * * 6 /usr/local/bin/rsnapshot weekly
30 23 1 * * /usr/local/bin/rsnapshot monthly

So on those occasions when I run a daily, a weekly and a monthly on the
same day they're 15 minutes apart. 5 minutes, or even 1 minute would be
enough.

--
David Cantrell | http://www.cantrell.org.uk/david

It wouldn't hurt to think like a serial killer every so often.
Purely for purposes of prevention, of course.

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Bram,

Du meintest am 29.03.12:

All other activities are much faster, most times it's only renaming.

I know the theory, but in practice all my snapshots seem to take a
LONG time. I suppose this could have to do with the fact that I am
backing up from a NAS to a USB disk. I.e. relatively slow NAS ->
(network) -> my computer running cygwin -> USB disk. Also, many if
not all the files on the nas are large ( > 500MB ), I don't know if
this changes anything...

You should change something ...

I've just taken a look onto a machine (using "lazy_delete") with an USB
HD. It needs about 3 minutes for backing up about 200 GByte and about
3.5 minutes for deleting an old backup.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Bram,

Du meintest am 29.03.12:

My question is: if a daily takes an unknown large amount of time is it
better to for example run ONLY one type of backup per day if you have
limited time?

"That depends" ...

Example: if my daily takes 5 hours to complete and I only have
(approx) 8 hours per day I can run stuff on, imagine trying to run a
daily, weekly and monthly on the same day (if the first of the week
happens to be the same day as the first of the month)...

I just repeat my proposal: first "monthly", then "weekly", and finally
"daily".

Combined with "lazy_delete" the "monthly" and "weekly" jobs shouldn't
block the next rsnapshot run for a long time.

Please try it and study the log file.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, David,

Du meintest am 29.03.12:

Ok - deleting may also need some time, but it doesn't lock a new
backup run.

Deletes are always done as part of rotation, and they're always done
last.

And that works fine, mostly.

But when the system is shut down too early then there is some remainder.

Rotation and backup need an exclusive lock, maintained by the
lockfile that DK mentioned. Deletion however does not need a lock
and can be deferred to run later. See the use_lazy_deletes option.

If you turn that on, then directories to be deleted are renamed so as
to not be in the way, the rotation carries on as normal, the lock is
released, and then the delete happens. If you turn this on, then
rotation is very fast.

Yes - that's my standard configuration.

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On Thu, Mar 29, 2012 at 7:09 AM, Bram de Jong <bram.dejong < at > samplesumo.com ([email]bram.dejong < at > samplesumo.com[/email])> wrote:
On Thu, Mar 29, 2012 at 12:52 PM, David Keegel <djk < at > cyber.com.au ([email]djk < at > cyber.com.au[/email])> wrote:
My rsnapshot.conf sets a lockfile, like this:

# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile        /var/run/rsnapshot.pid



But say I have a daily scheduled at 14:00 and a weekly scheduled at
16:00, this could mean that -of the backup takes longer than 2 hours-
the weekly might never be executed.

So, that's not really a good option in my case...
 
Do the monthly, then the weekly, then the daily, in temporal order. Monthly and weekly take moments.
 
If the daily takes more than 24 hours, you may have an issue, but that's what cron scripts and checking time stamps are for. 
 
 

On Thu, Mar 29, 2012 at 3:31 PM, Nico Kadel-Garcia <nkadel < at > gmail.com> wrote:

Do the monthly, then the weekly, then the daily, in temporal order. Monthly
and weekly take moments.

If the daily takes more than 24 hours, you may have an issue, but that's
what cron scripts and checking time stamps are for.

OK, I've switched this on and will let you guys know what happens...


- bram

--
Bram de Jong - CTO
SampleSumo BVBA, Wiedauwkaai 23 G, B-9000 Ghent, Belgium
Web: http://www.samplesumo.com
Twitter: http://twitter.com/SampleSumo
Facebook: http://facebook.com/SampleSumo
Phone: +32 9 3355925 - Mobile: +32 484 154730

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Hallo, Bram,

Du meintest am 29.03.12:

I just switched on lazy-delete and logging, we'll see what happens.
The backup finished a while ago, so I have a runtime of approx 4
hours. Perhaps you are backing up from a local directory and not to a
remote machine?

Backing up over the LAN (100 MBit/s and 1 GBit/s): 150 GByte need about
10 ... 15 minutes.

In a misconfigured net that can need much more time ...

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss