Backup Central

Hello

a simple question:

(How) Is it possible to make "logarithmic backups"?
I only se a way to delete "all older" backups,
but no way to "migrate" and keep -some- old backups.

I don't know the right term, so in other words:

backups should run "unattended"
Every hour a shot is done.
That hourly shots should be kept only for 48 hour
At the third day
one of those 24 shots (say one of 23:00 ) should
be kept as "day 0..7", all other "hourly" should be deleted.
After a week fridays shot should be kept as "week 1",
while all other "days" older than 7 days should be deleted.
After 4/5 weeks the last "week" of the previous Months
should be renamed to "month x", and all other deleted.
After 12 month the last backup of december should become "Year xxx"
After 5 years all older year backups should be deleted.

So when the System has run a longer time:

xx hourly backups of the current day (x depending on current time)
24 hourly backups of yesterday
x daily backups of the current week (x depending on current day of week)
7 daily backups of the previous week
x weekly backups of the current month (x depending on current week of month)
4/5 weekly backups of the previous month
x montly backups of the year (x depending on current month)
12 montly backups of previous year
5 yearly backups
__
giving 5+12+11+5+4+7+6+24+23= max. 97 backup sets
and never more. (Wortscase the "backups" would
need 100 time the original data space. But this
would only happen if every hourly shot finds all files
as entirely new.


Can this be done with rdiff-backup ?
Or what tool can be used?







Rainer

begin quotation of Rainer Zocholl on 2004-12-12 22:17:00 +0100:

Can this be done with rdiff-backup ?

rdiff-backup has no concept of backup levels and cannot selectively
purge revisions. If you want to emulate backup levels with
rdiff-backup, you're going to need a lot more space and use a separate
backup location for each level, which more or less negates any benefit
gained from using levels.

I'm not sure that implementing backup levels in rdiff-backup would
provide a huge space savings - rdiff-backup just stores the difference
between two revisions, and if the file doesn't change much, no big
deal, small diff.

and it's probably worth noting, just for clarity, that the reason that
rdiff-backup couldn't have this functionality is because of the nature
of incremental backups. since each backup is based on a diff from the
most recent one before it, they are like a chain. if you break one link
(delete any file in the middle) your chain is useless.

Alec Berryman wrote:
begin quotation of Rainer Zocholl on 2004-12-12 22:17:00 +0100:


Can this be done with rdiff-backup ?


rdiff-backup has no concept of backup levels and cannot selectively
purge revisions. If you want to emulate backup levels with
rdiff-backup, you're going to need a lot more space and use a separate
backup location for each level, which more or less negates any benefit
gained from using levels.

I'm not sure that implementing backup levels in rdiff-backup would
provide a huge space savings - rdiff-backup just stores the difference
between two revisions, and if the file doesn't change much, no big
deal, small diff.


------------------------------------------------------------------------

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

sburton < at > allafrica.com(Sherrard Burton) 12.12.04 18:20

Alec Berryman wrote:
begin quotation of Rainer Zocholl on 2004-12-12 22:17:00 +0100:


Can this be done with rdiff-backup ?


rdiff-backup has no concept of backup levels and cannot selectively
purge revisions. If you want to emulate backup levels with
rdiff-backup, you're going to need a lot more space and use a
separate backup location for each level, which more or less negates
any benefit gained from using levels.

I'm not sure that implementing backup levels in rdiff-backup would
provide a huge space savings - rdiff-backup just stores the
difference between two revisions, and if the file doesn't change
much, no big deal, small diff.

Yes. But if it changes a lot over a longer time interval
the (meantime unneeded) diffs may eat more space than
a "full" backup.

It's not very likely that i need the archived version exactly
1999, 11 Juli 9:00. Most likely i'll be happy with the version
31 Dezember 1999 from the "archive".

and it's probably worth noting, just for clarity, that the reason that
rdiff-backup couldn't have this functionality is because of the nature
of incremental backups. since each backup is based on a diff from the
most recent one before it, they are like a chain.

Yepp.

if you break one link (delete any file in the middle)
your chain is useless.

Except i could use some kind of "back annotation"/"renumbering"
that allows to delete steps between (by loosing the
ability to convert to that versions, of course).



Rainer---<=====> Vertraulich
//
//
<=====>--------------ocholl, Kiel, Germany ------------

begin quotation of Rainer Zocholl on 2004-12-13 01:47:00 +0100:

and it's probably worth noting, just for clarity, that the reason that
rdiff-backup couldn't have this functionality is because of the nature
of incremental backups. since each backup is based on a diff from the
most recent one before it, they are like a chain.

Yepp.

if you break one link (delete any file in the middle)
your chain is useless.

Except i could use some kind of "back annotation"/"renumbering"
that allows to delete steps between (by loosing the
ability to convert to that versions, of course).

Yes; you could reconstruct the file at two points, take a diff, and
use that to replace all of the diffs between the two points.

No one's stepped up with code yet.

alec < at > thened.net(Alec Berryman) 12.12.04 20:11

begin quotation of Rainer Zocholl on 2004-12-13 01:47:00 +0100:

and it's probably worth noting, just for clarity, that the reason
that rdiff-backup couldn't have this functionality is because of the
nature of incremental backups. since each backup is based on a diff
from the most recent one before it, they are like a chain.

Yepp.

if you break one link (delete any file in the middle)
your chain is useless.

Except i could use some kind of "back annotation"/"renumbering"
that allows to delete steps between (by loosing the
ability to convert to that versions, of course).

Yes; you could reconstruct the file at two points, take a diff, and
use that to replace all of the diffs between the two points.

No one's stepped up with code yet.

Thanks for info.

So i have to go to amanda, bacula or so? ;-(

begin quotation of Rainer Zocholl on 2004-12-13 16:25:00 +0100:

So i have to go to amanda, bacula or so? ;-(

You may not need something that heavy; I've heard hdup implements
simplified backup levels, although I've never used it myself.

On 12 Dec 2004, Rainer Zocholl wrote:

backups should run "unattended"
Every hour a shot is done.
That hourly shots should be kept only for 48 hour
At the third day
one of those 24 shots (say one of 23:00 ) should
be kept as "day 0..7", all other "hourly" should be deleted.
After a week fridays shot should be kept as "week 1",
while all other "days" older than 7 days should be deleted.
After 4/5 weeks the last "week" of the previous Months
should be renamed to "month x", and all other deleted.
After 12 month the last backup of december should become "Year xxx"
After 5 years all older year backups should be deleted.

dirvish does this. Dirvish has many similiarities with rdiff-backup but I
found rdiff-backup easier to work with (though Dirvish and faubackup were
close seconds).

I think faubackup can also do this.

It would indeed be great if rdiff-backup had this.

- Serge Wroclawski

On Sun, 12 Dec 2004, Alec Berryman wrote:

rdiff-backup has no concept of backup levels and cannot selectively
purge revisions. If you want to emulate backup levels with
rdiff-backup, you're going to need a lot more space and use a separate
backup location for each level, which more or less negates any benefit
gained from using levels.

Not really. The way the other programs I mentioned (dirvish and
faubackup) handle it is they maintain a full copy of the data from the
backup host. They then use hard links between files when no change is
made. Changed files are made from scratch and then, if they don't changed,
are linked to, so very little space is lost.

Keeping specific versions then becomes trivial, since every version is a
full.

- Serge Wroclawski

serge < at > tux.org(Serge Wroclawski) 13.12.04 11:47


On Sun, 12 Dec 2004, Alec Berryman wrote:

rdiff-backup has no concept of backup levels and cannot selectively
purge revisions. If you want to emulate backup levels with
rdiff-backup, you're going to need a lot more space and use a
separate backup location for each level, which more or less negates
any benefit gained from using levels.

Not really. The way the other programs I mentioned (dirvish and
faubackup) handle it is they maintain a full copy of the data from the
backup host. They then use hard links between files when no change is
made. Changed files are made from scratch and then, if they don't
changed, are linked to, so very little space is lost.

Sounds good.
"Ideally" something "inside" the filesystem (on inode level?)
would be the most efficient for this purpose.
But of cource that can't replace a "backup" as it only
protects the data against its worst enemies, the users, not
against a disk crash or roots "#rm -rf"
But there currently nothing visible in linux GPL world, or?

Keeping specific versions then becomes trivial,
since every version is a full.

The disadvantage (and the advantage of rdif-backup) seems to be
that you run into trouble with the "meta informations" like
user IDs...

Another disadvantage (and the bigest advantage of rdiff-backup)
are the much more complicated setup.
"rdiff-backup" is really clear and easy to use.
Maybe i expect too much, as this might lead to more "complicate"
programm...
There is no free lunch.




Rainer---<=====> Vertraulich
//
//
<=====>--------------ocholl, Kiel, Germany ------------

serge < at > tux.org(Serge Wroclawski) 13.12.04 11:44


On 12 Dec 2004, Rainer Zocholl wrote:

backups should run "unattended"
Every hour a shot is done.
That hourly shots should be kept only for 48 hour
At the third day
one of those 24 shots (say one of 23:00 ) should
be kept as "day 0..7", all other "hourly" should be deleted.
After a week fridays shot should be kept as "week 1",
while all other "days" older than 7 days should be deleted.
After 4/5 weeks the last "week" of the previous Months
should be renamed to "month x", and all other deleted.
After 12 month the last backup of december should become "Year xxx"
After 5 years all older year backups should be deleted.

dirvish does this. Dirvish has many similiarities with rdiff-backup
but I found rdiff-backup easier to work with (though Dirvish and
faubackup were close seconds).

I think faubackup can also do this.

Yes, faubackup can do this and is very easy to configure an
use.
Simply setup the backup destion in teh conf and do a manual
back up.
It has no problems if the source is a symbolic nor if
the destination does not exists.
(It's very inconvient solved in rdiff-backup when source is a symlink
it only gives an error like "directory does not exist").

I currently don't know if faubackup can link parts of files or
only entire files.

If can do only entire files it would be very inefficient on database
files, which are very big and change only some few bytes inside
without moving any other. That's what rdiff-backup can handle better, or?


dirvish has the charming feature that's pure perl and seems
to be more effecient over network as it uses rsync, with
allows to be used for non-unix clients too.



It would indeed be great if rdiff-backup had this.

ACK.


Thanks for the links (faubackup was entire new to me and seems
to be the right tool, as it's available as a debian pkg too.
BTW: It's renamed from "vbackup".)

begin quotation of Rainer Zocholl on 2004-12-14 01:01:00 +0100:

dirvish has the charming feature that's pure perl and seems
to be more effecient over network as it uses rsync, with
allows to be used for non-unix clients too.

You're ill-informed.

Python is no less portable than Perl. rdiff-backup can run on Windows
(+cygwin), as it says on the front page. I'd guess that it runs
without cygwin as long as SFU is installed, but I've never tested
that.

rdiff-backup uses rdiff from the librsync project - I doubt there's a
discernible difference in efficiency.

alec < at > thened.net(Alec Berryman) 13.12.04 21:35

Once upon a time "Alec Berryman " shaped the electrons to say...

begin quotation of Rainer Zocholl on 2004-12-14 01:01:00 +0100:

dirvish has the charming feature that's pure perl and seems
to be more effecient over network as it uses rsync, with
allows to be used for non-unix clients too.

You're ill-informed.

Python is no less portable than Perl.

Yes, but not so widely already installed on the systems
i am think of.

rdiff-backup can run on Windows
(+cygwin), as it says on the front page.

cygwin is not native windows ;-)

I'd guess that it runs without cygwin as long as SFU

Ah. interessting. What's SFU?

is installed, but I've never tested that.


rdiff-backup uses rdiff from the librsync project
- I doubt there's a discernible difference in efficiency.

Yepp. i wanted to compare it with "faubackup" not rdiff. It
does not use rsync, at least if i read it's "todo" list right.

Thanks for clarifying this.

Rainer

Alec Berryman wrote:
Python is no less portable than Perl. rdiff-backup can run on Windows
(+cygwin), as it says on the front page. I'd guess that it runs
without cygwin as long as SFU is installed, but I've never tested
that.

I doubt it does. rdiff-backup librsync functions are very unix like and
require the cgywin way of handling things. SFU doesn't provide anywhere
near enough functionality for it to run. I haven't actually tried it
either, but I have tried to get rdiff-backup to run on native python.
The librsync module just won't compile without a complete rewrite that
takes out all the unixisms.
btw, if you want rdiff-backup packages for windows you can get them here:

http://sol1.net/~dave/backup/

rdiff-backup is win32->unix only at this stage (unix<->unix) is fine.
quite possible we could make a win32 compatible version of rdiff-backup,
but I don't have the resources for that yet.
dave