SearchFAQMemberlist Log in
Reply to topic Page 1 of 1
Filechange during backup
Author Message
Post Filechange during backup 
Hi,

from the rdiff-backup error policy:
"These are usually caused by a file being modified as rdiff-backup is
processing it. The error log calls these UpdateErrors.
When this type of error occurs, rdiff-backup will pretend that the file
is the same as it is in the destination directory. For an incremental
backup, this means that no new increment will be written, and the mirror
copy will not be updated."

Found this out when I tried to backup web- and maillogs. Any workaround
for it? Not counting rotating logs before backup or making a separate
scp command for logfiles. I would be happy with something that makes
rdiff-backup ignore filesize changing during backup and just copy the
file :)

Any help appreciated.

Regards
Olav Langeland

Post Filechange during backup 
Hi,

one good workaround if you can is to use and lvm snapshot

you take a snapshot and back that up then remove the snapshot

http://www.tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html

Neil

On Friday 07 January 2005 10:31, Olav Langeland wrote:
Hi,

from the rdiff-backup error policy:
"These are usually caused by a file being modified as rdiff-backup is
processing it. The error log calls these UpdateErrors.
When this type of error occurs, rdiff-backup will pretend that the file
is the same as it is in the destination directory. For an incremental
backup, this means that no new increment will be written, and the mirror
copy will not be updated."

Found this out when I tried to backup web- and maillogs. Any workaround
for it? Not counting rotating logs before backup or making a separate
scp command for logfiles. I would be happy with something that makes
rdiff-backup ignore filesize changing during backup and just copy the
file :)

Any help appreciated.

Regards
Olav Langeland


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL:
http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Post Filechange during backup 
Hi,

maybe I should give some more details. Backupserver is Linux with LVM,
clients are FreeBSD with just straight /var, /local etc. mounted dirs.
rdiff-backup version is v0.12.7. I was hoping to backup /var and include
all current logs in that backuprun.

-olav

-----Original Message-----
From: Neil [mailto:rdiff-backup-users < at > iamafreeman.com]
Sent: 7. januar 2005 11:56
To: rdiff-backup-users < at > nongnu.org
Cc: Olav Langeland
Subject: Re: [rdiff-backup-users] Filechange during backup

Hi,

one good workaround if you can is to use and lvm snapshot

you take a snapshot and back that up then remove the snapshot

http://www.tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html

Neil

On Friday 07 January 2005 10:31, Olav Langeland wrote:
Hi,

from the rdiff-backup error policy:
"These are usually caused by a file being modified as
rdiff-backup is
processing it. The error log calls these UpdateErrors.
When this type of error occurs, rdiff-backup will pretend
that the file
is the same as it is in the destination directory. For an
incremental
backup, this means that no new increment will be written,
and the mirror
copy will not be updated."

Found this out when I tried to backup web- and maillogs.
Any workaround
for it? Not counting rotating logs before backup or making
a separate
scp command for logfiles. I would be happy with something that makes
rdiff-backup ignore filesize changing during backup and
just copy the
file :)

Any help appreciated.

Regards
Olav Langeland


_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users < at > nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL:
http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


Post Filechange during backup 
Hi

So you would need something with LVM snapshot like capabilities on the FreeBSD
boxes?

Back to the problem
What are you trying to get from var?

perhaps rdiff-backup isn't the best tool for log files anyway unless you are
filtering them at some later point and want to keep before and after

could using a syslog server be a better solution for the changing logs?
maillogs will happily go that way, though it might give web (apache?)
problems if you have high load
could run syslog server on a box with lvm and have the snapshot you need

then rdiff-backup everything apart from the logs

that should reduce the problem, not remove it though

Neil
On Friday 07 January 2005 11:03, Olav Langeland wrote:
Hi,

maybe I should give some more details. Backupserver is Linux with LVM,
clients are FreeBSD with just straight /var, /local etc. mounted dirs.
rdiff-backup version is v0.12.7. I was hoping to backup /var and include
all current logs in that backuprun.

-olav

-----Original Message-----
From: Neil [mailto:rdiff-backup-users < at > iamafreeman.com]
Sent: 7. januar 2005 11:56
To: rdiff-backup-users < at > nongnu.org
Cc: Olav Langeland
Subject: Re: [rdiff-backup-users] Filechange during backup

Hi,

one good workaround if you can is to use and lvm snapshot
you take a snapshot and back that up then remove the snapshot
http://www.tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html
Neil

Post Filechange during backup 
Hi,

On Fri, 7 Jan 2005, Olav Langeland wrote:

maybe I should give some more details. Backupserver is Linux with LVM,
clients are FreeBSD with just straight /var, /local etc. mounted dirs.
rdiff-backup version is v0.12.7. I was hoping to backup /var and include
all current logs in that backuprun.

While this seems OK for logfiles that do not change internally but are
appended only, it is quite hard to determine whether this is the case, or
the internal contents have changed, too.

Example 1: /var/log/syslog
With this file, it's OK to ignore the fact that, after starting transfer
of the data, new data has been appended to the logfile.

Example 2: /some/database/table.db
When no guarantee can be given that this file will not be modified during
backup, it is a bad thing (tm) to have some illegal state in the backup
tree. Say the file is 1MB, rdiff-backup has transfered 250KB, and then the
file is changed (updated at offset 10,000 and at offset 800,000 and some
data appended). What to do next? The only logical answer would be: bail
out.

For databases, other tools exist to make backups. For log files... you
could use tar to create a snapshot of the currently active log files just
before running rdiff-backup. But hey, if log files are that important, you
shouldn't rely on rdiff-backup but have them delivered to some other
remote loghost as well. If you lose /var on those boxes and need to
restore from an rdiff-backup snapshot, you'd have lost the most recent
data anyway. While there's a slight difference between losing a day's and
losing a week's log files, if losing log files is a problem, you're in
trouble no matter how much data is lost.

Regards,
Maarten Bezemer

Post Filechange during backup 
rdiff-backup-users < at > iamafreeman.com(Neil) 07.01.05 10:55

one good workaround if you can is to use and lvm snapshot
you take a snapshot and back that up then remove the snapshot

http://www.tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html

Wouldn't that double the space requirements?

The LVM-snapshot documentation is always talking about
"a copy of the data". I did not find any hints
how the snapshot feature is really working/implemented.

That sounds to me as the "snapshot" is just a "dd",
which needs a long time to process and only ensures
"stability" while reading the copy.
Is that true?
Where is it decscripted?

What happens if you have 2 files, say a database and an index.
Making the snapshot may need 10 minutes.
What if the index is already stored in the first minute but
in the 5th minute a change in the database is done which causes
an rebuilt of the index? In 7th minute the database will
be copied, and, oops, the index file already stored will not fit
anymore!


I know that there are snapshot file systems.
But the don't copy all the data but only the directory structures
meta intos, like a journaling filesystem.
So they are very fast and aware of changes during
the very short time the snapshot needs. Too they only need
to store the changed data fractions. (hm, where did i read
that recently? Wink)





Neil

On Friday 07 January 2005 10:31, Olav Langeland wrote:
Hi,

from the rdiff-backup error policy:
"These are usually caused by a file being modified as rdiff-backup
is processing it. The error log calls these UpdateErrors.
When this type of error occurs, rdiff-backup will pretend that the
file is the same as it is in the destination directory. For an
incremental backup, this means that no new increment will be
written, and the mirror copy will not be updated."

Found this out when I tried to backup web- and maillogs.

With "extended attributes" log fles can/should be set
"append only".
That may reduce the problem.


Any
workaround for it? Not counting rotating logs before backup or
making a separate scp command for logfiles.

That seems to be the only clean way, or?

I would be happy with
something that makes rdiff-backup ignore filesize changing during
backup and just copy the file :)

Any help appreciated.

Regards
Olav Langeland

Rainer---<=====> Vertraulich
//
//
<=====>--------------ocholl, Kiel, Germany ------------

Post Filechange during backup 
no. you have misunderstood lvm snapshots

LVM snapshots are far better than a dd (for this purpose at least) they hold a
copy of the any changed data before it was changed

from the lvcreate man page-
Snapshots provide a 'frozen image' of the contents of the origin while the
origin can still be updated. They enable consistent backups and online
recovery of removed/overwritten data/files. The snapshot does not need the
same amount of storage the origin has. In a typical scenario, 15-20% might be
enough.
---

if you want to imagine it in your head it is as from the time the snapshot
logical volume is created any data that would have been overwritten on the
origin logical volume is instead kept (on the snapshot logical volume). when
you read from the snapshot volume if that data has changed you get the copy
from the snapshot logical volume if the data has not changed it comes from
the origin logical volume

creating the snapshot is a fast operation, subsequent disk operations on the
origin have a ?slight? overhead whilst the snapshot exists so delete the
snapshot once the backup is done (and verified ;)

Any clearer? Have an experiment with them

be aware that just because it is a consistent snapshot of what was on the disk
at a point in time doesn't mean it will be useful. For example using a
database you need to get the database to write to disk all the info it needs
to be a valid usable snapshot. The easy generic way around this is to shut
the database down; take a snapshot; start the database up again; backup the
snapshot;remove the snapshot. This time this saves over shutdown database;
backup; startup database can be huge. or course you may not want to shut your
database down then you getting into database specific territory

On Saturday 08 January 2005 15:44, Rainer Zocholl wrote:
rdiff-backup-users < at > iamafreeman.com(Neil) 07.01.05 10:55

one good workaround if you can is to use and lvm snapshot
you take a snapshot and back that up then remove the snapshot

http://www.tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html

Wouldn't that double the space requirements?

The LVM-snapshot documentation is always talking about
"a copy of the data". I did not find any hints
how the snapshot feature is really working/implemented.

That sounds to me as the "snapshot" is just a "dd",
which needs a long time to process and only ensures
"stability" while reading the copy.
Is that true?
Where is it decscripted?

What happens if you have 2 files, say a database and an index.
Making the snapshot may need 10 minutes.
What if the index is already stored in the first minute but
in the 5th minute a change in the database is done which causes
an rebuilt of the index? In 7th minute the database will
be copied, and, oops, the index file already stored will not fit
anymore!


I know that there are snapshot file systems.
But the don't copy all the data but only the directory structures
meta intos, like a journaling filesystem.
So they are very fast and aware of changes during
the very short time the snapshot needs. Too they only need
to store the changed data fractions. (hm, where did i read
that recently? Wink)


Post Filechange during backup 
rdiff-backup-users < at > iamafreeman.com(Neil) 08.01.05 16:49


no. you have misunderstood lvm snapshots

Yes, i confirm ;-)


LVM snapshots are far better than a dd (for this purpose at least)
they hold a copy of the any changed data before it was changed

from the lvcreate man page-
Snapshots provide a 'frozen image'

that's wrong/missleading
No "Image of the *contens*" is frozen!

For someone who knows how that was meant the sentence is
clear.
But someone without that knowledge will be kicked in the
wrong direction.
Maybe "image" is not meant in the way i understand it.


of the contents of the origin while
the origin can still be updated.

The snapshot "freeze" the directory information (the "view")
and only the next changes are written to that snapshot(!) volume.

They enable consistent backups and
online recovery of removed/overwritten data/files. The snapshot does
not need the same amount of storage the origin has. In a typical
scenario, 15-20% might be enough.

The snapshot volume needs (mainly) only to keep the data that is newly
created after the snapsshot

Joe Baker wrote:

It's another view of the data files. When a snapshot is in effect
data is written differently to the drive in such a way that the
origional view is not altered. It's a pretty fancy technique.

That's makes it pretty clear how it works.


be aware that just because it is a consistent snapshot of what was on
the disk at a point in time doesn't mean it will be useful. For
example using a database you need to get the database to write to disk
all the info it needs to be a valid usable snapshot.

ACK. That is what i wanted to warn too.

The easy generic
way around this is to shut the database down; take a snapshot; start
the database up again; backup the snapshot;remove the snapshot.
This time this saves over shutdown database; backup;
startup database can be huge.
or course you may not want to shut your database down then
you getting into database specific territory

But you have typically the chance to let the database "flush"
and stop it without shuting down.

Another problem are database which have their own file systems...
but those have their own backups algorithm.



Rainer

Post Filechange during backup 
-----Original Message-----
From: Maarten Bezemer [mailto:mcbrdiff < at > robuust.nl]
Sent: 7. januar 2005 12:31
To: Olav Langeland
Cc: Neil; rdiff-backup-users < at > nongnu.org
Subject: RE: [rdiff-backup-users] Filechange during backup

Hi,

On Fri, 7 Jan 2005, Olav Langeland wrote:

maybe I should give some more details. Backupserver is Linux with
LVM,
clients are FreeBSD with just straight /var, /local etc. mounted
dirs.
rdiff-backup version is v0.12.7. I was hoping to backup /var and
include
all current logs in that backuprun.

While this seems OK for logfiles that do not change internally but are
appended only, it is quite hard to determine whether this is
the case, or the internal contents have changed, too.

Example 1: /var/log/syslog
With this file, it's OK to ignore the fact that, after
starting transfer of the data, new data has been appended to the
logfile.

Example 2: /some/database/table.db
When no guarantee can be given that this file will not be
modified during backup, it is a bad thing (tm) to have some illegal
state in
the backup tree. Say the file is 1MB, rdiff-backup has transfered
250KB,
and then the file is changed (updated at offset 10,000 and at offset
800,000 and some data appended). What to do next? The only logical
answer
would be: bail out.

For databases, other tools exist to make backups. For log files... you
could use tar to create a snapshot of the currently active log files
just
before running rdiff-backup. But hey, if log files are that important,
you
shouldn't rely on rdiff-backup but have them delivered to some other
remote loghost as well. If you lose /var on those boxes and need to
restore from an rdiff-backup snapshot, you'd have lost the most recent
data anyway. While there's a slight difference between losing a day's
and
losing a week's log files, if losing log files is a problem, you're in
trouble no matter how much data is lost.

Regards,
Maarten Bezemer

Agree. For databases it's either SQL backupclient, or the old
dump-to-file then backup. But I have some logs that for different
reasons I would prefer not to nightly logrotate but still would be nice
to have backed up.
Reason for the original post was to find out if rdiff-backup had some
workaround to generally ignore files being changed and just continue
backing them up, ofcourse at my own risk. If that is not an option I'll
just have to work something out.

Regards
Olav Langeland

Display posts from previous:
Reply to topic Page 1 of 1
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
  


Magic SEO URL for phpBB