Backup Central

Hello, I have been monitoring backups tonight and one of them was taking
an unusual longer time to backup. I noticed that it is downloading files
that are already in the pool. I know I have spoke about this before but
this is a different problem. I'm comparing the files that are being
downloaded to the files that are in the pool, and it seems that it will
download files that have been moved to another directory. Is that true?

So for example, I have some docs in /docs or whatever, these get backed
up fine and after the full backup, they are never downloaded. If these
docs are moved to say /old-docs, then they all get downloaded again. I
maybe wrong, but that's what's happening as far as I can tell.

I understand that after it's finished downloading all files, that the
linker will go through and remove the duplicates, creating hard links to
the original files, but I don't suppose there is way to make it not
download files that have just been moved by any chance is there?

Another thing I have noticed is NewFileList - is this the file that
BackupPC looks at to determine wheather or not to send an ALM (because
the backup has timed out)? I've been monitoring this file during this
particular backup and it has just stopped producing any output to the
file even though new files have been copied.

Thanks for any help,

Ben.



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Ben writes:

Hello, I have been monitoring backups tonight and one of them was taking
an unusual longer time to backup. I noticed that it is downloading files
that are already in the pool. I know I have spoke about this before but
this is a different problem. I'm comparing the files that are being
downloaded to the files that are in the pool, and it seems that it will
download files that have been moved to another directory. Is that true?

So for example, I have some docs in /docs or whatever, these get backed
up fine and after the full backup, they are never downloaded. If these
docs are moved to say /old-docs, then they all get downloaded again. I
maybe wrong, but that's what's happening as far as I can tell.

This is normal.

I understand that after it's finished downloading all files, that the
linker will go through and remove the duplicates, creating hard links to
the original files, but I don't suppose there is way to make it not
download files that have just been moved by any chance is there?

Each of the Xfer methods that BackupPC uses (smb, tar, rsync) can't
detect renamed files. So if you rename a file it will be transfered.

But after it is transferred it will be matched with the pool, so no
additional storage will be required. So renamed files cause network
overhead, but no storage overhead.

Another thing I have noticed is NewFileList - is this the file that
BackupPC looks at to determine wheather or not to send an ALM (because
the backup has timed out)? I've been monitoring this file during this
particular backup and it has just stopped producing any output to the
file even though new files have been copied.

Because of stdio buffering this file won't get updated for every
new file.

Craig

On Wed, 2004-03-17 at 18:59, Craig Barratt wrote:

Ben writes:

Hello, I have been monitoring backups tonight and one of them was taking
an unusual longer time to backup. I noticed that it is downloading files
that are already in the pool. I know I have spoke about this before but
this is a different problem. I'm comparing the files that are being
downloaded to the files that are in the pool, and it seems that it will
download files that have been moved to another directory. Is that true?

So for example, I have some docs in /docs or whatever, these get backed
up fine and after the full backup, they are never downloaded. If these
docs are moved to say /old-docs, then they all get downloaded again. I
maybe wrong, but that's what's happening as far as I can tell.

This is normal.

I understand that after it's finished downloading all files, that the
linker will go through and remove the duplicates, creating hard links to
the original files, but I don't suppose there is way to make it not
download files that have just been moved by any chance is there?

Each of the Xfer methods that BackupPC uses (smb, tar, rsync) can't
detect renamed files. So if you rename a file it will be transfered.

But after it is transferred it will be matched with the pool, so no
additional storage will be required. So renamed files cause network
overhead, but no storage overhead.

Ok, what I mean though is if a file is actually moved from one directory
to another, not renamed, then it will be downloaded again, and there is
no way to stop this?

Another thing I have noticed is NewFileList - is this the file that
BackupPC looks at to determine wheather or not to send an ALM (because
the backup has timed out)? I've been monitoring this file during this
particular backup and it has just stopped producing any output to the
file even though new files have been copied.

Because of stdio buffering this file won't get updated for every
new file.

Ok I understand =)

Craig

Ben writes:

I understand that after it's finished downloading all files, that the
linker will go through and remove the duplicates, creating hard links to
the original files, but I don't suppose there is way to make it not
download files that have just been moved by any chance is there?

Each of the Xfer methods that BackupPC uses (smb, tar, rsync) can't
detect renamed files. So if you rename a file it will be transfered.

But after it is transferred it will be matched with the pool, so no
additional storage will be required. So renamed files cause network
overhead, but no storage overhead.

Ok, what I mean though is if a file is actually moved from one directory
to another, not renamed, then it will be downloaded again, and there is
no way to stop this?

That's right. By "rename" I also mean moving a file to a new directory.
There's no simple way to detect this at transfer time (some heuristics
could be added to rsync, but it doesn't seem worth the trouble). But
pooling matches any identical file after the file is transferred.

Craig


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Tue, 16 Mar 2004, Craig Barratt wrote:
Ben writes:

[...]

Another thing I have noticed is NewFileList - is this the file that
BackupPC looks at to determine wheather or not to send an ALM
(because the backup has timed out)? I've been monitoring this file
during this particular backup and it has just stopped producing any
output to the file even though new files have been copied.

Because of stdio buffering this file won't get updated for every
new file.

Is there any chance that you could set the file handle for NewFileList
to auto-flush in the next release?

I occasionally want to see how progress is going with a backup, and that
would make it much easier...

Daniel

--
Every man has a mob self and an individual self, in varying proportions.
-- D. H. Lawrence



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Mon, 22 Mar 2004, Craig Barratt wrote:
Daniel Pittman writes:

Is there any chance that you could set the file handle for
NewFileList to auto-flush in the next release?

I occasionally want to see how progress is going with a backup, and
that would make it much easier...

NewFileList just shows files that aren't yet in the pool, so you could
go a decent way through the backup without much appearing in
NewFileList.

*nod*

If you recognize the file names and have an idea of the file system
layout you are backing up (which I expect you do), then NewFileList
can give you an idea of how far along you are. But it's probably
better to have an indication of the fraction of files done, eg:

current file # / total # files

That's possible with rsync, but not with smb or tar.

Well, that would be good.

With the other methods, the size / number of files in the previous
backup could serve as something of a guide as to what to expect; even
just displaying those figures and letting a human make the guess from
them.

I can add this to the todo list.

Thanks.
Daniel

--
In the attention economy, anyone trying to connect with an
audience must treat the user's time as the ultimate resource.
-- Jakob Neilsen



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/