Backup Central

On Sat, Apr 10, 2004 at 00:16:47, Craig Barratt wrote:

Hi,


* Added support for Rsync checksum caching. Rsync checksum are
appended to the compressed pool files. This means that block
and file checksums do not need to be recomputed on the server
when using rsync. Requires a patch to rsync to support fixed
checksum seeds. This patch is included in the cygwin-rsyncd
release on http://backuppc.sourceforge.net.

What do I need to do to enable this feature? Is installing the patched
rsync on my clients enough? I'm using rsync in deamon mode.

And what will happen to my current pool?

Regards
Lars
--
Lars Anderson mailto:lsa < at > business.aau.dk
Faculty of Social Sciences http://www.business.aau.dk/~lsa/
Aalborg University Voice: +45 96358225, Fax: +45 98153505
Denmark Office: Fib4-117


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Lars Anderson writes:

On Sat, Apr 10, 2004 at 00:16:47, Craig Barratt wrote:

* Added support for Rsync checksum caching. Rsync checksum are
appended to the compressed pool files. This means that block
and file checksums do not need to be recomputed on the server
when using rsync. Requires a patch to rsync to support fixed
checksum seeds. This patch is included in the cygwin-rsyncd
release on http://backuppc.sourceforge.net.

What do I need to do to enable this feature? Is installing the patched
rsync on my clients enough? I'm using rsync in deamon mode.

You will need to:

- run the patched rsync on your client, or rebuild rsync with the
patch on http://backuppc.sourceforge.net. This patch adds the
--fixed-csumseed option to rsync. I've submitted this patch
to the rsync project to make it standard, but it hasn't (yet)
been accepted.

- add '--fixed-csumseed' to $Conf{RsyncArgs}.

That's it. You can look in the XferLOG to see that it is
caching checksums - look for comments at the start of the
log file.

And what will happen to my current pool?

Nothing. The existing pool will be used as is, and, as each
file has checksums calculated the checksums will be appended
to each pool file.

Craig


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Tue, Apr 13, 2004 at 23:28:26, Craig Barratt wrote:

Hi,


You will need to:

- run the patched rsync on your client, or rebuild rsync with the
patch on http://backuppc.sourceforge.net. This patch adds the
--fixed-csumseed option to rsync. I've submitted this patch
to the rsync project to make it standard, but it hasn't (yet)
been accepted.

- add '--fixed-csumseed' to $Conf{RsyncArgs}.

That's it. You can look in the XferLOG to see that it is
caching checksums - look for comments at the start of the
log file.


Thanks, got it working. However I'm seeing some errors like:

fatal error: md4 doesn't match

in the XferLOG. Is this OK?

Lars

Lars Anderson writes:

You will need to:

- run the patched rsync on your client, or rebuild rsync with the
patch on http://backuppc.sourceforge.net. This patch adds the
--fixed-csumseed option to rsync. I've submitted this patch
to the rsync project to make it standard, but it hasn't (yet)
been accepted.

- add '--fixed-csumseed' to $Conf{RsyncArgs}.

That's it. You can look in the XferLOG to see that it is
caching checksums - look for comments at the start of the
log file.

Thanks, got it working. However I'm seeing some errors like:

fatal error: md4 doesn't match

in the XferLOG. Is this OK?

No, that's a serious error, meaning the overall file checksum didn't
match after it was transferred. That probably means there is some
problem with checksum caching on your system.

Please do the following checks:

- Please tell me your File:RsyncP and client rsync versions.

- Do you get this error on every file, or just certain files?

- Is there something special about the files that get an
error, eg: recently modified since last full backup,
certain file sizes etc. Are they the same files each time
you do a backup?

- If you remove '--fixed-csumseed' from $Conf{RsyncArgs}
does the error go away?

Craig

On Wed, Apr 14, 2004 at 23:39:11, Craig Barratt wrote:

That's it. You can look in the XferLOG to see that it is
caching checksums - look for comments at the start of the
log file.

Thanks, got it working. However I'm seeing some errors like:

fatal error: md4 doesn't match

in the XferLOG. Is this OK?

No, that's a serious error, meaning the overall file checksum didn't
match after it was transferred. That probably means there is some
problem with checksum caching on your system.

Please do the following checks:

- Please tell me your File:RsyncP and client rsync versions.


File-RsyncP-0.50.tar.gz
rsync 2.6.0 + cygwin-rsyncd-2.6.0_0.diff

- Do you get this error on every file, or just certain files?


Just some files, a couple of hundred files per backup.

- Is there something special about the files that get an
error, eg: recently modified since last full backup,
certain file sizes etc. Are they the same files each time
you do a backup?


I can't really see a pattern, it is not every new file. I'll look
harder.

- If you remove '--fixed-csumseed' from $Conf{RsyncArgs}
does the error go away?


The error wasn't there before I enabled --fixed-csumseed, but I was not
running rsync 2.6.0 but 2.5.6. I will disable it for the run tonight.

I just noticed that on some of the backups I also get this error, which
I have never seen before:

Unable to read 12288 bytes from /dumps/pc/flounder-home4/new//fhome4/RStmp got=0, seekPosn=0 (0,6,8684,0,17783358)

Lars

I'm seeing these errors as well, on one host, both of the errors, but the
md4 doesn't match on multiple hosts, a few files each (75 out of 20000).

They do not appear to have a pattern in regards to modification or file
size. I see recently modified files, but also ones that I know haven't been
touched.

I'm getting this on one host (my laptop) over multiple backup runs, 12000
instances each run, in addition to the md4 doesn't match:

Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)


Removing --fixed-checksum goes back to no errors (besides locked files)

Versions:

File::RsyncP Version 0.50, released 20 Mar 2004.

And

rsync version 2.6.0 protocol version 27
Copyright (C) 1996-2004 by Andrew Tridgell and others
<http://rsync.samba.org/>
Version 2.6.0_0 from <http://backuppc.sourceforge.net/>
(includes --fixed-csumseed and buffering patches)
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
no IPv6, 64-bit system inums, 64-bit internal inums


Greg Nicholson

-----Original Message-----
From: backuppc-users-admin < at > lists.sourceforge.net
[mailto:backuppc-users-admin < at > lists.sourceforge.net] On Behalf Of Lars
Anderson
Sent: Thursday, April 15, 2004 3:40 AM
To: Craig Barratt
Cc: backuppc-users < at > lists.sourceforge.net
Subject: Re: [BackupPC-users] BackupPC 2.1.0beta1 released

On Wed, Apr 14, 2004 at 23:39:11, Craig Barratt wrote:

That's it. You can look in the XferLOG to see that it is
caching checksums - look for comments at the start of the
log file.

Thanks, got it working. However I'm seeing some errors like:

fatal error: md4 doesn't match

in the XferLOG. Is this OK?

No, that's a serious error, meaning the overall file checksum didn't
match after it was transferred. That probably means there is some
problem with checksum caching on your system.

Please do the following checks:

- Please tell me your File:RsyncP and client rsync versions.


File-RsyncP-0.50.tar.gz
rsync 2.6.0 + cygwin-rsyncd-2.6.0_0.diff

- Do you get this error on every file, or just certain files?


Just some files, a couple of hundred files per backup.

- Is there something special about the files that get an
error, eg: recently modified since last full backup,
certain file sizes etc. Are they the same files each time
you do a backup?


I can't really see a pattern, it is not every new file. I'll look
harder.

- If you remove '--fixed-csumseed' from $Conf{RsyncArgs}
does the error go away?


The error wasn't there before I enabled --fixed-csumseed, but I was not
running rsync 2.6.0 but 2.5.6. I will disable it for the run tonight.

I just noticed that on some of the backups I also get this error, which
I have never seen before:

Unable to read 12288 bytes from /dumps/pc/flounder-home4/new//fhome4/RStmp
got=0, seekPosn=0 (0,6,8684,0,17783358)

Lars


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

"Greg Nicholson" writes:

I'm seeing these errors as well, on one host, both of the errors, but the
md4 doesn't match on multiple hosts, a few files each (75 out of 20000).

They do not appear to have a pattern in regards to modification or file
size. I see recently modified files, but also ones that I know haven't been
touched.

I'm getting this on one host (my laptop) over multiple backup runs, 12000
instances each run, in addition to the md4 doesn't match:

Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)

Removing --fixed-checksum goes back to no errors (besides locked files)

Ok, this definitely is a bug with checksum caching. And it looks
like Lars has seen the same problem too.

I need to replicate it to fix it. It appears the file that
fails is one that is modified (ie: not new), and is large
(ie: >16MB). I'll try cases like that. If I can't replicate
it I'll get back to you for more information.

Craig


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On Thu, Apr 15, 2004 at 19:57:56, Greg Nicholson wrote:

I'm seeing these errors as well, on one host, both of the errors, but the
md4 doesn't match on multiple hosts, a few files each (75 out of 20000).

They do not appear to have a pattern in regards to modification or file
size. I see recently modified files, but also ones that I know haven't been
touched.

I'm getting this on one host (my laptop) over multiple backup runs, 12000
instances each run, in addition to the md4 doesn't match:

Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)


Removing --fixed-checksum goes back to no errors (besides locked files)

Hmm, not for me, here's a snip from the Errorlog for one of my machines,
I'll go back to rsync 2.5.7 and see what happens.:

Connected to bowhead:873, remote version 27
Connected to module home1
Sending args: --server --sender --numeric-ids --perms --owner --group --devices --links --times --block-size=2048 --recursive . .
Xfer PIDs are now 29327
[ skipped 552 lines ]
ab/progdata/Outlook/outlook.pst: fatal error: md4 doesn't match
[ skipped 469 lines ]
akj/document/Sikkerhedskopi af Timeregnskab.wbk: fatal error: md4 doesn't match
akj/document/Timeregnskab.F03 AK-J.doc: fatal error: md4 doesn't match
[ skipped 216 lines ]
all/document/FREIA-A/2004/25-03-2004/Backup of Referat D1.wbk: fatal error: md4 doesn't match
all/document/FREIA-A/2004/25-03-2004/Referat D1.doc: fatal error: md4 doesn't match
[ skipped 51 lines ]
all/document/Template/MSO2000/Backup of Normal.wbk: fatal error: md4 doesn't match
[ skipped 2 lines ]
all/document/Template/MSO2000/FREIA skabeloner/debitorfolgeseddel.dot: fatal error: md4 doesn't match
[ skipped 2 lines ]
all/document/Template/MSO2000/Normal.dot: fatal error: md4 doesn't match
[ skipped 9 lines ]
all/document/index/Backup of Ryglabel med sorgerand.wbk: fatal error: md4 doesn't match
all/document/index/Ryglabel med sorgerand.doc: fatal error: md4 doesn't match
[ skipped 17 lines ]
Unable to read 15312 bytes from /dumps/pc/bowhead-home1/new//fhome1/RStmp got=0, seekPosn=0 (0,2,10001,0,76562432)
Unable to read 15312 bytes from /dumps/pc/bowhead-home1/new//fhome1/RStmp got=0, seekPosn=0 (0,2,10001,0,76562432)
Unable to read 15312 bytes from /dumps/pc/bowhead-home1/new//fhome1/RStmp got=0, seekPosn=0 (0,2,10001,0,76562432)
Unable to read 15312 bytes from /dumps/pc/bowhead-home1/new//fhome1/RStmp got=0, seekPosn=0 (0,2,10001,0,76562432)
Unable to read 15312 bytes from /dumps/pc/bowhead-home1/new//fhome1/RStmp got=0, seekPosn=0 (0,2,10001,0,76562432)
Unable to read 15312 bytes from /dumps/pc/bowhead-home1/new//fhome1/RStmp got=0, seekPosn=0 (0,2,10001,0,76562432)

And just as further information:

I backup my laptop at both work and home. The Home machine (Athlon
2800xp 512M memory backing up to XFS on LVM2 on top of Software Raid5 array)
gets the errors. I just checked the logs for the work machine (Dual Athlon
MPs, 1G Ram, XFS on LVM2 with Hardware (3ware) raid), and no errors on the
same client. Both Machines are running 2.6.5 linux kernel, Debian boxes.

The home machine is running Debian Unstable , with Perl 5.8.3. Work
is Debian Stable, Perl 5.6.1.

Could this be a problem with the Perl versions?

Greg Nicholson

-----Original Message-----
From: Craig Barratt [mailto:cbarratt < at > users.sourceforge.net]
Sent: Friday, April 16, 2004 2:21 AM
To: Greg Nicholson
Cc: Lars Anderson; backuppc-users < at > lists.sourceforge.net
Subject: Re: [BackupPC-users] BackupPC 2.1.0beta1 released

"Greg Nicholson" writes:

I'm seeing these errors as well, on one host, both of the errors, but the
md4 doesn't match on multiple hosts, a few files each (75 out of 20000).

They do not appear to have a pattern in regards to modification or file
size. I see recently modified files, but also ones that I know haven't
been
touched.

I'm getting this on one host (my laptop) over multiple backup runs, 12000
instances each run, in addition to the md4 doesn't match:

Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)

Removing --fixed-checksum goes back to no errors (besides locked files)

Ok, this definitely is a bug with checksum caching. And it looks
like Lars has seen the same problem too.

I need to replicate it to fix it. It appears the file that
fails is one that is modified (ie: not new), and is large
(ie: >16MB). I'll try cases like that. If I can't replicate
it I'll get back to you for more information.

Craig

Greg Nicholson writes:

And just as further information:

I backup my laptop at both work and home. The Home machine (Athlon
2800xp 512M memory backing up to XFS on LVM2 on top of Software Raid5 array)
gets the errors. I just checked the logs for the work machine (Dual Athlon
MPs, 1G Ram, XFS on LVM2 with Hardware (3ware) raid), and no errors on the
same client. Both Machines are running 2.6.5 linux kernel, Debian boxes.

The home machine is running Debian Unstable , with Perl 5.8.3. Work
is Debian Stable, Perl 5.6.1.

Could this be a problem with the Perl versions?

Not sure. I've tried to replicate this bug without success.

I've written test programs that have created thousands of
files of various sizes, done full and incremental backups,
changed random subsets of data in the files, and repeated
many times without any errors. I also upgraded to perl 5.8.3
and still no problems. I've tried rsync 2.6.0 and 2.5.6.

It appears that for both you and Lars the temporary file, RStmp,
is empty, which means it didn't uncompress the original pool file
correctly, or it got deleted prior to reading. This temporary file
is created when there are changes to a client file bigger than 16MB.
The only thing I can think of is that the pool file somehow got
corrupted when the checksums were appended to the file.

One question: do you have pool compression on or off?

We can proceed in a couple of different ways. I can add some more
debug messages to BackupPC::Xfer::RsyncFileIO.pm and get you to
re-run. Or if there is some way to get access to a machine that
fails I would be happy to debug it there.

When you next reply let's move this to the backuppc-devel list.
Also, if you want to send me log files etc or discuss access
to one of your machines then let's take this off list.

Craig

-----Original Message-----
From: Craig Barratt [mailto:cbarratt < at > users.sourceforge.net]
Sent: Friday, April 16, 2004 2:21 AM
To: Greg Nicholson
Cc: Lars Anderson; backuppc-users < at > lists.sourceforge.net
Subject: Re: [BackupPC-users] BackupPC 2.1.0beta1 released

"Greg Nicholson" writes:

I'm seeing these errors as well, on one host, both of the errors, but the
md4 doesn't match on multiple hosts, a few files each (75 out of 20000).

They do not appear to have a pattern in regards to modification or file
size. I see recently modified files, but also ones that I know haven't
been
touched.

I'm getting this on one host (my laptop) over multiple backup runs, 12000
instances each run, in addition to the md4 doesn't match:

Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)
Unable to read 16384 bytes from
/.1/backuppc_data/pc/greglap/new//fcDrive/RStmp got=0, seekPosn=0
(0,1,12650,0,207242240)

Removing --fixed-checksum goes back to no errors (besides locked files)

Ok, this definitely is a bug with checksum caching. And it looks
like Lars has seen the same problem too.

I need to replicate it to fix it. It appears the file that
fails is one that is modified (ie: not new), and is large
(ie: >16MB). I'll try cases like that. If I can't replicate
it I'll get back to you for more information.

Craig


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Craig Barratt wrote:

Not sure. I've tried to replicate this bug without success.

I've written test programs that have created thousands of
files of various sizes, done full and incremental backups,
changed random subsets of data in the files, and repeated
many times without any errors. I also upgraded to perl 5.8.3
and still no problems. I've tried rsync 2.6.0 and 2.5.6.

I looked at the checksum cache code. Whats with the condition a file
with none checksum cache do have the magic ?
I dislikes the idea of altering pool files to add the checksum in the body.

I suggest another solution:

1. Something like $filename.metas which can store addition informations
like checksums,...
2. Implement a real container format, it can be really simple, but
should provide flexibility for future developments.
3. A database driven checksum database. gdb would does the job, but
something like sqlite would allow feature usage. The nightly job would
simple remove the entry when the file is unlinked.

regards
Daniel

"daniel.poelzleithner" writes:

Not sure. I've tried to replicate this bug without success.

I've written test programs that have created thousands of
files of various sizes, done full and incremental backups,
changed random subsets of data in the files, and repeated
many times without any errors. I also upgraded to perl 5.8.3
and still no problems. I've tried rsync 2.6.0 and 2.5.6.

I looked at the checksum cache code. Whats with the condition a file
with none checksum cache do have the magic ?

I don't understand your question. A file only gets checksums added
when it is next used with the right checksum seed.

I dislikes the idea of altering pool files to add the checksum in the body.

I suggest another solution:

1. Something like $filename.metas which can store addition informations
like checksums,...
2. Implement a real container format, it can be really simple, but
should provide flexibility for future developments.
3. A database driven checksum database. gdb would does the job, but
something like sqlite would allow feature usage. The nightly job would
simple remove the entry when the file is unlinked.

Comments:

1. I didn't want to double the number of files in the pool. The pool
can already get pretty large. Also, the checksums are pretty small,
so typically they will occupy less than a disk block. Making them
separate files does reduce the storage efficiency somewhat.

2. Yes, a real container format would be better. The current method
is extensible, but it is quick a hack.

3. A database is another option, but I'm not sure how efficiently
something like mysql, sqlite or gdbm would store millions of
entries. I guess we could do some benchmarks are find out.

In an earlier email you also made this excellent point:

i saw on the changelog that the checksum is now cached. I would suggest
to add a program that checks the pool for integrity. For example a bit
flips and the checksum file is not valid against the file in the tree
anymore. Then backuppc should try to check if the checksum is bad or the
file. For this it could recieve the file again form one host or the
checksum and fix the pool.

Since rsync also sends a whole-file md4 digest, any errors in the
stored checksums will cause the whole-file md4 digest to not match.
This causes rsync to re-do the file with longer checksums. Currently
it will fail a second time, since the stored checksums are re-used.
What I should do is on rsync's second (retry) pass is recompute
the checksums from the original file. This will catch the case of
corrupted checksums.

You correctly point out that the file data could get corrupted and
BackupPC wouldn't notice when checksum caching is turned on. I'm
not sure of the best solution for this. Perhaps, with a certain
settable probability (eg: 2%), the checksums are recomputed and
re-checked, either as they are used or by BackupPC_nightly. That
way the cached checksums in the pool will (slowly) be rechecked.
I'll add this to the todo list.

Craig


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Craig Barratt wrote:

| Comments:
|
| 1. I didn't want to double the number of files in the pool. The pool
| can already get pretty large. Also, the checksums are pretty small,
| so typically they will occupy less than a disk block. Making them
| separate files does reduce the storage efficiency somewhat.

Year, i know. This was the worst idea

| 2. Yes, a real container format would be better. The current method
| is extensible, but it is quick a hack.
|
| 3. A database is another option, but I'm not sure how efficiently
| something like mysql, sqlite or gdbm would store millions of
| entries. I guess we could do some benchmarks are find out.

http://www.sqlite.org/speed.html

Sqlite is really a good solution here. The only disadantage is, that
concurrent writes can cause a small delay. Storing some millions of
lines are not a problem for a database.

I have tested sqlite with a simple (key, value) table and inserted
300000 key pairs. The key is a md5 hash value. The lookup didn't take
any noticable time.


regards
~ Daniel

- --
nihil me cirumdat

.. . .. ... . . .. . ... . .. . ... . . .
pgp key < at > http://files.poelzi.org/pgp.txt
ED80 E53D 5269 4BB1 1E73 3A53 CBF9 A421 0A7B 003D
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Debian - http://enigmail.mozdev.org

iD8DBQFAhWvXy/mkIQp7AD0RAnCOAKDD/RN1bSIuFgFtVexc08qERbmcYACg1b+N
6ceCuUjBBiQWrKADUyKghNU=
=p4o6
-----END PGP SIGNATURE-----