Backup Central

When does BackupPC place files into the cpool directory? I have a
small test installation running several weeks and files appear in
cpool. I have a new production installation and over the last two
weeks no files have ever appeared in cpool. Shouldn't it contain at
least some files? I fear that I have something misconfigured.

There are 5 hosts that have been backed up, for a total of:
12 full backups of total size 312.37GB (prior to pooling and compression),
23 incr backups of total size 46.47GB (prior to pooling and compression).

Pool is 0.00GB comprising 0 files and 1 directories (as of 11/21 01:00),
Pool hashing gives 0 repeated files with longest chain 0,
Nightly cleanup removed 0 files of size 0.00GB (around 11/21 01:00),

Bob

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Bob Proulx wrote:
When does BackupPC place files into the cpool directory? I have a
small test installation running several weeks and files appear in
cpool. I have a new production installation and over the last two
weeks no files have ever appeared in cpool. Shouldn't it contain at
least some files? I fear that I have something misconfigured.

Further investigation shows that BackupPC is throwing errors during
the link run. Seeing many of these in the log file. (It would be
great if errors like this were emailed to the admin as part of the
email notification from BackupPC. I had not noticed them.)

2011-11-21 06:01:57 BackupPC_link got error -4 when calling MakeFileLink(/srv/backuppc/pc/vb/8/f%2ftask/fshome/fs_zunija/f.bashrc, 8b7ad36a20b66d59737cad5b86de1c3a, 1)

And apparently due to this it isn't de-duplicating. This is that
example file from two different full backups.

# ls -li pc/vb/*/f%2ftask/fshome/fs_zunija/f.bashrc
17555973 -rw-r----- 1 backuppc backuppc 173 Nov 21 05:57 pc/vb/5/f%2ftask/fshome/fs_zunija/f.bashrc
14591371 -rw-r----- 1 backuppc backuppc 104 Nov 21 05:57 pc/vb/8/f%2ftask/fshome/fs_zunija/f.bashrc

# /usr/share/backuppc/bin/BackupPC_zcat pc/vb/5/f%2ftask/fshome/fs_zunija/f.bashrc | md5sum
f119c865306c35e64eb00f65d7279664 -
# /usr/share/backuppc/bin/BackupPC_zcat pc/vb/8/f%2ftask/fshome/fs_zunija/f.bashrc | md5sum
f119c865306c35e64eb00f65d7279664 -

The file contents have not changed between the backups. The md5sums
of the files are the same. Apparently it tried to link but failed. I
have placed the entire directory onto its own partition. It is an
ext3 volume on lvm mounted (as suggested) with noatime.

Any idea what error -4 from MakeFileLink would suggest?

I am digging through the code now...

Bob

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Mon, Nov 21, 2011 at 1:00 PM, Bob Proulx <bob < at > proulx.com> wrote:

Further investigation shows that BackupPC is throwing errors during
the link run.  Seeing many of these in the log file.  (It would be
great if errors like this were emailed to the admin as part of the
email notification from BackupPC.  I had not noticed them.)

The usual causes for this are that you either have the cpool and pc
directories on different filesystems (which can't work) or you changed
the TopDir location after installation (which doesn't work in versions
before 3.2). If you can't update to 3.2, the best approach is to
mount your archive in the expected location or symlink it so it
appears there. A less likely cause is that you have used a filesystem
that doesn't support hardlinks (like vfat).

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Les Mikesell wrote:
Bob Proulx wrote:
Further investigation shows that BackupPC is throwing errors during
the link run. Seeing many of these in the log file. (It would be
great if errors like this were emailed to the admin as part of the
email notification from BackupPC. I had not noticed them.)

The usual causes for this are that you ... changed the TopDir
location after installation (which doesn't work in versions before
3.2).

You have identified the problem!

Sorry for neglecting to provide a complete context. I am running
version 3.1.0 from Debian Stable Squeeze. All filesystems are ext3.
I installed it and the default installation placed the files in a new
directory at /var/lib/backuppc. But then to have sufficient space on
its own dedicated filesystem I created a new lvm partition and moved
it to /srv/backuppc afterward. I edited the configuration file and
changed to "$Conf{TopDir} = '/srv/backuppc';" which I thought would
have been sufficient. I did this before performing any backups so
that all data would be stored in the new location. Obviously I left
some remnant behind which is causing the problem.

I followed the docs on the topic "Copying the pool" which recommend
this process. Unfortunately that section of the docs were a little
sparse on details. I didn't see an equivalent topic page on the wiki
for this.

Update: I just now found this page that I had not found previously
even though I had looked for that type of page. It doesn't seem to be
very well linked into the upper documentation.

http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=Change_archive_directory

I am reading it in detail now and digging into what detail I missed
previously.

If you can't update to 3.2, the best approach is to mount your
archive in the expected location or symlink it so it appears there.

Obviously due to the advantages of a long term operating system
release I would prefer to remain on the Debian Stable release. If
this is simply a misconfiguration on my part then I should be able to
correct it. Now that you have pointed me to the area of the problem I
am sure I can debug it further and determine my problem.

Thanks!
Bob

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Mon, Nov 21, 2011 at 3:45 PM, Bob Proulx <bob < at > proulx.com> wrote:
Update: I just now found this page that I had not found previously
even though I had looked for that type of page.  It doesn't seem to be
very well linked into the upper documentation.

 http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=Change_archive_directory

In "Changing the name of the archive directory," the part about
changing Lib.pm is critical for the pre-3.2 version and easily missed.

A.

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Mon, Nov 21, 2011 at 2:56 PM, Steve <leperas < at > gmail.com> wrote:
On Mon, Nov 21, 2011 at 3:45 PM, Bob Proulx <bob < at > proulx.com> wrote:
Update: I just now found this page that I had not found previously
even though I had looked for that type of page.  It doesn't seem to be
very well linked into the upper documentation.

 http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=Change_archive_directory

In "Changing the name of the archive directory," the part about
changing Lib.pm is critical for the pre-3.2 version and easily missed.


The simple-minded approach is to mount your partition as
/var/lib/backuppc or put a symlink there before installing the
package. That way doesn't require any program changes.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Mon, Nov 21, 2011 at 2:45 PM, Bob Proulx <bob < at > proulx.com> wrote:

 I edited the configuration file and
changed to "$Conf{TopDir} = '/srv/backuppc';" which I thought would
have been sufficient.  I did this before performing any backups so
that all data would be stored in the new location.  Obviously I left
some remnant behind which is causing the problem.

If you install from the sourceforge tarball, there is an installation
script that lets you choose locations. However, in the
distribution-packaged versions this step has already been done. In
versions before 3.2, that involved substituting paths into the code so
it wasn't a runtime evaluation everywhere.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Steve wrote:
Bob Proulx wrote:
http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=Change_archive_directory

In "Changing the name of the archive directory," the part about
changing Lib.pm is critical for the pre-3.2 version and easily missed.

It looks to me like this has already been done. And in the upstream
source, not in any patches. I just reviewed the Debian specific
patches and they look fine to me but do not cover anything in this
area.

File Lib.pm:
sub new
...
$topDir = '/var/lib/backuppc' if ( $topDir eq "" );

I am guessing that the wisdom of the net had determined that the above
doesn't work as intended and doesn't get topDir from the config file.
Okay. I will put a debug statement in there (just for my own info)
and will override the value at that point.

I can certainly symlink from one location to the other. Have done so.
But wanting to have a better understanding I am going to poke into the
problem a little more. Because I would like to be able to recover the
now existing backups and have them linked into the cpool to save disk
space.

Bob

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Mon, Nov 21, 2011 at 4:46 PM, Bob Proulx <bob < at > proulx.com> wrote:

I can certainly symlink from one location to the other.  Have done so.
But wanting to have a better understanding I am going to poke into the
problem a little more.  Because I would like to be able to recover the
now existing backups and have them linked into the cpool to save disk
space.

If you are using rsync, copies that are unchanged across runs will be
linked to each other, but without the link in cpool you won't get
pooling of duplicates on different machines. And normal operations
won't re-create that link unless the file is new/changed. The tool
here might do it:
http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=BackupPC_FixLinks

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On Tue, Nov 22, 2011 at 3:45 AM, Bob Proulx <bob < at > proulx.com> wrote:

Obviously due to the advantages of a long term operating system release I would prefer to remain on the Debian Stable release.

I would advise continuing with Debian packaging system for all the
prerequisite "infrastructure" dependencies, but for BPC itself the
"installation from source" process is both easy and IMO pretty
bullet-proof, including future upgrades. Or using the package
management ability to pull just that one package from the "testing"
repo would also work fine, as long as you are/get familiar with how to
guard against dependency-hell confusion. . .

A completely separate point, but related - rather than messing with
changing TopDir or other config options in your BPC setup, I recommend
leaving everything where it wants to be and just using symlinks (or
fstab mountpoints) to point to wherever you'd like.

Related because doing it this way would allow you to stick to your old
BPC version and still put your folders wherever you like. . .

---------------------------------

Below is my own "experimentation", not something supported by the BPC
developers team nor even recommended by me without all the normal YMMV
disclaimers. . .

I use this method to duplicate the "old-school" setup of having
everything, including the configuration and log folders, under my
TopDir on a self-contained LV for maximum flexibility and portability,
and have successfully used this setup to move my BPC data set from one
host to another completely seamlessly, actually using it to rotate BPC
instances off-site, and have successfully restored using a LiveCD
running a completely different distro from the production host on a
spare user-desktop class PC available at the off-site location.

BackupPC rocks. . .

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

hansbkk < at > gmail.com wrote on 11/22/2011 05:07:51 AM:

On Tue, Nov 22, 2011 at 3:45 AM, Bob Proulx <bob < at > proulx.com> wrote:

Obviously due to the advantages of a long term operating system
release I would prefer to remain on the Debian Stable release.

A completely separate point, but related - rather than messing with
changing TopDir or other config options in your BPC setup, I recommend
leaving everything where it wants to be and just using symlinks (or
fstab mountpoints) to point to wherever you'd like.

I wholeheartedly agree. If you're gonna drink the Kool-Aid, drink it: if Debian wants the pool at X, leave it at X! If you're going to mess with things like that, then you've already moved away from a Stable configuration. A simple symlink gives you the ability to manage your data however you want, and yet remain 100% compatible with Stable.

Timothy J. Massey
Out of the Box Solutions, Inc.
Creative IT Solutions Made Simple!
[url=Arial]http://www.OutOfTheBoxSolutions.com[/url]
[url=Arial]tmassey < at > obscorp.com[/url] 22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796