Backup Central

At 04:37 PM 2/18/2004 -0600, Carl Wilhelm Soderstrom wrote:
further debugging information..
(again, the problem is that the backup *used* to work on one box; then
suddenly stopped; it won't work at all on a new box -- backup process
starts, then hangs on the client side with very little transferred, yet the
server side keeps munching CPU time).


I tried the debugging method described in the man page (run
[..]/bin/BackupPC_dump -v -f hostName by hand); and got the following in the
resulting log:

create d 755 0/0 4096 var/lib/apt/lists
create d 755 0/0 4096 var/lib/apt/lists/partial
create d 755 0/0 4096 var/lib/aptitude
...skipping...
create d 755 0/0 4096 var/log/ntpstats
create d2775 0/8 4096 var/mail
create d 755 0/0 4096 var/opt
create d 755 0/0 4096 var/run
create d 755 0/0 4096 var/run/cups
create d 755 0/0 4096 var/run/sshd
create d 755 0/0 4096 var/spool
create d 755 0/0 4096 var/spool/cron
create d 700 1/1 4096 var/spool/cron/atjobs
create d 700 1/1 4096 var/spool/cron/atspool
create d1730 0/101 4096 var/spool/cron/crontabs
create d 710 0/104 4096 var/spool/cups
create d 711 0/104 4096 var/spool/cups/certs
create d1770 0/104 4096 var/spool/cups/tmp
pool l 777 0/0 7 var/spool/mail
create d 755 8/0 4096 var/spool/nullmailer
create d 750 8/0 4096 var/spool/nullmailer/queue
create d 750 8/0 4096 var/spool/nullmailer/tmp
create p 622 8/0 0 var/spool/nullmailer/trigger
create d1777 0/0 4096 var/tmp
create d1777 0/0 4096 var/tmp/vi.recover
create l 777 0/0 25 vmlinuz
create l 777 0/0 25 vmlinuz.old
Can't open /var/lib/backuppc/pc/illusionist/new//f%2f/ for empty output
create 0 / 0
Can't open /var/lib/backuppc/pc/illusionist/new//f%2f/ for empty output
create 0 / 0
Can't open /var/lib/backuppc/pc/illusionist/new//f%2f/ for empty output
create 0 / 0


I see there was a similar problem referenced in:
http://sourceforge.net/mailarchive/forum.php?thread_id=3741514&forum_id=503

but no resolution.


I again tried to run rsync itself, with roughly the same options as BackupPC
uses:
# rsync -e ssh --perms --owner --group --devices --links --times
--block-size=2048 --recursive root < at > illusionist:/ .

and it seems to work just fine. trying to run it as 'backuppc' on the server
side, rather than as root, also seems to work.

When I traced down the RsyncP problem (which had symptoms versy similar to
yours), I started by bumping up the debugging logging level, say like:

$Conf{RsyncLogLevel} = 6;

and then looking to see what showed up in the log. That clue led to
other places to look.

Happy hunting...


Marlin Prowell
Cadalog, Inc.




-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Carl Wilhelm Soderstrom writes:

client is using rsync-2.6.0; so after some experimentation with bare rsync,
I discovered I needed to turn off the '--numeric-ids' option in
$Conf{RsyncArgs}, and it seems to help to turn on '--blocking-io' in the
same place.

You should definitely have --numeric-ids turned on. File::RsyncP won't
understand the data stream when --numeric-ids is omitted.

So I don't have a good recommendation, other than making sure that
native rsync runs cleanly, and increasing the RsyncLogLevel and
looking carefully at the output, as Marlin suggested.

Craig


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 02/18 11:52 , Craig Barratt wrote:

You should definitely have --numeric-ids turned on. File::RsyncP won't
understand the data stream when --numeric-ids is omitted.

hmm. ok. that seemed to solve the problem (at least with the debian box I'm
trying to back up... still testing the developer's workstation issue, where
I didn't change that setting). obviously some flaw in my testing somewhere
(or else the upgrade of File:RsyncP fixed it).

thanks much! I owe you a beer sometime if you're in the Minneapolis area.

Carl Soderstrom.
--
Systems Administrator
Real-Time Enterprises
www.real-time.com

Carl Wilhelm Soderstrom writes:

You should definitely have --numeric-ids turned on. File::RsyncP won't
understand the data stream when --numeric-ids is omitted.

hmm. ok. that seemed to solve the problem (at least with the debian box I'm
trying to back up... still testing the developer's workstation issue, where
I didn't change that setting). obviously some flaw in my testing somewhere
(or else the upgrade of File:RsyncP fixed it).

Good. The File:RsyncP upgrade certainly helped.

thanks much! I owe you a beer sometime if you're in the Minneapolis area.

Hey, I was there 2 weeks ago, but only for about 4 hours. Not sure when
I will make it next...

Craig


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 02/18 03:25 , Marlin Prowell wrote:
When I traced down the RsyncP problem (which had symptoms versy similar to
yours), I started by bumping up the debugging logging level, say like:

$Conf{RsyncLogLevel} = 6;

thanks for the advice!
tried that; and then running backuppc by hand.

after about 1/2 million files were compared and the changes transferred, I
got to this:

attribSet(dir=f%2f/home/username/vmware/kickstart, file=linux.vmdk)
attribWrite(dir=f%2f/home/username/vmware/kickstart) ->
/var/lib/backuppc/pc/samurai/new/f%2f/fhome/fusername/fvmware/fkickstart/attrib
attribSet(dir=f%2f/home/username/vmware, file=knoppix)
create d 755 500/100 4096 home/username/vmware/knoppix
makePath(/var/lib/backuppc/pc/samurai/new//f%2f/fhome/fusername/fvmware/fknoppix,
0777)
attribSet(dir=f%2f/home/username/vmware, file=knoppix)
Starting file 513059 (home/username/vmware/knoppix/knoppix-0.vmdk), blkCnt=0,
blkSize=2048, remainder=0
home/username/vmware/knoppix/knoppix-0.vmdk: size doesn't match (3770220544 vs
0)
Parent read EOF from child: fatal error!


I think this VMWare image was added about the time that backups stopped
working, so it may be at fault in some way. I don't think this VMWare image
is running; it certainly hasn't been running on every occasion that backups
have been tried.

on another note, examination of the previous working backups of this box,
indicate that there are somewhere around 800,000 files on it. (yes, over 3/4
millon files on this workstation... he's a very prolific developer who works
on a lot of Open Source projects). does this make any difference to BackupPC?


Carl Soderstrom
--
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

At 03:50 PM 2/19/2004 -0600, Carl Wilhelm Soderstrom wrote:
after about 1/2 million files were compared and the changes transferred, I
got to this:

attribSet(dir=f%2f/home/username/vmware/kickstart, file=linux.vmdk)
attribWrite(dir=f%2f/home/username/vmware/kickstart) ->
/var/lib/backuppc/pc/samurai/new/f%2f/fhome/fusername/fvmware/fkickstart/attrib
attribSet(dir=f%2f/home/username/vmware, file=knoppix)
create d 755 500/100 4096 home/username/vmware/knoppix
makePath(/var/lib/backuppc/pc/samurai/new//f%2f/fhome/fusername/fvmware/fknoppix,
0777)
attribSet(dir=f%2f/home/username/vmware, file=knoppix)
Starting file 513059 (home/username/vmware/knoppix/knoppix-0.vmdk), blkCnt=0,
blkSize=2048, remainder=0
home/username/vmware/knoppix/knoppix-0.vmdk: size doesn't match (3770220544 vs
0)
Parent read EOF from child: fatal error!


I think this VMWare image was added about the time that backups stopped
working, so it may be at fault in some way. I don't think this VMWare image
is running; it certainly hasn't been running on every occasion that backups
have been tried.

It looks like the remote rsync cannot read this file for some
reason. Permissions? Locks? What happens when you run rsync by hand and
try to copy just this file?

on another note, examination of the previous working backups of this box,
indicate that there are somewhere around 800,000 files on it. (yes, over 3/4
millon files on this workstation... he's a very prolific developer who works
on a lot of Open Source projects). does this make any difference to BackupPC?

It can, but indirectly. BackupPC needs to build internal tables of all the
files to back up. Essentially, the remote rsync first sends a directory
listing of all the files on the remote machine that the backup perl script
must save. You may need a *lot* of RAM for this to work for so many
files. Watch the size of the perl process on your BackupPC machine while
it is running during the first portion of the backup. You'll be
amazed. But this isn't really BackupPC's fault - it has to do this because
of the rsync protocol.

If your backup machine doesn't have lots of RAM, you could start swapping,
especially if there are other backups going on simultaneously. As a
remedy, you could try backing up one disk partition at a time on the remote
machine to help alleviate this. You could use the --exclude= or
--one-file-system options in the $Conf{RsyncArgs} parameter (but read the
documentation, as this is not enough information to do it right).

But, if your machine is not swapping, the number of files should not be a
problem. You *are* however, doing an excellent job of testing malloc(),
realloc(), and free() on your backup machine 8^)


Marlin Prowell
Cadalog, Inc.




-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

I think this VMWare image was added about the time that backups stopped
working, so it may be at fault in some way. I don't think this VMWare image
is running; it certainly hasn't been running on every occasion that backups
have been tried.

It looks like the remote rsync cannot read this file for some
reason.

I was monitoring bandwidth usage while this was running; and it seemed to be
happily copying at a couple of MB/sec, until it got the EOF error, at which
point it stopped, and the BackupPC process became a zombie.

Permissions? Locks? What happens when you run rsync by hand and
try to copy just this file?

when I run bare rsync, it copies the file just fine.

If your backup machine doesn't have lots of RAM, you could start swapping,

it's got 256MB (it's just a testbed box... was going to be a second Amanda
server until I started using BackupPC on it. Amanda is not memory-intensive,
unlike BackupPC); and yes, it's hitting swap pretty hard. the box we're
getting to replace it as our production BackupPC server, will have 2GB RAM.

especially if there are other backups going on simultaneously. As a
remedy, you could try backing up one disk partition at a time on the remote
machine to help alleviate this. You could use the --exclude= or
--one-file-system options in the $Conf{RsyncArgs} parameter (but read the
documentation, as this is not enough information to do it right).

I can figure it out. I'm reasonably familiar with exclude arguments.

The way I actually prefer to do rsync backups, is:
- have a special 'backup' user on the client machine
- the 'backup' user is allowed to run rsync as root without a password, via
sudo
- the 'backup' user is logged into via an RSA key, and the
.ssh/authorized_keys file which contains the key, only allows the specific
sudo+rsync command to be run. (this is where the actual exclude patterns
go). so it looks something like:
# cat authorized_keys
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,command="sudo
/usr/bin/rsync --server --sender -logDtpr --exclude='/proc' --exclude='/mnt'
--exclude='/staff' --delete --blocking-io . /" ssh-dss <insert rest of the
key here>== rsync backup login

this way, there's no remote login as root, and the user who does the rsync,
can only do the one rsync command. I was doing this with my own rsync
scripts before I found backuppc.

and yes, I've tried debugging my problems by just doing logins as root. (tho
I'll have to try it again).

But, if your machine is not swapping, the number of files should not be a
problem. You *are* however, doing an excellent job of testing malloc(),
realloc(), and free() on your backup machine 8^)

good to know.

Carl Soderstrom.
--
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

At 01:52 PM 2/20/2004 -0600, Carl Wilhelm Soderstrom wrote:
The way I actually prefer to do rsync backups, is:
- have a special 'backup' user on the client machine
- the 'backup' user is allowed to run rsync as root without a password, via
sudo
- the 'backup' user is logged into via an RSA key, and the
.ssh/authorized_keys file which contains the key, only allows the specific
sudo+rsync command to be run. (this is where the actual exclude patterns
go). so it looks something like:
# cat authorized_keys
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,command="sudo
/usr/bin/rsync --server --sender -logDtpr --exclude='/proc' --exclude='/mnt'
--exclude='/staff' --delete --blocking-io . /" ssh-dss <insert rest of the
key here>== rsync backup login

this way, there's no remote login as root, and the user who does the rsync,
can only do the one rsync command. I was doing this with my own rsync
scripts before I found backuppc.

This is clever. When I have a little time, I'll try implementing something
like this.

It doesn't look to me like your sudo command is completely correct. If you
look at the beginning of a log file, BackupPC prints out the rsync command
it is trying to execute. I've looked at one of my logs, and compared it to
your sudo command. Your sudo command looks like it is missing
"--numeric-ids" and "--block-size=2048". If you read the rsync
documentation, it seems like getting either of these wrong could confuse
File::RsyncP. Just don't ask me to explain why it has worked up to now.



Marlin Prowell
Cadalog, Inc.




-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 02/20 01:24 , Marlin Prowell wrote:
At 01:52 PM 2/20/2004 -0600, Carl Wilhelm Soderstrom wrote:
The way I actually prefer to do rsync backups, is:
- have a special 'backup' user on the client machine
- the 'backup' user is allowed to run rsync as root without a password, via
sudo
- the 'backup' user is logged into via an RSA key, and the
.ssh/authorized_keys file which contains the key, only allows the specific
sudo+rsync command to be run. (this is where the actual exclude patterns
go). so it looks something like:
# cat authorized_keys
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,command="sudo
/usr/bin/rsync --server --sender -logDtpr --exclude='/proc' --exclude='/mnt'
--exclude='/staff' --delete --blocking-io . /" ssh-dss <insert rest of the
key here>== rsync backup login

this way, there's no remote login as root, and the user who does the rsync,
can only do the one rsync command. I was doing this with my own rsync
scripts before I found backuppc.

This is clever. When I have a little time, I'll try implementing something
like this.

It doesn't look to me like your sudo command is completely correct. If you
look at the beginning of a log file, BackupPC prints out the rsync command
it is trying to execute. I've looked at one of my logs, and compared it to
your sudo command. Your sudo command looks like it is missing
"--numeric-ids" and "--block-size=2048". If you read the rsync
documentation, it seems like getting either of these wrong could confuse
File::RsyncP. Just don't ask me to explain why it has worked up to now.

ok. good catch. I'll try adding those.
I might have taken this from the actions of my previous scripts; or I might
have just looked at a 'ps ax' of a running rsync client after backuppc
started. don't remember.

thanks for the advice on the log file!

Carl Soderstrom.
--
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Carl Wilhelm Soderstrom writes:

On 02/18 03:25 , Marlin Prowell wrote:
When I traced down the RsyncP problem (which had symptoms versy similar to
yours), I started by bumping up the debugging logging level, say like:

$Conf{RsyncLogLevel} = 6;

thanks for the advice!
tried that; and then running backuppc by hand.

after about 1/2 million files were compared and the changes transferred, I
got to this:

attribSet(dir=f%2f/home/username/vmware/kickstart, file=linux.vmdk)
attribWrite(dir=f%2f/home/username/vmware/kickstart) ->
/var/lib/backuppc/pc/samurai/new/f%2f/fhome/fusername/fvmware/fkickstart/attrib
attribSet(dir=f%2f/home/username/vmware, file=knoppix)
create d 755 500/100 4096 home/username/vmware/knoppix
makePath(/var/lib/backuppc/pc/samurai/new//f%2f/fhome/fusername/fvmware/fknoppix,
0777)
attribSet(dir=f%2f/home/username/vmware, file=knoppix)
Starting file 513059 (home/username/vmware/knoppix/knoppix-0.vmdk), blkCnt=0,
blkSize=2048, remainder=0
home/username/vmware/knoppix/knoppix-0.vmdk: size doesn't match (3770220544 vs
0)
Parent read EOF from child: fatal error!

I'm curious whether the problem is specific to this one file. This
file is 3.7GB, which should be ok, but it might be exposing an issue
with large files (> 2GB).

Could you change, eg, $Conf{BackupFilesOnly} or $Conf{RsyncShareName}
to backup just this directory, and try again? Then we can tell
whether it is a large file problem (ie: deterministic, and hopefully
repeatable) or some less-repeatable protocol problem that only happens
when transferring large numbers of files.

Craig


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Craig Barratt writes:

Carl Wilhelm Soderstrom writes:

On 02/18 03:25 , Marlin Prowell wrote:
When I traced down the RsyncP problem (which had symptoms versy similar to
yours), I started by bumping up the debugging logging level, say like:

$Conf{RsyncLogLevel} = 6;

thanks for the advice!
tried that; and then running backuppc by hand.

after about 1/2 million files were compared and the changes transferred, I
got to this:

attribSet(dir=f%2f/home/username/vmware/kickstart, file=linux.vmdk)
attribWrite(dir=f%2f/home/username/vmware/kickstart) ->
/var/lib/backuppc/pc/samurai/new/f%2f/fhome/fusername/fvmware/fkickstart/attrib
attribSet(dir=f%2f/home/username/vmware, file=knoppix)
create d 755 500/100 4096 home/username/vmware/knoppix
makePath(/var/lib/backuppc/pc/samurai/new//f%2f/fhome/fusername/fvmware/fknoppix,
0777)
attribSet(dir=f%2f/home/username/vmware, file=knoppix)
Starting file 513059 (home/username/vmware/knoppix/knoppix-0.vmdk), blkCnt=0,
blkSize=2048, remainder=0
home/username/vmware/knoppix/knoppix-0.vmdk: size doesn't match (3770220544 vs
0)
Parent read EOF from child: fatal error!

I'm curious whether the problem is specific to this one file. This
file is 3.7GB, which should be ok, but it might be exposing an issue
with large files (> 2GB).

Could you change, eg, $Conf{BackupFilesOnly} or $Conf{RsyncShareName}
to backup just this directory, and try again? Then we can tell
whether it is a large file problem (ie: deterministic, and hopefully
repeatable) or some less-repeatable protocol problem that only happens
when transferring large numbers of files.

Nick Knight just reported a similar problem (child exiting on
a >2GB file with rsync) and it turns out his installed perl
wasn't built with large file support.

What output do you get from:

perl -V | egrep large

Is it "uselargefiles=define" or "uselargefiles=undef"?

If it's the latter then that explains your problem; you will
need to rebuild perl with large file support enabled.

Craig


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

On 02/22 02:47 , Craig Barratt wrote:
What output do you get from:

perl -V | egrep large

Is it "uselargefiles=define" or "uselargefiles=undef"?

If it's the latter then that explains your problem; you will
need to rebuild perl with large file support enabled.

just now had a moment to try this..

that's the problem. will try rebuilding perl and see if I can straighten
that out.

thanks!

Carl.
--
Systems Administrator
Real-Time Enterprises
www.real-time.com

Just to let you all know; building perl with '-Dlargefiles' (instead of
'-Ulargefiles') solved the problem, and it now backs up 2GB+ files just
fine.

thanks a bundle!

Carl Soderstrom.
--
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/