Backup Central

Hi,
Yesterday morning one of my DLEs failed the estimate phase, and it
happened again this morning. The host is running Debian Linux with
software RAID devices (md). Since I use a combination of dump and tar
among all my clients, I use disk names in my disklist wherever possible,
to keep everything consistent. On the hosts that use tar, AMANDA
figures out the corresponding directory and everyone's happy. For
safety, we'll call the host in question "<HOST>".

So, on <HOST> the failing DLE is disk "md0" which corresponds to /usr.
Sendsize debug reports:

sendsize[29880]: estimate time for md0 level 1: 0.464
sendsize[29880]: no size line match in /bin/tar output for "md0"

"Aha!" I say. Something's happening to tar and it's not completing the
estimate. Running the tar command from a shell produces a segfault due
to a "permission denied" error on one of the directories below /usr:

backup < at > <HOST>:/usr$ /bin/tar --create --file /dev/null \
--directory /usr \
--one-file-system --listed-incremental \
/var/lib/amanda/gnutar-lists/<HOST>md0_1.new --sparse \
--ignore-failed-read --totals \
--exclude-from /tmp/amanda/sendsize.md0.20040609041435000.exclude .
/bin/tar: ./local/yoda: Cannot savedir: Permission denied
Segmentation fault


Now that's weird. Shouldn't tar ignore such a permissions error, or at
least just warn on it and continue? I have an identical host using the
same version of tar (GNU 1.13.25) with the same directory structure and
permissions. It has no trouble with the estimate and AMANDA does not
indicate any strangeness with backing up that DLE. Additionally, it has
been working for over a year on <HOST> and just started failing
yesterday. The directory entry itself has not been modified in almost a
year. I cannot think of what else could cause such a change in tar's
behavior.

Anyone have any ideas for further investigation?
Thanks,
Eric

On Fri, Jun 11, 2004 at 08:35:19AM -0400, Eric Sproul wrote:
Hi,
...

"Aha!" I say. Something's happening to tar and it's not completing the
estimate. Running the tar command from a shell produces a segfault due
to a "permission denied" error on one of the directories below /usr:

backup < at > <HOST>:/usr$ /bin/tar --create --file /dev/null \
--directory /usr \
--one-file-system --listed-incremental \
/var/lib/amanda/gnutar-lists/<HOST>md0_1.new --sparse \
--ignore-failed-read --totals \
--exclude-from /tmp/amanda/sendsize.md0.20040609041435000.exclude .
/bin/tar: ./local/yoda: Cannot savedir: Permission denied
Segmentation fault


Now that's weird. Shouldn't tar ignore such a permissions error, or at
least just warn on it and continue? I have an identical host using the
same version of tar (GNU 1.13.25) with the same directory structure and
permissions. It has no trouble with the estimate and AMANDA does not
indicate any strangeness with backing up that DLE. Additionally, it has
been working for over a year on <HOST> and just started failing
yesterday. The directory entry itself has not been modified in almost a
year. I cannot think of what else could cause such a change in tar's
behavior.

Anyone have any ideas for further investigation?

Well, I feel it shouldn't segfault under any conditions.

Just two considerations, is "/bin/tar" the one amanda uses?
And I'm pretty sure that when run by amanda, tar is setuid root.
It does most things as the amanda-user, then invokes tar with
a setuid program called "runtar". You might check that perms
on runtar are unchanged.

--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
4455 Province Line Road (609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)

On Fri, 2004-06-11 at 10:44, Jon LaBadie wrote:
Well, I feel it shouldn't segfault under any conditions.

Hi Jon,
I completely agree.


Just two considerations, is "/bin/tar" the one amanda uses?

Yes. I copied the command directly from the sendsize debug, but I was
confused because it mentions both /bin/tar and /usr/lib/amanda/runtar:

sendsize[29880]: time 0.500: getting size via gnutar for md0 level 0
sendsize[29880]: time 0.501: spawning /usr/lib/amanda/runtar in pipeline
sendsize[29880]: argument list: /bin/tar (... rest of the options)

I found that when I tested from the command line I needed to leave out
"/bin/tar". See below.

And I'm pretty sure that when run by amanda, tar is setuid root.
It does most things as the amanda-user, then invokes tar with
a setuid program called "runtar". You might check that perms
on runtar are unchanged.

This host uses the Debian package, amanda-client_2.4.4p2-1. The runtar
binary looks correct. I installed this version of amanda-client on
2/25, so a mod-date of 2/15 is reasonable, as we are tracking
Debian-sid.

# ls -l /usr/lib/amanda/runtar
-rwsr-xr-- 1 root backup 4716 Feb 15 21:44 /usr/lib/amanda/runtar

I tried as user "backup" running the estimate command using runtar
instead, and still got a segfault.

backup < at > <HOST>:~$ /usr/lib/amanda/runtar --create --file /dev/null \
--directory /usr --one-file-system --listed-incremental \
/var/lib/amanda/gnutar-lists/dhcp02md0_1.new --sparse \
--ignore-failed-read --totals --exclude-from \
/tmp/amanda/sendsize.md0.20040609041435000.exclude .
Segmentation fault

But... it didn't mention the directory permissions this time, as runtar
is suid-root (which you noted). The segfault seems to be coming from
somewhere else then. I guess I'll have to dig deeper into what is
actually being backed up. Maybe one of my files is causing this.

Thanks,
Eric

Hi Eric,

I had a similar problem with several versions of gtar (1.13, 1.13.25 and
1.14) on Solaris 8. By running the gtar command used by amanda
interactively with the -v flag, I found that gtar was SEGV-ing on a
specific user directory. When I added that directory to the amanda
exclude file, gtar worked fine, and the amanda backup succeeded.

This didn't really solve the problem, but it gives me some comfort that
it was caused by gtar, and not amanda. I suppose running a debug version
of gtar could shed some light on what is causing the SEGV.

HTH,
Marc.


On Fri, 2004-06-11 at 11:01, Eric Sproul wrote:
On Fri, 2004-06-11 at 10:44, Jon LaBadie wrote:
Well, I feel it shouldn't segfault under any conditions.

Hi Jon,
I completely agree.


Just two considerations, is "/bin/tar" the one amanda uses?

Yes. I copied the command directly from the sendsize debug, but I was
confused because it mentions both /bin/tar and /usr/lib/amanda/runtar:

sendsize[29880]: time 0.500: getting size via gnutar for md0 level 0
sendsize[29880]: time 0.501: spawning /usr/lib/amanda/runtar in pipeline
sendsize[29880]: argument list: /bin/tar (... rest of the options)

I found that when I tested from the command line I needed to leave out
"/bin/tar". See below.

And I'm pretty sure that when run by amanda, tar is setuid root.
It does most things as the amanda-user, then invokes tar with
a setuid program called "runtar". You might check that perms
on runtar are unchanged.

This host uses the Debian package, amanda-client_2.4.4p2-1. The runtar
binary looks correct. I installed this version of amanda-client on
2/25, so a mod-date of 2/15 is reasonable, as we are tracking
Debian-sid.

# ls -l /usr/lib/amanda/runtar
-rwsr-xr-- 1 root backup 4716 Feb 15 21:44 /usr/lib/amanda/runtar

I tried as user "backup" running the estimate command using runtar
instead, and still got a segfault.

backup < at > <HOST>:~$ /usr/lib/amanda/runtar --create --file /dev/null \
--directory /usr --one-file-system --listed-incremental \
/var/lib/amanda/gnutar-lists/dhcp02md0_1.new --sparse \
--ignore-failed-read --totals --exclude-from \
/tmp/amanda/sendsize.md0.20040609041435000.exclude .
Segmentation fault

But... it didn't mention the directory permissions this time, as runtar
is suid-root (which you noted). The segfault seems to be coming from
somewhere else then. I guess I'll have to dig deeper into what is
actually being backed up. Maybe one of my files is causing this.

Thanks,
Eric

On Tue, 2004-06-15 at 13:22, Marc Langlois wrote:
This didn't really solve the problem, but it gives me some comfort that
it was caused by gtar, and not amanda. I suppose running a debug version
of gtar could shed some light on what is causing the SEGV.

Hi Marc,
Actually it turned out to be a larger system problem. Something
happened to either main memory or swap and we had much bigger problems
that gtar segfaulting. Those have been remedied now so we will see how
the backup goes tomorrow morning.

Thanks,
Eric