Backup Central

Hello...

I ran into a couple bugs with amrestore failing with split dumps and
amcrypt-ossl-asym. I cannot test without amcrypt-ossl-asym so YMMV.
The systems are all RHEL 5.4 x86_64 and amanda version 2.6.1p2 on the
server side and 2.6.0p2 on the client side. (I also realized that I've
been running amanda for over 9 years


When running amrestore on a backup with only one chunk, after the
restore is done, amrestore keeps going. This is annoying, but
everything does succeed.


[amanda < at > gobo RESTOREDIR]$ /usr/sbin/amrestore -f 86 -b256k -p /dev/nst0
some.server / | /sbin/restore -ivf -
Verify tape and initialize maps
Input is from a local file/pipe
Restoring from tape 000127L4 starting with file 86.
amrestore: 86: skipping FILE: date 20100216000502 host some.server
disk / lev 0 comp .gz program /sbin/dump crypt enc
server_encrypt /usr/sbin/amcrypt-ossl-asym server_decrypt_option -d
amrestore: 87: restoring FILE: date 20100216000502 host some.server
disk / lev 0 comp .gz program /sbin/dump crypt enc
server_encrypt /usr/sbin/amcrypt-ossl-asym server_decrypt_option -d
Input block size is 32
Dump date: Tue Feb 16 05:17:37 2010
Dumped from: the epoch
Level 0 dump of an unlisted file system on some.server:/dev/root
Label: /
Extract directories from tape
Initialize symbol table.
/sbin/restore > add /etc/passwd
Make node ./etc
/sbin/restore > extract
Extract requested files
extract file ./etc/passwd
Add links
Set directory mode, owner, and times.
set owner/mode for '.'? [yn] y
/sbin/restore: .: EA set security.selinux:system_u:object_r:root_t:s0
failed: Operation not permitted
/sbin/restore: ./etc: EA set security.selinux:system_u:object_r:etc_t:s0
failed: Operation not permitted
/sbin/restore > quit

gzip: stdout: Broken pipe
error writing output file
amrestore: 88: skipping FILE: date 20100216000502 host other.server
disk / lev 0 comp .gz program /sbin/dump crypt enc
server_encrypt /usr/sbin/amcrypt-ossl-asym server_decrypt_option -d
...
...



But when there are multiple chunks, the restore fails completely.

[amanda < at > gobo RESTOREDIR]$ /usr/sbin/amrestore -f 62 -b256k -p /dev/nst0
Split.Server /data | /sbin/restore -ivf -
Verify tape and initialize maps
Input is from a local file/pipe
Restoring from tape 000127L4 starting with file 62.
amrestore: 62: skipping split dumpfile: date 20100216000502 host
other.server1 disk / part 3/3 lev 0 comp .gz program /sbin/dump crypt
enc server_encrypt /usr/sbin/amcrypt-ossl-asym server_decrypt_option -d
amrestore: 63: restoring split dumpfile: date 20100216000502 host
Split.Server disk /data part 1/3 lev 1 comp .gz program /sbin/dump crypt
enc server_encrypt /usr/sbin/amcrypt-ossl-asym server_decrypt_option -d
Checksum error 16412104523, inode 0 file (null)
/sbin/restore: Tape is not a dump tape
Error writing fd 1: Broken pipe
amrestore: 64: restoring split dumpfile: date 20100216000502 host
Split.Server disk /data part 2/3 lev 1 comp .gz program /sbin/dump crypt
enc server_encrypt /usr/sbin/amcrypt-ossl-asym server_decrypt_option -d
Error writing fd 1: Broken pipe
amrestore: 65: restoring split dumpfile: date 20100216000502 host
Split.Server disk /data part 3/3 lev 1 comp .gz program /sbin/dump crypt
enc server_encrypt /usr/sbin/amcrypt-ossl-asym server_decrypt_option -d
Error writing fd 1: Broken pipe
amrestore: 66: skipping split dumpfile: date 20100216000502 host
other.server2 disk /data part 1/3 lev 1 comp .gz program /bin/tar crypt
enc server_encrypt /usr/sbin/amcrypt-ossl-asym server_decrypt_option -d




For the archives, however this does work:
/usr/sbin/amrestore -f 62 -b256k /dev/nst0 server /data

[amanda < at > gobo RESTOREDIR]$ cat server._data.20100216000502.1.{1,2,3}
| /usr/sbin/amcrypt-ossl-asym -d | gzip --decompress --to-stdout
| /sbin/restore -ivf -

Verify tape and initialize maps
Input is from a local file/pipe
Input block size is 32
Dump date: Tue Feb 16 00:06:35 2010
Dumped from: Tue Feb 9 00:08:38 2010
Level 1 dump of /data on server:/dev/mapper/mpath3
Label: md-data
Extract directories from tape
Initialize symbol table.
/sbin/restore >



amfetchdump does not seem to have this problem, but you cannot run
amfetchdump while amdump is running (which would be a cool feature

I suspect something in amrestore does not like the 'part X/Y' text. I
looked at the source, but nothing jumped out at me as the cause.




--
Christopher McCrory
"The guy that keeps the servers running"

chrismcc < at > pricegrabber.com
http://www.pricegrabber.com

Let's face it, there's no Hollow Earth, no robots, and
no 'mute rays.' And even if there were, waxed paper is
no defense. I tried it. Only tinfoil works.

On Wed, Feb 24, 2010 at 5:08 PM, Christopher <chrismcc < at > pricegrabber.com> wrote:
amfetchdump does not seem to have this problem, but you cannot run
amfetchdump while amdump is running (which would be a cool feature

We're working on that. In fact, I think it should be doable in 3.1.
I'll make a note to test it before the release.

I suspect something in amrestore does not like the 'part X/Y' text.  I
looked at the source, but nothing jumped out at me as the cause.

Your first case, of amrestore trucking right on after finding the dump
it was looking for, is "expected" behavior - amrestore needs to scan
the whole tape, in case there are other dumps matching some.server /.

Amrestore is really not very good at re-assembling dumps. I suspect
amrestore is outputting something to stdout despite the use of -p,
which is getting mixed into the datastream. You could try the -p
command piped to a file instead of to /bin/restore. Since it looks
like restore failed its checksum on the first bytes in the file, I
expect a hex editor would show the offending message in the first few
bytes of the piped output.

Dustin

--
Open Source Storage Engineer
http://www.zmanda.com

Hello...

On Wed, 2010-02-24 at 17:33 -0600, Dustin J. Mitchell wrote:
On Wed, Feb 24, 2010 at 5:08 PM, Christopher <chrismcc < at > pricegrabber.com> wrote:
amfetchdump does not seem to have this problem, but you cannot run
amfetchdump while amdump is running (which would be a cool feature

We're working on that. In fact, I think it should be doable in 3.1.
I'll make a note to test it before the release.


yea!

I suspect something in amrestore does not like the 'part X/Y' text. I
looked at the source, but nothing jumped out at me as the cause.

Your first case, of amrestore trucking right on after finding the dump
it was looking for, is "expected" behavior - amrestore needs to scan
the whole tape, in case there are other dumps matching some.server /.


Was it always that way? I haven't used amrestore since amfetchdump came
out.


Amrestore is really not very good at re-assembling dumps. I suspect
amrestore is outputting something to stdout despite the use of -p,
which is getting mixed into the datastream. You could try the -p
command piped to a file instead of to /bin/restore. Since it looks
like restore failed its checksum on the first bytes in the file, I
expect a hex editor would show the offending message in the first few
bytes of the piped output.


I'll spend some time tracking down exactly what is not being stripped
out. Maybe stderr is being included? My C programing skills are weak,
so I might not be able to provide a patch, but I can do the legwork to
find out exactly where the problem data is. Currently my amdump runs
last 19 to 23 hours. I got a third tape drive for the jukebox
specifically to be able to do restores without killing the running
amdump. So being able to run amrestore without a workaround would be
good (amfetchdump --yes-I-know-what-I-am-doing --ignore-running-amdump
--use-drive /dev/nst2 would be better . In the example I gave the
data was only 25G, but the same workaround would be "interesting" with a
-gt 1Tb restore.


Dustin



--
Christopher McCrory
"The guy that keeps the servers running"

chrismcc < at > pricegrabber.com
http://www.pricegrabber.com

Let's face it, there's no Hollow Earth, no robots, and
no 'mute rays.' And even if there were, waxed paper is
no defense. I tried it. Only tinfoil works.

On Wed, Feb 24, 2010 at 6:40 PM, Christopher <chrismcc < at > pricegrabber.com> wrote:
Was it always that way?  I haven't used amrestore since amfetchdump came
out.

Yes. Or, if not, it was a bug

I'll spend some time tracking down exactly what is not being stripped
out.  Maybe stderr is being included?  My C programing skills are weak,
so I might not be able to provide a patch, but I can do the legwork to
find out exactly where the problem data is.  Currently my amdump runs
last 19 to 23 hours.  I got a third tape drive for the jukebox
specifically to be able to do restores without killing the running
amdump.  So being able to run amrestore without a workaround would be
good (amfetchdump --yes-I-know-what-I-am-doing --ignore-running-amdump
--use-drive /dev/nst2  would be better .  In the example I gave the
data was only 25G, but the same workaround would be "interesting" with a
-gt 1Tb restore.

Stderr would not be included in the shell pipeline you pasted. If you
can figure out what message is being included, you can probably
comment it out. amrestore has been completely rewritten in 3.1, so
there's no sense producing a patch.

amidxtaped is careful to only try to lock the config if the tape drive
it's using is the same as the default for the config. I thought
amfetchdump did the same thing, but perhaps not. Anyway, amfetchdump
has also been rewritten (all of the recovery tools have been rewritte
- amidxtaped should be committed in a day or two once I sort out some
Amanda-2.4.5-compatibility issues), and now does not try to lock the
config at all.

By the way, the new code manages to not stomp on its own feet by
reserving particular volumes and/or drives in the changer, rather than
locking the entire config.

Dustin

P.S. You give yourself away as a shell programmer when you write "-gt 1Tb"

--
Open Source Storage Engineer
http://www.zmanda.com