Welcome! » Log In » Create A New Profile

Amtapetype wrongly reporting compression?

Posted by Luc Lalonde 
Luc Lalonde
Amtapetype wrongly reporting compression?
January 11, 2018 11:59AM
Hello Folks,

I'm wondering if there's a bug with 'Amtapetype' (version 3.5.1).

We're migrating from LTO5 to LTO7 and I'm getting strange results when I
run 'amtapetype'.

Here's what I get after roughly 35 hours:

define tapetype LTO7 {
comment "Created by amtapetype; compression enabled"
length 5874932832 kbytes
filemark 1813 kbytes
speed 118310 kps
blocksize 32 kbytes
}

Why is it saying that it can write roughly 6Tb if it's supposedly
hardware compressed?  LTO7 is supposed to give a compression ration of
2.5 to 1.

I've disabled compression... Here's the ouptut of 'tapeinfo:

[root@ulysses ~]# tapeinfo -f /dev/sg12
Product Type: Tape Drive
Vendor ID: 'IBM '
Product ID: 'ULTRIUM-HH7 '
Revision: 'G9Q1'
Attached Changer API: No
SerialNumber: '123456789A'
MinBlock: 1
MaxBlock: 8388608
SCSI ID: 4
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: 0x78
Density Code: 0x5c
BlockSize: 32768
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0xff
DeCompType: 0xff
Block Position: 2
Partition 0 Remaining Kbytes: -1
Partition 0 Size in Kbytes: -1
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 3

Am I missing something?

Thanks!|

--
Luc Lalonde, analyste
-----------------------------
Département de génie informatique:
École polytechnique de MTL
(514) 340-4711 x5049
Luc.Lalonde@polymtl.ca
-----------------------------
This message was imported via the External PhorumMail Module
Luc Lalonde
Re: Amtapetype wrongly reporting compression?
January 11, 2018 12:00PM
I sent the wrong output in the last email.... Here's the correct 'tapeinfo':

Product Type: Tape Drive
Vendor ID: 'IBM     '
Product ID: 'ULTRIUM-HH7     '
Revision: 'G9Q1'
Attached Changer API: No
SerialNumber: '123456789A'
MinBlock: 1
MaxBlock: 8388608
SCSI ID: 4
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: 0x78
Density Code: 0x5c
BlockSize: 32768
DataCompEnabled: no
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0xff
DeCompType: 0xff
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: -1
Partition 0 Size in Kbytes: -1
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 3

Further, here's my '/etc/stinit.def':

manufacturer="IBM" model = "ULTRIUM-HH7" {
scsi2logical=1
can-bsr=1
auto-lock=1
two-fms=0
drive-buffering=1
buffer-writes
read-ahead=1
async-writes=1
can-partitions=1
fast-mteom=0
sysv=1
timeout=180
long-timeout=14400
mode1 blocksize=32768 compression=1
mode2 blocksize=32768 compression=0
mode3 disabled=1
mode4 disabled=1

Sorry, about that!

On 2018-01-11 02:43 PM, Luc Lalonde wrote:
> Hello Folks,
>
> I'm wondering if there's a bug with 'Amtapetype' (version 3.5.1).
>
> We're migrating from LTO5 to LTO7 and I'm getting strange results when
> I run 'amtapetype'.
>
> Here's what I get after roughly 35 hours:
>
> define tapetype LTO7 {
>     comment "Created by amtapetype; compression enabled"
>     length 5874932832 kbytes
>     filemark 1813 kbytes
>     speed 118310 kps
>     blocksize 32 kbytes
> }
>
> Why is it saying that it can write roughly 6Tb if it's supposedly
> hardware compressed?  LTO7 is supposed to give a compression ration of
> 2.5 to 1.
>
> I've disabled compression... Here's the ouptut of 'tapeinfo:
>
> [root@ulysses ~]# tapeinfo -f /dev/sg12
> Product Type: Tape Drive
> Vendor ID: 'IBM     '
> Product ID: 'ULTRIUM-HH7     '
> Revision: 'G9Q1'
> Attached Changer API: No
> SerialNumber: '123456789A'
> MinBlock: 1
> MaxBlock: 8388608
> SCSI ID: 4
> SCSI LUN: 0
> Ready: yes
> BufferedMode: yes
> Medium Type: 0x78
> Density Code: 0x5c
> BlockSize: 32768
> DataCompEnabled: yes
> DataCompCapable: yes
> DataDeCompEnabled: yes
> CompType: 0xff
> DeCompType: 0xff
> Block Position: 2
> Partition 0 Remaining Kbytes: -1
> Partition 0 Size in Kbytes: -1
> ActivePartition: 0
> EarlyWarningSize: 0
> NumPartitions: 0
> MaxPartitions: 3
>
> Am I missing something?
>
> Thanks!|
>

--
Luc Lalonde, analyste
-----------------------------
Département de génie informatique:
École polytechnique de MTL
(514) 340-4711 x5049
Luc.Lalonde@polymtl.ca
-----------------------------
This message was imported via the External PhorumMail Module
Jean-Louis Martineau
Re: Amtapetype wrongly reporting compression?
January 11, 2018 12:59PM
Luc,

amtapetype use speed heuristic to detect if the drive is in compressed
mode, it might not be good for newer drives.

Why you didn't post the complete amtapetype output? I can't tell you why
the heuristic is bad without those numbers.

The default block size of 32k was good 15 years ago, but it is really
bad for new drives. You should really think about increasing it a lot.

Jean-Louis

On 11/01/18 02:56 PM, Luc Lalonde wrote:
> I sent the wrong output in the last email.... Here's the correct
> 'tapeinfo':
>
> Product Type: Tape Drive
> Vendor ID: 'IBM '
> Product ID: 'ULTRIUM-HH7 '
> Revision: 'G9Q1'
> Attached Changer API: No
> SerialNumber: '123456789A'
> MinBlock: 1
> MaxBlock: 8388608
> SCSI ID: 4
> SCSI LUN: 0
> Ready: yes
> BufferedMode: yes
> Medium Type: 0x78
> Density Code: 0x5c
> BlockSize: 32768
> DataCompEnabled: no
> DataCompCapable: yes
> DataDeCompEnabled: yes
> CompType: 0xff
> DeCompType: 0xff
> BOP: yes
> Block Position: 0
> Partition 0 Remaining Kbytes: -1
> Partition 0 Size in Kbytes: -1
> ActivePartition: 0
> EarlyWarningSize: 0
> NumPartitions: 0
> MaxPartitions: 3
>
> Further, here's my '/etc/stinit.def':
>
> manufacturer="IBM" model = "ULTRIUM-HH7" {
> scsi2logical=1
> can-bsr=1
> auto-lock=1
> two-fms=0
> drive-buffering=1
> buffer-writes
> read-ahead=1
> async-writes=1
> can-partitions=1
> fast-mteom=0
> sysv=1
> timeout=180
> long-timeout=14400
> mode1 blocksize=32768 compression=1
> mode2 blocksize=32768 compression=0
> mode3 disabled=1
> mode4 disabled=1
>
> Sorry, about that!
>
> On 2018-01-11 02:43 PM, Luc Lalonde wrote:
>> Hello Folks,
>>
>> I'm wondering if there's a bug with 'Amtapetype' (version 3.5.1).
>>
>> We're migrating from LTO5 to LTO7 and I'm getting strange results
>> when I run 'amtapetype'.
>>
>> Here's what I get after roughly 35 hours:
>>
>> define tapetype LTO7 {
>> comment "Created by amtapetype; compression enabled"
>> length 5874932832 kbytes
>> filemark 1813 kbytes
>> speed 118310 kps
>> blocksize 32 kbytes
>> }
>>
>> Why is it saying that it can write roughly 6Tb if it's supposedly
>> hardware compressed? LTO7 is supposed to give a compression ration
>> of 2.5 to 1.
>>
>> I've disabled compression... Here's the ouptut of 'tapeinfo:
>>
>> [root@ulysses ~]# tapeinfo -f /dev/sg12
>> Product Type: Tape Drive
>> Vendor ID: 'IBM '
>> Product ID: 'ULTRIUM-HH7 '
>> Revision: 'G9Q1'
>> Attached Changer API: No
>> SerialNumber: '123456789A'
>> MinBlock: 1
>> MaxBlock: 8388608
>> SCSI ID: 4
>> SCSI LUN: 0
>> Ready: yes
>> BufferedMode: yes
>> Medium Type: 0x78
>> Density Code: 0x5c
>> BlockSize: 32768
>> DataCompEnabled: yes
>> DataCompCapable: yes
>> DataDeCompEnabled: yes
>> CompType: 0xff
>> DeCompType: 0xff
>> Block Position: 2
>> Partition 0 Remaining Kbytes: -1
>> Partition 0 Size in Kbytes: -1
>> ActivePartition: 0
>> EarlyWarningSize: 0
>> NumPartitions: 0
>> MaxPartitions: 3
>>
>> Am I missing something?
>>
>> Thanks!|
>>
>
This message is the property of CARBONITE, INC. and may contain confidential or privileged information.
If this message has been delivered to you by mistake, then do not copy or deliver this message to anyone. Instead, destroy it and notify me by reply e-mail
This message was imported via the External PhorumMail Module
Gene Heskett
Re: Amtapetype wrongly reporting compression?
January 11, 2018 07:59PM
On Thursday 11 January 2018 14:43:49 Luc Lalonde wrote:

> Hello Folks,
>
> I'm wondering if there's a bug with 'Amtapetype' (version 3.5.1).
>
> We're migrating from LTO5 to LTO7 and I'm getting strange results when
> I run 'amtapetype'.
>
> Here's what I get after roughly 35 hours:
>
> define tapetype LTO7 {
> comment "Created by amtapetype; compression enabled"
> length 5874932832 kbytes
> filemark 1813 kbytes
> speed 118310 kps
> blocksize 32 kbytes
> }
>
> Why is it saying that it can write roughly 6Tb if it's supposedly
> hardware compressed?  LTO7 is supposed to give a compression ration of
> 2.5 to 1.

That can vary widely, dependent on the data. If already compressed, don't
waste your cpu's power trying to gain another 5%, 'tain't worth it.

>
> I've disabled compression... Here's the ouptut of 'tapeinfo:
>
> [root@ulysses ~]# tapeinfo -f /dev/sg12
> Product Type: Tape Drive
> Vendor ID: 'IBM '
> Product ID: 'ULTRIUM-HH7 '
> Revision: 'G9Q1'
> Attached Changer API: No
> SerialNumber: '123456789A'
> MinBlock: 1
> MaxBlock: 8388608
> SCSI ID: 4
> SCSI LUN: 0
> Ready: yes
> BufferedMode: yes
> Medium Type: 0x78
> Density Code: 0x5c
> BlockSize: 32768
> DataCompEnabled: yes
_________________________
> DataCompCapable: yes
> DataDeCompEnabled: yes
> CompType: 0xff
> DeCompType: 0xff
> Block Position: 2
> Partition 0 Remaining Kbytes: -1
> Partition 0 Size in Kbytes: -1
> ActivePartition: 0
> EarlyWarningSize: 0
> NumPartitions: 0
> MaxPartitions: 3
>
> Am I missing something?
>
> Thanks!|

Yes, you wrote that tapes label with compression enabled, so despite your
turning it off, when the drive scanned that tape to mount it, it found
the compression was enabled in a hidden header only the drive can see,
located in front of the label block, so it re-enabled it. BTDT, had fun,
finally developed a method to turn it off:

1. rewind the tape.
1a. Do NOT remove tape from drive, or cause it to read the tape other
than what I write here until after step 5.
2. read the label out to a 32k file.
3. rewind the tape.
4. Turn the compression off, probably with mt.
5. Immediately re-write that label while the tape is rewound, and the
hidden tape id block in front of the label will get rewritten too, with
that compression flag off.

6. You must do this with every tape that was previously labeled with
compression enabled else its a virus that will re-infect every tape
after the one you missed.

I do not personally believe in using drive compression for the simple
reason that it hides the true tape capacity from amanda. If you use
local compression, gzip or ???, amanda counts bytes sent down the cable
to the drive and this is already compressed data if your dumptype in use
says to, and can then know pretty precisely how much more data this tape
can hold. Otherwise its a SWAG*, and we all know just how far off that
can be under the wrong circumstances.

*SWAG, thats a Scientific Wild Assed Guess.

In any event, I've likely started a flame war over where to do the
compression, so I'll go get my nomex underwear out.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page http://geneslinuxbox.net:6309/gene
This message was imported via the External PhorumMail Module
Luc Lalonde
Re: Amtapetype wrongly reporting compression?
January 12, 2018 12:59PM
Hello Gene,

Would a variant like this:

        /usr/sbin/mtx load 1 0;
        /usr/bin/mt -f /dev/st0 compression 0;
        /usr/bin/mt -f /dev/st0 setblk 524288;
        /usr/bin/dd if=/dev/zero of=/dev/st0 bs=512k count=1;
        su amandabackup -c "/usr/sbin/amlabel -f Monthly-LTO7 000321L7
slot 1";
        /usr/sbin/mtx unload 1 0;

work?

During my tests with this, the hardware compression stays disabled when
I load a new tape.

Thanks!


On 2018-01-11 10:51 PM, Gene Heskett wrote:
> 1. rewind the tape.
> 1a. Do NOT remove tape from drive, or cause it to read the tape other
> than what I write here until after step 5.
> 2. read the label out to a 32k file.
> 3. rewind the tape.
> 4. Turn the compression off, probably with mt.
> 5. Immediately re-write that label while the tape is rewound, and the
> hidden tape id block in front of the label will get rewritten too, with
> that compression flag off.

--
Luc Lalonde, analyste
-----------------------------
Département de génie informatique:
École polytechnique de MTL
(514) 340-4711 x5049
Luc.Lalonde@polymtl.ca
-----------------------------
This message was imported via the External PhorumMail Module
Jon LaBadie
Re: Amtapetype wrongly reporting compression?
January 12, 2018 02:59PM
On Fri, Jan 12, 2018 at 03:20:51PM -0500, Luc Lalonde wrote:
> Hello Gene,
>
> Would a variant like this:
>
>         /usr/sbin/mtx load 1 0;
>         /usr/bin/mt -f /dev/st0 compression 0;
>         /usr/bin/mt -f /dev/st0 setblk 524288;
>         /usr/bin/dd if=/dev/zero of=/dev/st0 bs=512k count=1;
>         su amandabackup -c "/usr/sbin/amlabel -f Monthly-LTO7 000321L7 slot
> 1";
>         /usr/sbin/mtx unload 1 0;
>
> work?
>
> During my tests with this, the hardware compression stays disabled when I
> load a new tape.
>
The critical event is whether it remains disabled after the
amanda program opens the tape drive for reading (or R/W).
All amanda programs that deal with the tape drive do a read
of the tape header before doing anything else.

> Thanks!
>
>
> On 2018-01-11 10:51 PM, Gene Heskett wrote:
> > 1. rewind the tape.
> > 1a. Do NOT remove tape from drive, or cause it to read the tape other
> > than what I write here until after step 5.
> > 2. read the label out to a 32k file.
> > 3. rewind the tape.
> > 4. Turn the compression off, probably with mt.
> > 5. Immediately re-write that label while the tape is rewound, and the
> > hidden tape id block in front of the label will get rewritten too, with
> > that compression flag off.
>
> --
> Luc Lalonde, analyste
> -----------------------------
> Département de génie informatique:
> École polytechnique de MTL
> (514) 340-4711 x5049
> Luc.Lalonde@polymtl.ca
> -----------------------------
>>> End of included message <<<

--
Jon H. LaBadie jon@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (703) 935-6720 (C)
This message was imported via the External PhorumMail Module
Gene Heskett
Re: Amtapetype wrongly reporting compression?
January 12, 2018 04:59PM
On Friday 12 January 2018 15:20:51 Luc Lalonde wrote:

> Hello Gene,
>
> Would a variant like this:
>
>         /usr/sbin/mtx load 1 0;
>         /usr/bin/mt -f /dev/st0 compression 0;
>         /usr/bin/mt -f /dev/st0 setblk 524288;
>         /usr/bin/dd if=/dev/zero of=/dev/st0 bs=512k count=1;
>         su amandabackup -c "/usr/sbin/amlabel -f Monthly-LTO7
> 000321L7 slot 1";
>         /usr/sbin/mtx unload 1 0;
>
> work?
>
> During my tests with this, the hardware compression stays disabled
> when I load a new tape.
>
> Thanks!
>
Hey, if it works, its right. ;-)

> On 2018-01-11 10:51 PM, Gene Heskett wrote:
> > 1. rewind the tape.
> > 1a. Do NOT remove tape from drive, or cause it to read the tape
> > other than what I write here until after step 5.
> > 2. read the label out to a 32k file.
> > 3. rewind the tape.
> > 4. Turn the compression off, probably with mt.
> > 5. Immediately re-write that label while the tape is rewound, and
> > the hidden tape id block in front of the label will get rewritten
> > too, with that compression flag off.


Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page http://geneslinuxbox.net:6309/gene
This message was imported via the External PhorumMail Module
Jean-Louis Martineau
Re: Amtapetype wrongly reporting compression?
January 15, 2018 05:59AM
Luc,

You should keep the driver in variable block size:
eg . /usr/bin/mt -f /dev/st0 setblk 0

Jean-Louis

On 12/01/18 03:20 PM, Luc Lalonde wrote:
> Hello Gene,
>
> Would a variant like this:
>
> /usr/sbin/mtx load 1 0;
> /usr/bin/mt -f /dev/st0 compression 0;
> /usr/bin/mt -f /dev/st0 setblk 524288;
> /usr/bin/dd if=/dev/zero of=/dev/st0 bs=512k count=1;
> su amandabackup -c "/usr/sbin/amlabel -f Monthly-LTO7 000321L7
> slot 1";
> /usr/sbin/mtx unload 1 0;
>
> work?
>
> During my tests with this, the hardware compression stays disabled
> when I load a new tape.
>
> Thanks!
>
>
> On 2018-01-11 10:51 PM, Gene Heskett wrote:
>> 1. rewind the tape.
>> 1a. Do NOT remove tape from drive, or cause it to read the tape other
>> than what I write here until after step 5.
>> 2. read the label out to a 32k file.
>> 3. rewind the tape.
>> 4. Turn the compression off, probably with mt.
>> 5. Immediately re-write that label while the tape is rewound, and the
>> hidden tape id block in front of the label will get rewritten too, with
>> that compression flag off.
>
This message is the property of CARBONITE, INC. and may contain confidential or privileged information.
If this message has been delivered to you by mistake, then do not copy or deliver this message to anyone. Instead, destroy it and notify me by reply e-mail
This message was imported via the External PhorumMail Module
Gene Heskett
Re: Amtapetype wrongly reporting compression?
January 15, 2018 06:59AM
On Monday 15 January 2018 08:11:37 Jean-Louis Martineau wrote:

> Luc,
>
> You should keep the driver in variable block size:
> eg . /usr/bin/mt -f /dev/st0 setblk 0
>
> Jean-Louis
>
> On 12/01/18 03:20 PM, Luc Lalonde wrote:
> > Hello Gene,
> >
> > Would a variant like this:
> >
> > /usr/sbin/mtx load 1 0;
> > /usr/bin/mt -f /dev/st0 compression 0;
> > /usr/bin/mt -f /dev/st0 setblk 524288;
> > /usr/bin/dd if=/dev/zero of=/dev/st0 bs=512k count=1;
> > su amandabackup -c "/usr/sbin/amlabel -f Monthly-LTO7
> > 000321L7 slot 1";
> > /usr/sbin/mtx unload 1 0;
> >
> > work?
> >
> > During my tests with this, the hardware compression stays disabled
> > when I load a new tape.
> >
> > Thanks!
> >
> > On 2018-01-11 10:51 PM, Gene Heskett wrote:
> >> 1. rewind the tape.
> >> 1a. Do NOT remove tape from drive, or cause it to read the tape
> >> other than what I write here until after step 5.
> >> 2. read the label out to a 32k file.
> >> 3. rewind the tape.
> >> 4. Turn the compression off, probably with mt.
> >> 5. Immediately re-write that label while the tape is rewound, and
> >> the hidden tape id block in front of the label will get rewritten
> >> too, with that compression flag off.

The above quote of my previous message is Copyright 2018 by Maurice E.
Heskett.

> This message is the property of CARBONITE, INC. and may contain
> confidential or privileged information. If this message has been
> delivered to you by mistake, then do not copy or deliver this message
> to anyone. Instead, destroy it and notify me by reply e-mail

And I publically object to CARBINITE, INC's attempt to establish
copyright claims on what I write by claiming its their property. Yahoo
used to do that too. Am I going to revert to a sig establishing my own
copyright claims, something I haven't had to do in a decade or more?

I don't mind reading a "commercial" touting the product thats paying the
net bills. I am a firm believer in the TANSTAAFL principle, but what I
write is NOT CARBONITE INC.'s property.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page http://geneslinuxbox.net:6309/gene
This message was imported via the External PhorumMail Module
Sorry, only registered users may post in this forum.

Click here to login