Welcome! » Log In » Create A New Profile

Repeated ANR8341I "end-of-volume" messages for some LTO7 volumes

Posted by Skylar Thompson 
Skylar Thompson
Repeated ANR8341I "end-of-volume" messages for some LTO7 volumes
January 08, 2018 10:59AM
Content preview: Hi ADSM-L, We've been having a problem where TSM logs an ANR8341I
message ("End-of-volume reached for LTO volume nnnnnn") for a volume, but
it never actually transitions the volume from filling to full. This means
TSM will dismount the volume, but then try to remount it elsewhere. Currently
we're seeing this problem on two volumes, and it doesn't seem to be tied
to any subset of our drives. [...]

Content analysis details: (0.6 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.7 SPF_NEUTRAL SPF: sender does not match SPF record (neutral)
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
X-Barracuda-Start-Time: 1515433530
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://148.100.49.27:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at marist.edu
X-Barracuda-Scan-Msg-Size: 1534
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.46670
Rule breakdown below
pts rule name description
---- ---------------------- --------------------------------------------------

Hi ADSM-L,

We've been having a problem where TSM logs an ANR8341I message
("End-of-volume reached for LTO volume nnnnnn") for a volume, but it never
actually transitions the volume from filling to full. This means TSM will
dismount the volume, but then try to remount it elsewhere. Currently we're
seeing this problem on two volumes, and it doesn't seem to be tied to any
subset of our drives.

We had this problem previously in November, and after opening a PMR, TSM
support suggested that it was a hardware issue of some kind (drive or
library). I've updated all of our drives to the latest firmware that we can
get from Oracle (we're running SL3000s), which is G9Q2. This seemed to
solve it for a little bit, but now we're seeing it again. I'm not seeing
any obvious errors in the activity log, ACSLS, or the library logs. The
only things I can really see are the ANR8341I messages, and the times
mounted counter go up (we have some volumes that have been mounted well
over a thousand times in less than a year because of this).

We're running TSM v7.1.8.0 on RHEL6 x86_64, with a SL3000 (ACSLS) and IBM
LTO7 drives all running firmware G9Q2.

Before I open another PMR, has anyone else seen this problem? I suspect the
next step will be server tracing, which is likely to be a big performance
hit.

Thanks!

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Schofield, Neil (Contractor - Storage & Middleware, Backup & Restore)
Re: Repeated ANR8341I "end-of-volume" messages for some LTO7 volumes
January 09, 2018 06:59AM
Skylar

It's probably worth ruling out the obvious stuff first.

If the end-of-volume is reached then a new scratch tape will be required. If for some reason a new scratch tape cannot be mounted (eg there are no scratch left) to complete the writing of the spanned file/object then the store operation will fail and the previous end-of-data point on the tape will be used as the starting point the next time the tape is mounted. I've seen many occasions where a paucity of scratch tapes has resulted in the end-of-tape being reached repeatedly on the handful of remaining 'filling' volumes.

In the cases where the end-of-volume has been reached, has a scratch volume been successfully mounted subsequently to complete the operation?

Regards
Neil


Neil Schofield
IBM Spectrum Protect SME
Backup & Recovery | Storage & Middleware | Platform Technologies | Infrastructure Technology Services | Group CIO & IT Change
LLOYDS BANKING GROUP
________________________________



Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555.

Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500.

Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801.

Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority.

Halifax is a division of Bank of Scotland plc.

HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813.

This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.
This message was imported via the External PhorumMail Module
Content preview: Indeed, that's a good sanity check. I just checked and both
of our LTO7 pools are below their MAXSCRATCH setting, and we have scratch
tapes available in that library. I'm also not seeing any messages in the
activity or mount logs indicating a failure to mount scratch tapes. [...]

Content analysis details: (0.6 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.7 SPF_NEUTRAL SPF: sender does not match SPF record (neutral)
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
X-Barracuda-Start-Time: 1515508459
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://148.100.49.27:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at marist.edu
X-Barracuda-Scan-Msg-Size: 3278
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.46703
Rule breakdown below
pts rule name description
---- ---------------------- --------------------------------------------------

Indeed, that's a good sanity check. I just checked and both of our LTO7
pools are below their MAXSCRATCH setting, and we have scratch tapes
available in that library. I'm also not seeing any messages in the activity
or mount logs indicating a failure to mount scratch tapes.

Interestingly, the percent utilized for these two volumes was at 40% which,
with a device class format of ULTRIUM7C, would be right at the expected
point where a volume with uncompressible data should be full. I wonder if
there's some logic failure within TSM where it assumes >0% compression for
the last file.

In any event, TSM finally marked these two volumes as full after a few
hundred attempts. I would be sort of curious to know what the last file was
on each of them...

On Tue, Jan 09, 2018 at 02:21:50PM +0000, Schofield, Neil (Contractor - Storage & Middleware, Backup & Restore) wrote:
> Skylar
>
> It's probably worth ruling out the obvious stuff first.
>
> If the end-of-volume is reached then a new scratch tape will be required. If for some reason a new scratch tape cannot be mounted (eg there are no scratch left) to complete the writing of the spanned file/object then the store operation will fail and the previous end-of-data point on the tape will be used as the starting point the next time the tape is mounted. I've seen many occasions where a paucity of scratch tapes has resulted in the end-of-tape being reached repeatedly on the handful of remaining 'filling' volumes.
>
> In the cases where the end-of-volume has been reached, has a scratch volume been successfully mounted subsequently to complete the operation?
>
> Regards
> Neil
>
>
> Neil Schofield
> IBM Spectrum Protect SME
> Backup & Recovery | Storage & Middleware | Platform Technologies | Infrastructure Technology Services | Group CIO & IT Change
> LLOYDS BANKING GROUP
> ________________________________
>
>
>
> Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555.
>
> Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500.
>
> Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801.
>
> Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority.
>
> Halifax is a division of Bank of Scotland plc.
>
> HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813.
>
> This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Sorry, only registered users may post in this forum.

Click here to login