Backup Central

We've been having jobs fail with a status 84 seemingly at random. These
jobs are being written to a DataDomain 890 configured as an OST target for
the backups. Both of the media servers in the environment (one AIX and one
Windows) are seeing these failures. Since NetBackup is reporting a plugin
error, we've engaged DataDomain technical support in addition to Symantec
technical support. These are the messages we see in the job details:

10/18/2011 17:52:31 - Critical bptm (pid=30605402) image write failed:
error 2060046: plugin error
10/18/2011 17:52:31 - Critical bptm (pid=30605402) sts_get_image_prop
failed: error 2060046: plugin error
10/18/2011 17:52:31 - Critical bptm (pid=30605402) image delete failed:
error 2060046: plugin error
10/18/2011 17:52:38 - Error bptm (pid=30605402) cannot write image to disk,
Invalid argument
10/18/2011 17:52:38 - Info bptm (pid=30605402) EXITING with status 84
<----------

Other vitals:
NetBackup version: 7.1.0.2
OST Plugin: 2.3.1.0
DDOS: 5.0.0.7-226726

Anyone else have a similar problem?
--
Jack Forester, Jr.
Sr. Data Protection Administrator
Global Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501

jack.forester < at > mylan.com

Phone: +1.304.554.6039
Cell: +1.412.805.5313


==============================================================================
CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
==============================================================================

_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

We have seem similar issues when we exceed the max streams on the DD
system. Synth jobs take up 3 streams, duplication jobs take up 2
streams, so you need to be careful with the max jobs on the storage
unit. If this is happening you will see messages logged in your daily
autosupports (No alerts, logged messages)

Something like this:

Current Alerts
--------------
No active alerts.

Recent Alerts and Log Messages

------------------------------
Sep 4 11:01:34 DDOM ddfs[26313]: WARNING: MSG-RPC-00001: Rejected 242505 connection attempts to port 2049 in the last 10039 minutes.
Sep 4 11:11:34 DDOM
ddfs[26313]: WARNING: MSG-RPC-00001: Rejected 274829 connection attempts to port 2049 in the last 10 minutes.
Sep 4 11:21:35 DDOM
ddfs[26313]: WARNING: MSG-RPC-00001: Rejected 369665 connection attempts to port 2049 in the last 10 minutes.
Sep 4 11:31:36 DDOM
ddfs[26313]: WARNING: MSG-RPC-00001: Rejected 417173 connection attempts to port 2049 in the last 10 minutes.
Sep 4 11:48:53 DDOM
ddfs[26313]: WARNING: MSG-RPC-00001: Rejected 256045 connection attempts to port 2049 in the last 17 minutes.




Do you have more than one OST device with SLP and opt-dupe as well?
There are some tweaks needed to make sure the correct media server is
used for duplication and synthetic fragment reads if this is the case,
which if you don't do, you will likely be clogging up your WAN link with
non-optimized backup traffic.

On 10/19/2011 2:06 PM, Jack.Forester < at > mylan.com wrote:
We've been having jobs fail with a status 84 seemingly at random. These
jobs are being written to a DataDomain 890 configured as an OST target for
the backups. Both of the media servers in the environment (one AIX and one
Windows) are seeing these failures. Since NetBackup is reporting a plugin
error, we've engaged DataDomain technical support in addition to Symantec
technical support. These are the messages we see in the job details:

10/18/2011 17:52:31 - Critical bptm (pid=30605402) image write failed:
error 2060046: plugin error
10/18/2011 17:52:31 - Critical bptm (pid=30605402) sts_get_image_prop
failed: error 2060046: plugin error
10/18/2011 17:52:31 - Critical bptm (pid=30605402) image delete failed:
error 2060046: plugin error
10/18/2011 17:52:38 - Error bptm (pid=30605402) cannot write image to disk,
Invalid argument
10/18/2011 17:52:38 - Info bptm (pid=30605402) EXITING with status 84
<----------

Other vitals:
NetBackup version: 7.1.0.2
OST Plugin: 2.3.1.0
DDOS: 5.0.0.7-226726

Anyone else have a similar problem?
--
Jack Forester, Jr.
Sr. Data Protection Administrator
Global Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501

jack.forester < at > mylan.com

Phone: +1.304.554.6039
Cell: +1.412.805.5313


==============================================================================
CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
==============================================================================

_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

We had the same problem, media write errors. Seems to happen with long running jobs most of the time. We ugpraded the Data domain code and OST code which increased the timeouts to 3hrs from data domain. We also noticed are long running jobs in which most of the 84 errors were coming from were Window 2008 or Windows 2008 R2. there is a known hotfix from Symantec on this issue.
Here is the link for this:
http://www.symantec.com/business/support/index?page=content&id=TECH150081

We upgraded to the following:
DD670
5.0.1.7-250021
OST plugin 2.3.2.0
NBU - 7.0.1

Once we upgraded to the current versions above and applied the hotfixes we now get a media write error 84, maybe once a week and at times we will go for weeks with not issues. I am to believe the OST plugin and/or NBU 7.0.1 still has a bug but neither vendor has a compelte fix.

DataDomain has also recommended that we upgrade to DDOS 5.0.2.3 for the
exact same reason you cite. It's good to get confirmation from someone
that this could more or less resolve the problem. Thanks for everyone's
feedback...it is much appreciated!
--
Jack Forester, Jr.
Sr. Data Protection Administrator
Global Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501

jack.forester < at > mylan.com

Phone: +1.304.554.6039
Cell: +1.412.805.5313



From: "rsavage" <nbu-forum < at > backupcentral.com>
To: VERITAS-BU < at > MAILMAN.ENG.AUBURN.EDU
Date: 10/20/2011 04:00 PM
Subject: [Veritas-bu] Status 84 - plugin error
Sent by: veritas-bu-bounces < at > mailman.eng.auburn.edu



We had the same problem, media write errors. Seems to happen with long
running jobs most of the time. We ugpraded the Data domain code and OST
code which increased the timeouts to 3hrs from data domain. We also noticed
are long running jobs in which most of the 84 errors were coming from were
Window 2008 or Windows 2008 R2. there is a known hotfix from Symantec on
this issue.
Here is the link for this:
http://www.symantec.com/business/support/index?page=content&id=TECH150081

We upgraded to the following:
DD670
5.0.1.7-250021
OST plugin 2.3.2.0
NBU - 7.0.1

Once we upgraded to the current versions above and applied the hotfixes we
now get a media write error 84, maybe once a week and at times we will go
for weeks with not issues. I am to believe the OST plugin and/or NBU 7.0.1
still has a bug but neither vendor has a compelte fix.

+----------------------------------------------------------------------
|This was sent by rich.savage < at > bcbsne.com via Backup Central.
|Forward SPAM to abuse < at > backupcentral.com.
+----------------------------------------------------------------------


_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu




==============================================================================
CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
==============================================================================

_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Hi Jack,
i'll second rsavage. we had to upgrade to 5.0.1.7 and use the symantec patch for 2008 servers as well which definitely helped. We worked for a few weeks before we got a hold of the right person at DD to figure this out. DD support has been great but they need to do a better job keeping their users informed of fixes or changes like this timeout config. Didn't receive any notice of updates. 5.0.2.3? crazy!

Yea, Data Domain want us to upgrade to 5.0.2.3 also but they cannot pinpoint any reason for the code 84. Right now we are staying at 5.0.1.7 as the media write errors are very very few now. I even asked datadomain to give the solution id that specifically addresses the media write error in the new code and they really couldn't. They gave me a few solution id's but it did not fit the criteria of media write errors, just addressed increasing the timeouts.