I'm in a tough spot here. I have a Windows Server 2008 R2 SP1 (ESX 4 VM) file server being protected by a FS iDA. This server has 4 volumes (1 for O/S and 3 volumes for user directories and other shares). All of these volumes are VMDKs from FC LUNs. I run weekly fulls to disk on all volumes and have no issues. I then go to Aux Copy those fulls to tape and get the message in the subject line about data being corrupted. Sure enough, when I run a verification job against those weekly fulls, the verification fails. I'm receiving this corruption error on two of the three non-O/S volumes (volume sizes are 550GB and 1.24TB) and the corruption issue started at the same time, three weeks ago. I do not receive the corruption error on my daily incrementals, which merely implies that the issue is with data that hasn't changed anytime recently.
I've opened a ticket with support and they state that the issue is with the client (their full analysis is below). The thing is, running chkdsk in read-only reports no errors on any of the volumes. There is no anti-virus installed on the client, in fact, nothing other than Windows, the iDA, and VMTools is installed. There are no scheduled tasks and no recent changes to the config on the server other than installing SP3, 3a, and 3b. There have been no recent SAN or ESX events or issues.
I've tried:
- Disabling VSS
- Disabling/Enabling Client-Side Dedupe
- Rebooted client
- Remove/re-present Virtual Disks to client
- Uninstall/reinstall FS iDA on client
I don't know what else to do at this point. How do I pinpoint where the alleged corruption is on my client? If chkdsk reports no errors in read-only mode it's not going to fix anything when I specify the /f switch. What would YOU do? I'm out of ideas.
Support Log File Analysis:
*****Job Starts
3124 1338 09/27 17:19:20 141686 Servant [---- IMMEDIATE BACKUP REQUEST ----], taskid [446] Clnt[servername] AppType[Windows File System][33] BkpSet[defaultBackupSet] SubClnt[default] BkpLevel[Full][1]
*****Job Completes Successful, no errors reported
3124 370 09/28 03:42:48 141686 Scheduler -> COMPLETED <- BACKUP; Backup job [141686] completed. Client [servername], Agent Type [Windows File System], Subclient [default], Backup Level [Full], Objects [1538219], Failed [5], Duration [10:23:26], Total Size [411.62 GB], Media or Mount Path Used [^10%s].
*****Data Verification Job
*****Job Manager Log indicates failure to copy chunk errors due to media corruption.
JobManager_1.log
*******************
3124 dd8 09/28 17:17:41 141938 Scheduler Set pending cause [Failed to Copy or verify Chunk [1033574] in media [CV_MAGNETIC], Storage Policy [DeDupe_Files], Copy [Primary To Disk], Host [mediaagent], Path [\mediaagentf$CVCV_MA2_FCV_MAGNETICV_282566], File Number [91], Backup Jobs [ 141686].
Data from the media is corrupted.aData read from media appears to be corrupted.a]::Client [media agent] Application [AuxCopy] Message Id [671088731] RCID [793600] ReservationId [0]. Level [0] flags [0] id [0] overwrite [0] append [1] CustId[671088672].
*****AuxCopy log (which handles the Data Verification) shows errors are due to an invalid tag header.
AuxCopy.log
**************
3676 f20 09/28 17:17:27 141938 Reader Source <7/6>: Reporting PROGRESS to the auxcopy manager. ChunkId [1013815] Bytes copied [536894703]
3676 974 09/28 17:17:41 ###### [ NULL] invalid tag header
3676 974 09/28 17:17:41 141938 793600-640015 [DM_BASE ] **ERROR** DataMoverSingleInstancer::ReadTagHeader: Failed to convert the tag header to Host byte order. Tag header got =
The tag header is used to verify the integrity of the chunk, similar to a CRC check, and is coming back as invalid, showing that the chunk itself is corrupt.
At this point, the errors seem to be isolated to this specific client, and the corruption is occurring during the primary backup itself. We could try to run a backup of the Media Agent itself using this storage policy, but from our discussion, other clients do not appear to be affected, so this would probably be an unnecessary step.
Eric Isakson | Network Engineer | Information Technology | Ogden Clinic
This email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else, unless expressly approved by the sender or an authorized addressee, is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action omitted or taken in reliance on it, is prohibited and may be unlawful. If you believe that you have received this email in error, please contact the sender, delete this email, and destroy all copies.
__._,_.___
Reply to sender ([email]eric.isakson < at > ogdenclinic.com?subject=Re%3A%20CV9%20SP3b%20-%20WinFS%20iDA%20-%20Data%20read%20from%20media%20appears%20to%20be%20corrupted[/email]) | Reply to group ([email]commvault < at > yahoogroups.com?subject=Re%3A%20CV9%20SP3b%20-%20WinFS%20iDA%20-%20Data%20read%20from%20media%20appears%20to%20be%20corrupted[/email]) | Reply via web post | Start a New Topic
Messages in this topic (1)
Recent Activity: Visit Your Group
Commvault Documentation here:
http://documentation.commvault.com/
MARKETPLACE
[url=http://global.ard.yahoo.com/SIG=15o5ur4ji/M=493064.14543979.14562481.13298430/D=groups/S=1707277520:MKP1/Y=YAHOO/EXP=1317800480/L=a74f951a-ef14-11e0-8c53-dff14016f045/B=GdZIBWKJiSA-/J=1317793280670553/K=dO6O8BuOaKn.cuj4xiAXHQ/A=6060255/R=0/SIG=1194m4keh/*http://us.toolbar.yahoo.com/?.cpdl=grpj]Stay on top of your group activity without leaving the page you're on - Get the Yahoo! Toolbar now.[/url] if(window.yzq_d==null)window.yzq_d=new Object(); window.yzq_d['GdZIBWKJiSA-']='&U=13ca2gq9k%2fN%3dGdZIBWKJiSA-%2fC%3d493064.14543979.14562481.13298430%2fD%3dMKP1%2fB%3d6060255%2fV%3d1'; [img]http://us.bc.yahoo.com/b?P=a74f951a-ef14-11e0-8c53-dff14016f045&T=1chffdbbc%2fX%3d1317793280%2fE%3d1707277520%2fR%3dgroups%2fK%3d5%2fV%3d2.1%2fW%3dH%2fY%3dYAHOO%2fF%3d1712628938%2fH%3dY29udGVudD0iR287V2lkZ2V0cztQb2RjYXN0cztNb2JpbGU7RmxpY2tyO0dyb3VwcztBdWN0aW9ucztTbWFsbF9CdXNpbmVzcztCcmllZmNhc2U7RmluYW5jZTsiIGRpc2FibGVzaHVmZmxpbmc9IjEiIHNlcnZlSWQ9ImE3NGY5NTFhLWVmMTQtMTFlMC04YzUzLWRmZjE0MDE2ZjA0NSIgc2l0ZUlkPSI0NDUyNTUxIiB0U3RtcD0iMTMxNzc5MzI4MDY1MTM1MSIg%2fQ%3d-1%2fS%3d1%2fJ%3dAD238962&U=13ca2gq9k%2fN%3dGdZIBWKJiSA-%2fC%3d493064.14543979.14562481.13298430%2fD%3dMKP1%2fB%3d6060255%2fV%3d1[/img]
Text-Only ([email]commvault-traditional < at > yahoogroups.com?subject=Change Delivery Format: Traditional[/email]), Daily Digest ([email]commvault-digest < at > yahoogroups.com?subject=Email Delivery: Digest[/email]) • Unsubscribe ([email]commvault-unsubscribe < at > yahoogroups.com?subject=Unsubscribe[/email]) • Terms of Use
.
[img]http://geo.yahoo.com/serv?s=97359714/grpId=7450851/grpspId=1707277520/msgId=5308/stime=1317793280/nc1=5898812/nc2=4025373/nc3=4507179[/img]
__,_._,___
