Welcome! » Log In » Create A New Profile

Removing Replication "with extreme prejudice"

Posted by Zoltan Forray 
Zoltan Forray
Removing Replication "with extreme prejudice"
July 02, 2019 09:59AM
ISP 7.1.7.400 on Linux - Back a while ago, we were working to eliminate an
ISP server. So, we removed replication from all of the nodes (or so we
thought) and removed the target replication server.

Well, somewhere along the way, it didn't completely/successfully remove
replication from one of the nodes. This wasn't discovered until after we
removed the target server.

Now this node is in a limbo state. Can't backup to it (we ended up
renaming it and recreating the node) - can't delete it - can't expire it -
can't remove replication from it (this is where we hope there is a hidden
"force" command) - can't re-add the target replication server.

*Expire Inventory produces this error:*

7/2/2019 12:04:05 PM ANR2017I Administrator ZFORRAY issued command: EXPIRE
INVENTORY NODE=RO-ARIAPROD-BAD
7/2/2019 12:04:05 PM ANR0984I Process 54 for EXPIRE INVENTORY started in
the BACKGROUND at 12:04:06 PM.
7/2/2019 12:04:05 PM ANR0811I Inventory client file expiration started as
process 54.
7/2/2019 12:04:05 PM ANR0405I Session 8351 ended for administrator ZFORRAY
(WinNT).
7/2/2019 12:04:05 PM ANR9999D_0415665470
admIsNodeConfiguredForRepl(admrepl.c:4966) Thread<202543>: Replication Mode
not found for node RO-ARIAPROD-BAD
7/2/2019 12:04:05 PM ANR9999D Thread<202543> issued message 9999 from:
7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x0000000106b995
OutDiagToCons
7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x0000000106ebcc outDiagfExt
7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x000000006284e4
admIsNodeConfiguredForRepl
7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x0000000062868b
admIsNodeReplica
7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x0000000062876c
admIsNodeIdReplica
7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x000000009e345f
ExpirationProcessThread
7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x00000001122f02 StartThread
7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x00003978807aa1 *UNKNOWN*
7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x000039784e8c4d *UNKNOWN*
7/2/2019 12:04:06 PM ANR2369I Database backup volume and recovery plan file
expiration starting under process 54.
7/2/2019 12:04:06 PM ANR0167I Inventory file expiration process 54
processed for 0 minutes.
7/2/2019 12:04:06 PM ANR0812I Inventory file expiration process 54
completed: processed 0 nodes, examined 0 objects, deleting 0 backup
objects, 0 archive objects, 0 DB backup volumes, and 0 recovery plan files.
0 objects were retried 0 errors were encountered and 0 objects were skipped.
7/2/2019 12:04:06 PM ANR0985I Process 54 for EXPIRE INVENTORY running in
the BACKGROUND completed with completion state FAILURE at 12:04:07 PM.

*Trying to delete its filespace generates a similar error:*

7/2/2019 12:05:38 PM ANR2017I Administrator ZFORRAY issued command: DELETE
FILESPACE RO-ARIAPROD-BAD 7 t=ba namet=fsid
7/2/2019 12:05:38 PM ANR9999D_0415665470
admIsNodeConfiguredForRepl(admrepl.c:4966) Thread<202566>: Replication Mode
not found for node RO-ARIAPROD-BAD
7/2/2019 12:05:38 PM ANR9999D Thread<202566> issued message 9999 from:
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000106b995
OutDiagToCons
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000106ebcc outDiagfExt
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000000006284e4
admIsNodeConfiguredForRepl
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000000005a0ccf
AdmDeleteFilespace
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000000555fe0
AdmCommandLocal
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000055772f admCommand
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000000d90ee2
SmAdminCommandThread
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000001122f02 StartThread
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00003978807aa1 *UNKNOWN*
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000039784e8c4d *UNKNOWN*
7/2/2019 12:05:38 PM ANR9999D_3725110895 AdmDeleteFilespace(admfs.c:1186)
Thread<202566>: Error 9990 obtaining replication information on node
RO-ARIAPROD-BAD
7/2/2019 12:05:38 PM ANR9999D Thread<202566> issued message 9999 from:
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000106b995
OutDiagToCons
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000106ebcc outDiagfExt
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000000005a0cfe
AdmDeleteFilespace
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000000555fe0
AdmCommandLocal
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000055772f admCommand
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000000d90ee2
SmAdminCommandThread
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000001122f02 StartThread
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00003978807aa1 *UNKNOWN*
7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000039784e8c4d *UNKNOWN*
7/2/2019 12:05:38 PM ANR2017I Administrator ZFORRAY issued command:
ROLLBACK
7/2/2019 12:05:38 PM ANR0405I Session 8359 ended for administrator ZFORRAY
(WinNT).

--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
VMware Administrator
Xymon Monitor Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Sorry, only registered users may post in this forum.

Click here to login