GST hard crash after cumulative hotfix 18.2.0.2

Hi,

Anyone come across something like the following after installing
cumulative hotfix 18.2.0.2?

Everything is fine, except GST...

When I roll back GST and database to the previous state (18.2.0.0) it
starts up ok.

Update essentially was: 1) rpm -e 2) rpm -i 3) nmc_config

Can we run GST 18.2.0.0 when everything else is at 18.2.0.2?



129343 1559857988 2 0 0 4047329088 23604 0 nwr gstd NSR warning 56 Using
authentication service on '%s' host and '%d' port. 2 0 3 nwr 1 4 9090
0 1559857988 1 5 0 4047329088 23604 0 nwr gstd NSR notice 5 %s %s 2 0 24
06/06/19 23:53:08.235938 0 89 ERROR generated: file
"/disks/nasbld/nas83/nw/18.2/gst/modules/lem/licenses.c" line #1649
0 1559857988 1 5 0 4047329088 23604 0 nwr gstd NSR notice 5 %s %s 2 0 24
06/06/19 23:53:08.236027 0 89 ERROR generated: file
"/disks/nasbld/nas83/nw/18.2/gst/modules/lem/licenses.c" line #1649
*** Error in `/opt/lgtonmc/bin/gstd': double free or corruption (out):
0x00007f55c4259b60 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x740ef)[0x7f55eedcb0ef]
/lib64/libc.so.6(+0x79646)[0x7f55eedd0646]
/lib64/libc.so.6(+0x7a393)[0x7f55eedd1393]
/opt/lgtonmc/mod/gstnsm.so(+0x114f34)[0x7f55e841df34]
/opt/lgtonmc/mod/gstnsm.so(addClientNodeJobsData+0x1225)[0x7f55e8422d65]
/opt/lgtonmc/mod/gstnsm.so(addActionJobsData+0x563)[0x7f55e84239d3]
/opt/lgtonmc/mod/gstnsm.so(getHarmonyPolicyData+0x7a9)[0x7f55e8426099]
/opt/lgtonmc/mod/gstnsm.so(+0xe281f)[0x7f55e83eb81f]
/opt/lgtonmc/mod/gstnsm.so(+0xe398a)[0x7f55e83ec98a]
/opt/lgtonmc/mod/gstnsm.so(+0xe53ea)[0x7f55e83ee3ea]
/opt/lgtonmc/bin/gstd[0x4b5479]
/lib64/libpthread.so.0(+0x8724)[0x7f55ef9a3724]
/lib64/libc.so.6(clone+0x6d)[0x7f55eee43e8d]
======= Memory map: ========
00400000-00513000 r-xp 00000000 08:01 19303                             
/opt/lgtonmc/bin/gstd
00713000-0071a000 rw-p 00113000 08:01 19303                             
/opt/lgtonmc/bin/gstd
0071a000-00c83000 rw-p 00000000 00:00 0
0297a000-02a6f000 rw-p 00000000 00:00 0                                 
[heap]
02a6f000-02c1e000 rw-p 00000000 00:00 0                                 
[heap]
....


--
This list is hosted as a public service at Temple University by Stan Horwitz
If you wish to sign off this list or adjust your subscription settings, please do so via http://listserv.temple.edu/archives/emc-dataprotection-l.html
If you have any questions regarding management of this list, please send email to owner-emc-dataprotection-l@listserv.temple.edu
This message was imported via the External PhorumMail Module

FYI We have isolated this to a fully patched SLES12-SP3. Works on
SLES12-SP2.

Note: GST up to 18.2.0.1 works fine.


On 6/7/2019 10:59 AM, marki wrote:
>
> 129343 1559857988 2 0 0 4047329088 23604 0 nwr gstd NSR warning 56
> Using authentication service on '%s' host and '%d' port. 2 0 3 nwr 1 4
> 9090
> 0 1559857988 1 5 0 4047329088 23604 0 nwr gstd NSR notice 5 %s %s 2 0
> 24 06/06/19 23:53:08.235938 0 89 ERROR generated: file
> "/disks/nasbld/nas83/nw/18.2/gst/modules/lem/licenses.c" line #1649
> 0 1559857988 1 5 0 4047329088 23604 0 nwr gstd NSR notice 5 %s %s 2 0
> 24 06/06/19 23:53:08.236027 0 89 ERROR generated: file
> "/disks/nasbld/nas83/nw/18.2/gst/modules/lem/licenses.c" line #1649
> *** Error in `/opt/lgtonmc/bin/gstd': double free or corruption (out):
> 0x00007f55c4259b60 ***
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x740ef)[0x7f55eedcb0ef]
> /lib64/libc.so.6(+0x79646)[0x7f55eedd0646]
> /lib64/libc.so.6(+0x7a393)[0x7f55eedd1393]
> /opt/lgtonmc/mod/gstnsm.so(+0x114f34)[0x7f55e841df34]
> /opt/lgtonmc/mod/gstnsm.so(addClientNodeJobsData+0x1225)[0x7f55e8422d65]
> /opt/lgtonmc/mod/gstnsm.so(addActionJobsData+0x563)[0x7f55e84239d3]
> /opt/lgtonmc/mod/gstnsm.so(getHarmonyPolicyData+0x7a9)[0x7f55e8426099]
> /opt/lgtonmc/mod/gstnsm.so(+0xe281f)[0x7f55e83eb81f]
> /opt/lgtonmc/mod/gstnsm.so(+0xe398a)[0x7f55e83ec98a]
> /opt/lgtonmc/mod/gstnsm.so(+0xe53ea)[0x7f55e83ee3ea]
> /opt/lgtonmc/bin/gstd[0x4b5479]
> /lib64/libpthread.so.0(+0x8724)[0x7f55ef9a3724]
> /lib64/libc.so.6(clone+0x6d)[0x7f55eee43e8d]


--
This list is hosted as a public service at Temple University by Stan Horwitz
If you wish to sign off this list or adjust your subscription settings, please do so via http://listserv.temple.edu/archives/emc-dataprotection-l.html
If you have any questions regarding management of this list, please send email to owner-emc-dataprotection-l@listserv.temple.edu
This message was imported via the External PhorumMail Module

In regard to: [EMC-DataProtection-L] [SOLVED] Re: GST hard crash after...:

> FYI We have isolated this to a fully patched SLES12-SP3. Works on SLES12-SP2.
>
> Note: GST up to 18.2.0.1 works fine.

You may want to check the release notes for SLES, to see if they made
any changes to the memory allocator between SP2 and SP3.

Ultimately, the problem is in gstnsm.so, but it's possible that there
were also changes to e.g. malloc() that are now exposing the problem.
It should be reported to EMC, but you may also want to look into whether
you can relax any additional checks until EMC addresses the issue.

If you do some searching for the environment variable MALLOC_CHECK_ ,
you'll see an example of what I mean. I don't know if it applies to
SLES at all, but it did apply to Red Hat at one point.

Tim

> On 6/7/2019 10:59 AM, marki wrote:
>>
>> 129343 1559857988 2 0 0 4047329088 23604 0 nwr gstd NSR warning 56 Using
>> authentication service on '%s' host and '%d' port. 2 0 3 nwr 1 4 9090
>> 0 1559857988 1 5 0 4047329088 23604 0 nwr gstd NSR notice 5 %s %s 2 0 24
>> 06/06/19 23:53:08.235938 0 89 ERROR generated: file
>> "/disks/nasbld/nas83/nw/18.2/gst/modules/lem/licenses.c" line #1649
>> 0 1559857988 1 5 0 4047329088 23604 0 nwr gstd NSR notice 5 %s %s 2 0 24
>> 06/06/19 23:53:08.236027 0 89 ERROR generated: file
>> "/disks/nasbld/nas83/nw/18.2/gst/modules/lem/licenses.c" line #1649
>> *** Error in `/opt/lgtonmc/bin/gstd': double free or corruption (out):
>> 0x00007f55c4259b60 ***
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x740ef)[0x7f55eedcb0ef]
>> /lib64/libc.so.6(+0x79646)[0x7f55eedd0646]
>> /lib64/libc.so.6(+0x7a393)[0x7f55eedd1393]
>> /opt/lgtonmc/mod/gstnsm.so(+0x114f34)[0x7f55e841df34]
>> /opt/lgtonmc/mod/gstnsm.so(addClientNodeJobsData+0x1225)[0x7f55e8422d65]
>> /opt/lgtonmc/mod/gstnsm.so(addActionJobsData+0x563)[0x7f55e84239d3]
>> /opt/lgtonmc/mod/gstnsm.so(getHarmonyPolicyData+0x7a9)[0x7f55e8426099]
>> /opt/lgtonmc/mod/gstnsm.so(+0xe281f)[0x7f55e83eb81f]
>> /opt/lgtonmc/mod/gstnsm.so(+0xe398a)[0x7f55e83ec98a]
>> /opt/lgtonmc/mod/gstnsm.so(+0xe53ea)[0x7f55e83ee3ea]
>> /opt/lgtonmc/bin/gstd[0x4b5479]
>> /lib64/libpthread.so.0(+0x8724)[0x7f55ef9a3724]
>> /lib64/libc.so.6(clone+0x6d)[0x7f55eee43e8d]
>
>
> --
> This list is hosted as a public service at Temple University by Stan Horwitz
> If you wish to sign off this list or adjust your subscription settings, please
> do so via http://listserv.temple.edu/archives/emc-dataprotection-l.html
> If you have any questions regarding management of this list, please send email
> to owner-emc-dataprotection-l@listserv.temple.edu
>

--
Tim Mooney Tim.Mooney@ndsu.edu
Enterprise Computing & Infrastructure 701-231-1076 (Voice)
Room 242-J6, Quentin Burdick Building 701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164


--
This list is hosted as a public service at Temple University by Stan Horwitz
If you wish to sign off this list or adjust your subscription settings, please do so via http://listserv.temple.edu/archives/emc-dataprotection-l.html
If you have any questions regarding management of this list, please send email to owner-emc-dataprotection-l@listserv.temple.edu
This message was imported via the External PhorumMail Module

Just another note: We tried upgrading again to get more info for
support, and it didn't anymore.

I believe there must have been some bad stuff in jobs DB which has since
expired (jobs db retention).

In any case it wasn't stuff that 18.2.0.1 would choke on (or crash for
that matter).

So in the end it might not have been an issue related to the OS version
after all.

On 6/7/2019 4:27 PM, marki wrote:
> FYI We have isolated this to a fully patched SLES12-SP3. Works on
> SLES12-SP2.
>
> Note: GST up to 18.2.0.1 works fine.
>
>
> On 6/7/2019 10:59 AM, marki wrote:
>>
>> 129343 1559857988 2 0 0 4047329088 23604 0 nwr gstd NSR warning 56
>> Using authentication service on '%s' host and '%d' port. 2 0 3 nwr 1
>> 4 9090
>> 0 1559857988 1 5 0 4047329088 23604 0 nwr gstd NSR notice 5 %s %s 2 0
>> 24 06/06/19 23:53:08.235938 0 89 ERROR generated: file
>> "/disks/nasbld/nas83/nw/18.2/gst/modules/lem/licenses.c" line #1649
>> 0 1559857988 1 5 0 4047329088 23604 0 nwr gstd NSR notice 5 %s %s 2 0
>> 24 06/06/19 23:53:08.236027 0 89 ERROR generated: file
>> "/disks/nasbld/nas83/nw/18.2/gst/modules/lem/licenses.c" line #1649
>> *** Error in `/opt/lgtonmc/bin/gstd': double free or corruption
>> (out): 0x00007f55c4259b60 ***
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x740ef)[0x7f55eedcb0ef]
>> /lib64/libc.so.6(+0x79646)[0x7f55eedd0646]
>> /lib64/libc.so.6(+0x7a393)[0x7f55eedd1393]
>> /opt/lgtonmc/mod/gstnsm.so(+0x114f34)[0x7f55e841df34]
>> /opt/lgtonmc/mod/gstnsm.so(addClientNodeJobsData+0x1225)[0x7f55e8422d65]
>> /opt/lgtonmc/mod/gstnsm.so(addActionJobsData+0x563)[0x7f55e84239d3]
>> /opt/lgtonmc/mod/gstnsm.so(getHarmonyPolicyData+0x7a9)[0x7f55e8426099]
>> /opt/lgtonmc/mod/gstnsm.so(+0xe281f)[0x7f55e83eb81f]
>> /opt/lgtonmc/mod/gstnsm.so(+0xe398a)[0x7f55e83ec98a]
>> /opt/lgtonmc/mod/gstnsm.so(+0xe53ea)[0x7f55e83ee3ea]
>> /opt/lgtonmc/bin/gstd[0x4b5479]
>> /lib64/libpthread.so.0(+0x8724)[0x7f55ef9a3724]
>> /lib64/libc.so.6(clone+0x6d)[0x7f55eee43e8d]


--
This list is hosted as a public service at Temple University by Stan Horwitz
If you wish to sign off this list or adjust your subscription settings, please do so via http://listserv.temple.edu/archives/emc-dataprotection-l.html
If you have any questions regarding management of this list, please send email to owner-emc-dataprotection-l@listserv.temple.edu
This message was imported via the External PhorumMail Module

Hi again,

FYI
We were told by Dell that this was a regression and that it was again
fixed in 19.1. So it looks like an awful mess. Also they would not be
fixing any more things in v18 and we would be urged to update to 19.1!?
Support for v18 only ends in July 2021. So I wonder where they are going
with this.


--
This list is hosted as a public service at Temple University by Stan Horwitz
If you wish to sign off this list or adjust your subscription settings, please do so via http://listserv.temple.edu/archives/emc-dataprotection-l.html
If you have any questions regarding management of this list, please send email to owner-emc-dataprotection-l@listserv.temple.edu
This message was imported via the External PhorumMail Module