Micro Focus Frequent Contributor
Micro Focus Frequent Contributor
1215 views

GMS removed every user

Hello

Has anybody encountered such an error and if yes how can it be avoided in the future?

10 gms servers, 6 on sles11 4 on sles12, same gms version 14.2.2 Build: 195
4 eDirectory server combined with NetIq LDAP Proxy as source for GMS.
Users are added to GMS via eDirectory groups.

On a nice winter day all 4 eDirectory servers encountered a fatal error and went down roughly about the same time. Every gms server noticed the outage and removed every user from every GMS server. After the eDirectory servers came back GMS noticed the users in the groups and added them back. This caused about a 12 hour downtime in the service.

So how can I prevent GMS to remove users when it cannot reach eDirectory?

Regards,
Gellert
Labels (1)
Tags (2)
0 Likes
9 Replies
Micro Focus Expert
Micro Focus Expert

Re: GMS removed every user

Gehorvath,

> On a nice winter day all 4 eDirectory servers encountered a fatal error
> and went down roughly about the same time. Every gms server noticed the
> outage and removed every user from every GMS server. After the
> eDirectory servers came back GMS noticed the users in the groups and
> added them back. This caused about a 12 hour downtime in the service.
>
> So how can I prevent GMS to remove users when it cannot reach
> eDirectory?


I've not seen this before but let me do some research on it for you.

Pam

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: GMS removed every user

Gehorvath,

> I've not seen this before but let me do some research on it for you.


I am not sure if it matters but....

Can I get the eDir version on the SLES11SP4 boxes. Can I get the patch
level of the SLES12 boxes as well as the eDir version there. How many
other servers are in the tree besides these 10? Where is the master
replica? Are all 10 of the GMS boxes holding a Read/Write replica or
Read Only?

Thanks,

Pam

0 Likes
Micro Focus Frequent Contributor
Micro Focus Frequent Contributor

Re: GMS removed every user

Hello Pam,

eDirectory:
There are about 250 servers in the tree, Almost every of them is OES 2015 with the almost the latest patches.
There 4 replica servers in the HQ in the load-balancer and they are up-to-date atm. (8.8SP8 Binary Version: 20812.20) Server no1 has all the master replicas, server no2,3,4 have rw replicas of every partition. The other servers are branch office servers, they have rw replicas of the corresponding subtree only and they don't play a role in the load balancer.

GMS:
gms1-6 are SLES 11 SP4 servers and are not fully patched. I think that they were last patched when the Meltdown and Spectre vulnerabilities were fixed. I am in the process of upgrading these servers to SLES12SP3.
gms7-10 are SLES12 SP3 servers and are not fully patched, just like the SLES11SP4 servers.
GMS version is Version: 14.2.2 Build: 19 on every server.
GMS servers are dedicated to GMS only.

Last week the XDAS audit has been turned on on every server in the tree. The side effect were random ndsd crashes on the 4 HQ servers. On Monday all 4 servers went down at the same time. The GMS incident happened about this time.
We had edir outages before but GMS just reported "LDAP server down" and nothing else happened. But this time all 4 servers went down.

Regards,
Gellert
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: GMS removed every user

Gehorvath,

> Can I get the eDir version on the SLES11SP4 boxes. Can I get the patch
> level of the SLES12 boxes as well as the eDir version there. How many
> other servers are in the tree besides these 10? Where is the master
> replica? Are all 10 of the GMS boxes holding a Read/Write replica or
> Read Only?


Also is your GMS logging set to Debug and do you have debug logs from the
time that this happened? If so, would you give me the configengine.log
and the engine.log. I did check with our engineering team and this was
indeed not something that should have happened.

thanks,

Pam

0 Likes
Micro Focus Frequent Contributor
Micro Focus Frequent Contributor

Re: GMS removed every user

Hello,

I've sent you the logs.
As far as I see GMS did the ldap poll as usual and the ldap server returned an empty member list (Members: []). Then GMS removed the users as designed. So maybe its not a GMS but an eDirectory or LDAP Proxy thing.

Regards,
Gellért
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: GMS removed every user

gehorvath;2475883 wrote:
Hello,

I've sent you the logs.
As far as I see GMS did the ldap poll as usual and the ldap server returned an empty member list (Members: []). Then GMS removed the users as designed. So maybe its not a GMS but an eDirectory or LDAP Proxy thing.

Regards,
Gellért


Thanks. I did get the logs and have sent all this info over to the engineer that I am working with.

Pam
0 Likes
Knowledge Partner
Knowledge Partner

Re: GMS removed every user

On 21.02.2018 15:44, gehorvath wrote:
>
> Hello,
>
> I've sent you the logs.
> As far as I see GMS did the ldap poll as usual and the ldap server
> returned an empty member list (Members: []). Then GMS removed the users
> as designed. So maybe its not a GMS but an eDirectory or LDAP Proxy
> thing.


I was about to answer your original post wanting to ask about an ldap
proxy or load balancer or something in the mix. The issue clearly is
there. "Succesfully" returning an empty group when ldap is completely
down is a truly desastrous bug.

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Micro Focus Frequent Contributor
Micro Focus Frequent Contributor

Re: GMS removed every user

Hello Massimo,

More than a year ago there was a similar issue with Vibe. Vibe does also gets its users from eDirectory via an LDAP poll. If it does not finds a user, then Vibe disables it in its own database. On a Sunday morning Vibe did its usual LDAP poll and it returned no users at all. So Vibe disabled every user. On the next LDAP poll the users were available and were enabled by Vibe. As far as I could tell, there was no error on eDirectory side. No outage, no freeze, nothing. And there was no LDAP load balancer at that time. So I think that LDAP Proxy innocent in this case. Nevertheless I can send you the LDAP Proxy config. Is it safe if I send it to you via the forums "Send Private Message" feature?

Regards,
Gellert
0 Likes
Knowledge Partner
Knowledge Partner

Re: GMS removed every user

On 22.02.2018 12:34, gehorvath wrote:
>
> Hello Massimo,
>
> More than a year ago there was a similar issue with Vibe. Vibe does also
> gets its users from eDirectory via an LDAP poll. If it does not finds a
> user, then Vibe disables it in its own database. On a Sunday morning
> Vibe did its usual LDAP poll and it returned no users at all. So Vibe
> disabled every user. On the next LDAP poll the users were available and
> were enabled by Vibe. As far as I could tell, there was no error on
> eDirectory side. No outage, no freeze, nothing. And there was no LDAP
> load balancer at that time. So I think that LDAP Proxy innocent in this
> case. Nevertheless I can send you the LDAP Proxy config. Is it safe if I
> send it to you via the forums "Send Private Message" feature?
>
> Regards,
> Gellert
>
>

I'm not sure if that would help. I have had countless situations at
customers where the one and only LDAP server for GMS was down, and
nothing bad happened. So I can pretty certainly exclude that such a bug
in GMS exists. To know exactly what happened one would need to take a
LAN Trace or some other low level log to see what the LDAP Proxy really
returns when the LDAP Server are down. Of course, there's already a
difference in that with the proxy in the mix, GMS still succesfully
connects to the LDAP port at least, whereas when it talks to eDirectory
directly and all servers are down, it doesn't even get a TCP level
connection.

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.