pborenich Absent Member.
Absent Member.
433 views

IDM 4.7.2 - RRSD removes/adds resources

Dear community,


we are currently experiencing massive problems with the RRSD in a customer environment.

Setup:
- SLES12 SP3
- IDM Engine 4.7.2
- UA/RRSD 4.7.2

The customer has quite a few dynamic groups in place which in turn are being used by UA to assign nrfRoles (> 20.000 dynamic groups/roles).

Up until now the setting for dynamic group reevaluation was set to 60 min, here the section from the start-up trace:

<dyn-group-interval display-name="Frequency of reevaluation of dynamic and nested groups (in minutes)">60</dyn-group-interval>


This setting(s) seemed to work fine. However, since monday RRSD seems to be constantly recalculating/resetting the group/roles assignments. As an example, here the resulting DirXML-EntitlementRef changes that pop up in the AD driver cache:


<modify cached-time="20190502103741.449Z" class-name="User" event-id="host1#20190502103741#3#6:5387acab-7e88-4620-95b2-abac8753887e" qualified-src-dn="O=data\OU=users\OU=aktiv\OU=Mitarbeiter\CN=user1" src-dn="\IDM\data\users\aktiv\Mitarbeiter\user1" src-entry-id="53964" timestamp="0#0">
<modify-attr attr-name="DirXML-EntitlementRef">
<remove-value>
<value timestamp="1556626547#28" type="structured">
<component name="nameSpace">1</component>
<component name="volume">\T=IDM\O=system\CN=driverset1\CN=Active Directory Driver\CN=ExchangeMailbox</component>
<component name="path.xml">
<ref>
<src>UA</src>
<id/>
<param>{"ID":"CN=MXDB_03,CN=Databases,CN=Exchange Administrative Group (123),CN=Administrative Groups,CN=ABC,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=dc1,DC=local"}</param>
</ref>
</component>
</value>
</remove-value>
<add-value>
<value timestamp="1556793461#60" type="structured">
<component name="nameSpace">0</component>
<component name="volume">\T=IDM\O=system\CN=driverset1\CN=Active Directory Driver\CN=ExchangeMailbox</component>
<component name="path.xml">
<ref>
<src>UA</src>
<id/>
<param>{"ID":"CN=MXDB_03,CN=Databases,CN=Exchange Administrative Group (123),CN=Administrative Groups,CN=ABC,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=dc1,DC=local"}</param>
</ref>
</component>
</value>
</add-value>
</modify-attr>
</modify>


In the above case, the entitlement is being removed (namespace 1>0). Some time later, RRSD will then re-add the entitlement (namespace 0>1).

Now, RRSD periodically triggers these entitlement "resets" for thousands of assignments, causing the de-provisioning of the permission assignments in the corresponding target system (of course, we already stopped the drivers so we don't mess up the connected systems).

We already tried to increase the value of dynamic group reevaluation, but RRSD is still not back to "normal" behaviour.

What we can often find in the trace is the following:

[05/02/19 15:34:57.707]:ROLERES ST:
DirXML Log Event -------------------
Driver: \IDM\system\driverset1\Role and Resource Service Driver
Channel: Subscriber
Status: Retry
Message: Code(-9006) The driver returned a "retry" status indicating that the operation should be retried later. Detail from driver: Thread ID:637948 Cant update the command in Driver Storage. Driver storage is full!!!


Did any of you experience this behaviour?

What would you suggest we should do to fix the problem?

Any other ideas?

Any help would be greatly appreciated! 🙂


Thank you and best regards,
Philipp
Labels (1)
0 Likes
6 Replies
Knowledge Partner
Knowledge Partner

Re: IDM 4.7.2 - RRSD removes/adds resources

pborenich <pborenich@no-mx.forums.microfocus.com> wrote:
>
> In the above case, the entitlement is being removed (namespace 1>0).

Some time later, RRSD will then re-add the entitlement (namespace 0>1).
>
> Now, RRSD periodically triggers these entitlement "resets" for thousands

of assignments, causing the de-provisioning of the permission
assignments in the corresponding target system (of course, we already
stopped the drivers so we don't mess up the connected systems).
>
> We already tried to increase the value of dynamic group reevaluation,

but RRSD is still not back to "normal" behaviour.
>


There are some issues here. Raise a service request. That is what we have
done. Suspect there are one or more bugs/regressions related to dynamic
groups and RRRSD here.

> Did any of you experience this behaviour?
>
> What would you suggest we should do to fix the problem?
>
> Any other ideas?
>


Workarounds
1. Fix broken users by remigrating them through RRSD shim.

2. Redesign to avoid overlapping child roles.

Dyn group x has Role A and child roles 1 and 2
Dyn group y has Role B and child roles 2 and 3

That kind of setup caused issues for us. The overlapping role got removed
and/or re-added when membership changes at dyn group level. It shouldn’t.

3. Switch to static groups until the Issues are fixed.

Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
pborenich Absent Member.
Absent Member.

Re: IDM 4.7.2 - RRSD removes/adds resources

alexmchugh;2499173 wrote:
pborenich <pborenich@no-mx.forums.microfocus.com> wrote:
>
> In the above case, the entitlement is being removed (namespace 1>0).

Some time later, RRSD will then re-add the entitlement (namespace 0>1).
>
> Now, RRSD periodically triggers these entitlement "resets" for thousands

of assignments, causing the de-provisioning of the permission
assignments in the corresponding target system (of course, we already
stopped the drivers so we don't mess up the connected systems).
>
> We already tried to increase the value of dynamic group reevaluation,

but RRSD is still not back to "normal" behaviour.
>


There are some issues here. Raise a service request. That is what we have
done. Suspect there are one or more bugs/regressions related to dynamic
groups and RRRSD here.

> Did any of you experience this behaviour?
>
> What would you suggest we should do to fix the problem?
>
> Any other ideas?
>


Workarounds
1. Fix broken users by remigrating them through RRSD shim.

2. Redesign to avoid overlapping child roles.

Dyn group x has Role A and child roles 1 and 2
Dyn group y has Role B and child roles 2 and 3

That kind of setup caused issues for us. The overlapping role got removed
and/or re-added when membership changes at dyn group level. It shouldn’t.

3. Switch to static groups until the Issues are fixed.


Hi Alex,

thanks for your input!

We observed a similiar behaviour concerning overlapping (child-)roles assigned through different dynamic groups. There seem to be combinations where RRSD does not properly handle the role/resource inheritance model when dynamic group memberships change (as you described). We managed fix the most pressing issues by switching to static role assignments for critical roles/resources and RRSD seems to be back to "normal" behaviour for now.

But we will keep an eye on the matter and plan to improve the implemented role/resource model to mitigate the risk of running into similiar issues again.

Best regards,
Philipp
0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.7.2 - RRSD removes/adds resources

RRSD and dynamic groups is just crazy dangerous, we have numerous customers that have issues.
0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.7.2 - RRSD removes/adds resources

alekz wrote:

>
> RRSD and dynamic groups is just crazy dangerous, we have numerous
> customers that have issues.


Let us keep on raising SRs until they fix it properly then!

--
If you find this post helpful, and are viewing this using the web, please show
your appreciation by clicking on the star below
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
pborenich Absent Member.
Absent Member.

Re: IDM 4.7.2 - RRSD removes/adds resources

alexmchugh;2500067 wrote:
alekz wrote:

>
> RRSD and dynamic groups is just crazy dangerous, we have numerous
> customers that have issues.


Let us keep on raising SRs until they fix it properly then!

--
If you find this post helpful, and are viewing this using the web, please show
your appreciation by clicking on the star below


Hi alekz, hi alexmchugh,

just to report back:
The SR on this issue has been resolved by WWS. The problem in this particular case was, that eDirectory is consuming new TCP connections for each dynamic group query. Due to the large number of dynamic groups/roles the periodic re-evaluation by RRSD occasionally fails, leading to the described symptoms.
We got the information, that this issue will be addressed/fixed in the upcoming eDirectory 9.1.4 release.

Best regards,
Philipp
0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.7.2 - RRSD removes/adds resources

pborenich <pborenich@no-mx.forums.microfocus.com> wrote:
>

alexmchugh;2500067 Wrote:
> alekz wrote:
>>
>>>
>>> RRSD and dynamic groups is just crazy dangerous, we have numerous
>>> customers that have issues.

>>
>> Let us keep on raising SRs until they fix it properly then!
>>
>> --
>> If you find this post helpful, and are viewing this using the web,

> please show
>> your appreciation by clicking on the star below

>
> Hi alekz, hi alexmchugh,
>
> just to report back:
> The SR on this issue has been resolved by WWS. The problem in this

particular case was, that eDirectory is consuming new TCP connections
for each dynamic group query. Due to the large number of dynamic
groups/roles the periodic re-evaluation by RRSD occasionally fails,
leading to the described symptoms.
> We got the information, that this issue will be addressed/fixed in the

upcoming eDirectory 9.1.4 release.
>


Thanks for that status update. It was me who originally found the ephemeral
port exhaustion issue about two years back.

It seems that this issue touches many code paths internally in eDir. They
have had several iterations of fixes so far.

The big challenge here has been achieving a fix that balances security,
performance and reliability. Using less ports has caused performance drops
in some of the fixes tested earlier.

I really do hope the upcoming fix you refer to is solid. (Tested a build
quite a few months back to confirm that it fixed the bugs we had, but hard
to performance test all aspects).

Also note that there is a RRSD shim fix just released that also addresses
some scenarios where roles can be added or removed incorrectly.
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.