simeonof Trusted Contributor.
Trusted Contributor.
612 views

Managing huge group memberships calculation

Hi,

I have a nice IDM system with a few Loopback drivers. One of the drivers is monitoring for changes in a specific attribute in the User class. Basically, when the attribute gets a certain value, the user becomes member of a group. Respectively, if the attribute gets another specific value - the user is removed from the group. Adding/removing a user to the group means "get rid of all group members, then check which users have this attribute with the specific value, and make all of them members of the group". There is another loopback driver, who is responsible for maintaining the SecurityEquals...... reciprocal attributes between the user and the group. This driver kicks in when the group membership changes. So far so good, or at least until the group got more than 20k members. Currently, when the group membership needs to be recalculated, the server load gets very high and things happen rather slowly.

Here's how we recalculate the group membership (in a CommandTransformation rule):
<rule disabled="true" notrace="true">
<description>Update myGroup membership when the User attribute myAttribute changes</description>
<conditions>
<and>
<if-class-name mode="nocase" op="equal">User</if-class-name>
<if-op-attr name="myAttribute" op="changing"/>
</and>
</conditions>
<actions>
<do-clear-dest-attr-value class-name="Group" name="Member">
<arg-dn>
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-dn>
</do-clear-dest-attr-value>
<do-for-each>
<arg-node-set>
<token-query class-name="User" datastore="src">
<arg-match-attr name="myAttribute">
<arg-value type="string">
<token-text xml:space="preserve">1</token-text>
</arg-value>
</arg-match-attr>
</token-query>
</arg-node-set>
<arg-actions>
<do-add-dest-attr-value class-name="Group" name="Member">
<arg-dn>
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-dn>
<arg-value type="dn">
<token-xpath expression="$current-node/@src-dn"/>
</arg-value>
</do-add-dest-attr-value>
</arg-actions>
</do-for-each>
</actions>
</rule>


I'm looking to optimize the above process somehow, as when this rule fires up and clears the group members, the other loopback driver fires up and starts removing the SecurityEquals, then it starts creating them again when the group members reappear. Any ideas would be much appreciated!

Cheers
Labels (1)
0 Likes
5 Replies
Knowledge Partner
Knowledge Partner

Re: Managing huge group memberships calculation

On 01/09/2019 10:14 AM, Simeonof wrote:
>
> I have a nice IDM system with a few Loopback drivers. One of the drivers
> is monitoring for changes in a specific attribute in the User class.
> Basically, when the attribute gets a certain value, the user becomes
> member of a group. Respectively, if the attribute gets another specific


So far, so good, until this next sentence.

> value - the user is removed from the group. Adding/removing a user to
> the group means "get rid of all group members, then check which users


Why would you remove all members? You know the user with the value
changing, if I understand this properly, so why would you remove anybody
other than the one user, or add anybody but the one user? This is going
to kill performance by turning one operation, which then turns into a few
operations (for different attributes), all the way up to an n*4 operation,
which of course meant will take n-times longer. Instead of four (4)
operations, you have 20000*4 == 80000 operations.

The whole point of having an event-driven system is you do not need to do
horribly inefficient things like this. If a change happens, you'll get
it, and you act on just that. If you get multiple changes, you'll act on
those. Only if you cannot guarantee events coming in, which is the sign
of a lesser system by far, should you need to reconcile differences
manually like this.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
simeonof Trusted Contributor.
Trusted Contributor.

Re: Managing huge group memberships calculation

ab;2493340 wrote:
On 01/09/2019 10:14 AM, Simeonof wrote:
Why would you remove all members?
The whole point of having an event-driven system is you do not need to do
horribly inefficient things like this.


Agree. As far as I remember the developer had faced some issues in the past (old IDM versions maybe, when the driver didn't have the "reciprocal attributes" mappings) with inconsistent reciprocal attributes, so he most probably wanted to make sure data is 100% consistent. The original rule guarantees that consistency, however it is quite inefficient when it has to recalculate many thousands of members. I've figured out a simple replacement and just wanted to exchange some thoughts with fellow admins on what would be the most efficient way to do it (there's more than one for sure). Here's mine, I'd appreciate any thoughts/suggestions on it:


<rule>
<description>ZZZ Manage myGroup membership - additions</description>
<conditions>
<and>
<if-class-name mode="nocase" op="equal">User</if-class-name>
<if-op-attr mode="nocase" name="myAttribute" op="changing-to">1</if-op-attr>
</and>
</conditions>
<actions>
<do-add-dest-attr-value class-name="User" name="Group Membership">
<arg-value type="dn">
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-value>
</do-add-dest-attr-value>
<do-add-dest-attr-value class-name="User" name="Security Equals">
<arg-value type="dn">
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-value>
</do-add-dest-attr-value>
</actions>
</rule>
<rule>
<description>YYY Manage myGroup membership - removals</description>
<conditions>
<and>
<if-class-name mode="nocase" op="equal">User</if-class-name>
<if-op-attr mode="regex" name="myAttribute" op="changing-to">[^1]+</if-op-attr>
</and>
</conditions>
<actions>
<do-remove-dest-attr-value class-name="User" name="Group Membership">
<arg-value type="dn">
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-value>
</do-remove-dest-attr-value>
<do-remove-dest-attr-value class-name="User" name="Security Equals">
<arg-value type="dn">
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-value>
</do-remove-dest-attr-value>
</actions>
</rule>

0 Likes
Knowledge Partner
Knowledge Partner

Re: Managing huge group memberships calculation

On 01/10/2019 02:04 PM, Simeonof wrote:
>
> Agree. As far as I remember the developer had faced some issues in the
> past (old IDM versions maybe, when the driver didn't have the


Just to be pedantic, for the benefit of you or others reading this who may
be new to IDM, it is not the driver (meaning the shim) which does anything
like this, but the engine. The driver config object controls reciprocal
attribute mappings, but by default, since at least DirXML 1.1, this has
been default functionality. With that written, there have been many
customers who have accidentally broken those mappings, and threads are in
the forums describing those issues. Basically if you do nothing, you get
the functionality implicitly. If you create a new default policy and make
it empty, which looks like "nothing was done", it is very different
because now you are being explicit, thus you lose all functionality. If
you then configure the policy properly (which I think it does by default),
you get what you want back. It's great functionality, but if you are not
careful when you are playing in there you can get a situation where it
doesn't work. With that written...

> "reciprocal attributes" mappings) with inconsistent reciprocal
> attributes, so he most probably wanted to make sure data is 100%
> consistent. The original rule guarantees that consistency, however it is


This doesn't make sense. Reciprocal attribute mappings have nothing to do
with other objects, except the group or user directly connected to the
current event. It would never, at all, go and fix (or break) objects
outside of the two, so making something verify the whole group again must
have either been done for some other reason, or because something else was
seriously broken. I cannot imagine a good scenario for that, unless the
previous version of this logic used a set (vs. an add) to add the new
value, in which case all others would have been removed, but if done that
would be a newbie mistake.

those who have been around too long may remember the
still-present-and-defaulted-wrong option on the microsoft active directory
(MAD) driver config.option to "Enable Dirsync Incremental Values". The
default is 'false',and was required for MAD domains at windows 2000 domain
functional level. Since MAD 2003 functional levels, this value should be
set to 'true', since with that version microsoft realizing that removing
all values just to add them all back with one more (for an add) or
removing all values just to add them all back except one (for a remove)
was a terrible idea. You're doing the same thing here, and yes, the
performance stinks. It's the very first thing I change with every MAD
driver config I touch, whether I created it originally or not.

>
> Code:
> --------------------
>
> <rule>
> <description>ZZZ Manage myGroup membership - additions</description>
> <conditions>
> <and>
> <if-class-name mode="nocase" op="equal">User</if-class-name>
> <if-op-attr mode="nocase" name="myAttribute" op="changing-to">1</if-op-attr>
> </and>
> </conditions>
> <actions>
> <do-add-dest-attr-value class-name="User" name="Group Membership">
> <arg-value type="dn">
> <token-text xml:space="preserve">blabla\groups\myGroup</token-text>
> </arg-value>
> </do-add-dest-attr-value>
> <do-add-dest-attr-value class-name="User" name="Security Equals">
> <arg-value type="dn">
> <token-text xml:space="preserve">blabla\groups\myGroup</token-text>
> </arg-value>
> </do-add-dest-attr-value>
> </actions>
> </rule>
> <rule>
> <description>YYY Manage myGroup membership - removals</description>
> <conditions>
> <and>
> <if-class-name mode="nocase" op="equal">User</if-class-name>
> <if-op-attr mode="regex" name="myAttribute" op="changing-to">[^1]+</if-op-attr>


You had me until here. This literally means to match one or more
characters that is/are not a '1'. Not knowing what 'myAttribute' is for,
I'd likely do this some other way, both because a regex is slower (but
very powerful) and also because if your "add" value is "1" then your
remove value is probably either "0", or maybe removal of the attribute
entirely. Also, unless you specified when creating the attribute in
schema, attributes are multi-valued by default, so these two rules could
even clash; if you mean it to be single-valued, be sure it is defined that
way in schema so everything from LDAP to IDM can behave appropriately and
efficiently.

Back to the issue, I would probably, if the assumptions above are true,
change this watch for the other value you mentioned might be in here (e.g.
"0"), or if it's any other value, then check if value not-equal '1'. As
it is, this second rule will NOT fire if the value of myAttribute is
removed, because a non-existent value will not match a regex of [^1]+
which will only match one (1) or more characters. You could also have the
condition fire if the op-attr is "changing-from" the character '1', if
that logic makes sense.

> </and>
> </conditions>
> <actions>
> <do-remove-dest-attr-value class-name="User" name="Group Membership">
> <arg-value type="dn">
> <token-text xml:space="preserve">blabla\groups\myGroup</token-text>
> </arg-value>
> </do-remove-dest-attr-value>
> <do-remove-dest-attr-value class-name="User" name="Security Equals">
> <arg-value type="dn">
> <token-text xml:space="preserve">blabla\groups\myGroup</token-text>
> </arg-value>
> </do-remove-dest-attr-value>
> </actions>
> </rule>
>
>
> --------------------


--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
simeonof Trusted Contributor.
Trusted Contributor.

Re: Managing huge group memberships calculation

Thanks for the suggestions, here's the final rule that I've made (hope it will be useful for anyone):



<rule>
<description>Manage myGroup membership</description>
<comment xml:space="preserve">when the user attribute myAttribute changes</comment>
<conditions>
<and>
<if-class-name mode="nocase" op="equal">User</if-class-name>
<if-op-attr name="myAttribute" op="changing"/>
</and>
</conditions>
<actions>
<do-if>
<arg-conditions>
<and>
<if-op-attr mode="nocase" name="myAttribute" op="changing-to">1</if-op-attr>
</and>
</arg-conditions>
<arg-actions>
<do-add-dest-attr-value class-name="User" name="Group Membership">
<arg-value type="dn">
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-value>
</do-add-dest-attr-value>
<do-add-dest-attr-value class-name="User" name="Security Equals">
<arg-value type="dn">
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-value>
</do-add-dest-attr-value>
</arg-actions>
<arg-actions/>
</do-if>
<do-if>
<arg-conditions>
<and>
<if-op-attr mode="nocase" name="myAttribute" op="not-equal">1</if-op-attr>
</and>
</arg-conditions>
<arg-actions>
<do-remove-dest-attr-value class-name="User" name="Group Membership">
<arg-value type="dn">
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-value>
</do-remove-dest-attr-value>
<do-remove-dest-attr-value class-name="User" name="Security Equals">
<arg-value type="dn">
<token-text xml:space="preserve">blabla\groups\myGroup</token-text>
</arg-value>
</do-remove-dest-attr-value>
</arg-actions>
<arg-actions/>
</do-if>
</actions>
</rule>


Cheers
0 Likes
Knowledge Partner
Knowledge Partner

Re: Managing huge group memberships calculation

On 01/17/2019 05:14 AM, Simeonof wrote:
>
> Thanks for the suggestions, here's the final rule that I've made (hope
> it will be useful for anyone):


It is likely that it will be, so thank-you for sharing. It's satisfying
to have solutions to threads, as it seems too often I see a thread started
on a topic (not here, just online in general) and then no resolution, or a
resolution of, 'I fixed it" and nothing else.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.