Highlighted
robert_maury Absent Member.
Absent Member.
259 views

Max number driver


Hello,

I would like to know the max number of driver i can use by driver engine
server.

Best regards


--
robert_maury
------------------------------------------------------------------------
robert_maury's Profile: https://forums.netiq.com/member.php?userid=9145
View this thread: https://forums.netiq.com/showthread.php?t=54330

Labels (1)
0 Likes
16 Replies
Knowledge Partner
Knowledge Partner

Re: Max number driver

robert maury wrote:

>
> I would like to know the max number of driver i can use by driver engine
> server.


That varies greatly depending on your environment, the real factor to consider is number of events is how busy your drivers will be (and what kinds of operations they will run). Not so much how many drivers you have.
Also it is recommended that where possible you host your driver shims in a remote loader where possible, this generally will improve the performance and scalability of the number of drivers your engine can handle.
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver

Alex McHugh wrote:

> > I would like to know the max number of driver i can use by driver engine
> > server.

>
> That varies greatly depending on your environment, the real factor to
> consider is number of events is how busy your drivers will be (and what kinds
> of operations they will run). Not so much how many drivers you have. Also it
> is recommended that where possible you host your driver shims in a remote
> loader where possible, this generally will improve the performance and
> scalability of the number of drivers your engine can handle.


On top of that, it also depends on how much RAM and CPU your server has, how
fast the storage is and what kind of driver you run. I know a large production
environment with 20000+ active users where ~50 busy drivers are spread over
basically two Linux boxes. If you tell us what kind of environment you plan to
build, we might be able to recommend how many servers of what kind seem
reasonable. If you start out with just a handful of drivers you're almost
always just fine with a single engine server (plus a second edir server holding
all replicas as cold/warm standby)
______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Max number driver

Things that matter in the realm of possibility:

RAM. If you lack RAM for running things, it just will not work. Many
years ago, when things were still 32-bit, a system was built with 100
drivers without modifying the engine or using Remote Loaders. They were
all the same driver (text) and it was just to show that the small max heap
used by default in the 32-bit days could do it, but it did. Today we use
x86_64, and the max heap is several times larger. You do the math.

Driver operations. A driver synchronizing on the subscriber channel all
day, especially if it's just sending regular users and groups, could
probably have hundreds on a server. Larger operations (query results,
large groups) take more memory per operation, and if you have hundreds of
drivers you will need the RAM to handle it. With that said, Subscriber
channel operations typically do not deal with tens of thousands of values
on an object at a time (other than during a merge). Publisher channel
operations, though, depend a lot on the application in question. If the
SOAP-API for an application cannot just return differences from a modified
object, you'll get bigger objects sent back to the engine all of the time.
If it cannot return a single object, you'll get large query results. If
your old microsoft active directory environment modifies a group, the
Publisher channel will show the entire group because that's how MAD used
to work (it's been fixed for several years now, but the driver config
still defaults to the old way to not use 'Incremental Values').

Considering how much RAM systems can have these days, and the capabilities
of x86_64 systems and the default heap there, you can probably run all
drivers on one server.

Going beyond what is simply possible we need to think about disk and CPU.
If you have a VM with two logical processors and 100 drivers, any tiny
modification will need to happen on those two drivers, maybe on each
driver (if each driver picks up the change made, like a user creation or a
password change). With a small operation (sans tracing) taking 1/10 of a
second, 100 drivers may take five seconds to complete the one change, and
so we see the environment needs more CPUs.

Similarly, any operation picked up by a driver's filter on the subscriber
channel, even if it will be vetoed (lack of entitlement, out of scope,
etc.) will be written to the disk cache for operations. Do this for 100
drivers and that's a little bit of I/O. Do this for thousands of objects
(bulk load operation, morning password changes, etc.) and it'll be still
more. Enterprise-grade disk systems should be fine with this, but it
depends a lot on the types of operations. All operations, whether
Subscriber or Publisher-channel initiated, could modify objects in the
vault, which will cause still more I/O. As this is Identity data there is
often a desire to flush things to disk quickly to avoid data loss in the
event of hardware issues (power outage, etc.), so systems are often tuned
more for reliability than speed.

I once converted a customer's environment of eighty or so drivers on four
servers to the same number of drivers on one server. They were concerned
it would not work, as they had had some issues in the past with timing of
events, etc. This was upgrading to IDM 4.0 SP2, and it has been just fine
since then; the server now has a lower load than it did before, despite
70k active users in the environment, because of some optimizations made to
the driver config at the time. As may be obvious, poorly-configured
driver objects can cause unnecessary work which could artificially inflate
hardware requirements.

My personal preference is to always have all drivers on one server, and
I've yet to hit an environment large enough that it did not work unless
something was very-badly broken because of an application that just
doesn't return data well (for example, having no way to query a single
user, so any query returns all users, consuming huge amounts of memory).
Not only is this simpler from the administration side of things, but it
also gives better overall performance because every change is seen by all
drivers at once instead of possibly having replication delays as changes
move from one engine server to another, then back to the first based on
changes made on the other, etc. Maintenance is also simpler with one
server to patch, one server to troubleshoot letting you see all traces on
a single system instead of pulling some from this server, some from that, etc.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver

On Tue, 22 Sep 2015 13:57:39 +0000, ab wrote:

> Things that matter in the realm of possibility:


Additionally, the WorkOrder driver can be especially memory hungry due to
the way it works.

There's also a difference between the RAM a driver needs during initial
startup and how much it needs when it's running. In the 32-bit days, I
was experiencing RAM crunch to the point where I could have all of my
drivers running, but only if I started them one at a time and waited for
each to initialize before starting the next. I think Java is also doing
some code sharing, so that loading multiples of the same driver does not
linearly increase the RAM needed to do so.


> Going beyond what is simply possible we need to think about disk and
> CPU.


In my experience, disk performance is much more important than CPU. Write
speed and especially write latency, are the things to watch for in a busy
IDM environment.


> Similarly, any operation picked up by a driver's filter on the
> subscriber channel, even if it will be vetoed (lack of entitlement, out
> of scope, etc.) will be written to the disk cache for operations.


Also be aware that one event can really be lots and lots of events for a
driver to process. If you have the defaults configured for password
synchronization, you're synching changes to the DirXML-PasswordSyncStatus
attribute, which all drivers update. So one password change may really be
50+ attribute updates as each driver processes the password change and
updates DirXML-PasswordSyncStatus. This is a useful attribute, if you're
using the UserApp / RBPM, and expose "password synchronization status" to
your users. If you aren't, it's just noise.


> My personal preference is to always have all drivers on one server, and
> I've yet to hit an environment large enough that it did not work


I kind of agree, but I've found it useful to have the shims hosted
closest to the data. Where the data is on Windows servers, that either
means splitting the vault (Linux / Windows) with multiple engines, or
using a bunch of remote loaders and one engine. Either works, of course.


> Not only is this simpler from the administration side of things, but it
> also gives better overall performance because every change is seen by
> all drivers at once instead of possibly having replication delays as
> changes move from one engine server to another, then back to the first
> based on changes made on the other, etc.


Sometimes I like to have it so that I can make a bulk load of changes to
a server without the engine, then let replication do the dirty work. It's
faster to update the database on a host without the engine running on it.


--
--------------------------------------------------------------------------
David Gersic dgersic_@_niu.edu
Knowledge Partner http://forums.microfocus.com

Please post questions in the forums. No support provided via email.
If you find this post helpful, please click on the star below.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver


Another "memory killer" can be driver, that use "wide" LDAP query (to
store results in Nodeset).


--
If you find this post helpful, please show your appreciation by clicking
on the star below :cool:
------------------------------------------------------------------------
al_b's Profile: https://forums.netiq.com/member.php?userid=209
View this thread: https://forums.netiq.com/showthread.php?t=54330

0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver

al b wrote:

>
> Another "memory killer" can be driver, that use "wide" LDAP query (to
> store results in Nodeset).


That could be avoided if the query token was enhanced further.
Most of the time LDAP queries are run due to limitations of the query token.
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver

On 9/22/2015 11:18 AM, Alex McHugh wrote:
> al b wrote:
>
>>
>> Another "memory killer" can be driver, that use "wide" LDAP query (to
>> store results in Nodeset).

>
> That could be avoided if the query token was enhanced further.
> Most of the time LDAP queries are run due to limitations of the query token.


I have nudged the dev team about this many times. I keep hoping it will
happen. But no word on that at all.

0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver


May be we have to create "collective" request for product extension and
put more voices under this request?
More people will sign this petition - more chance that we will receive
this extension for *query token*!


--
If you find this post helpful, please show your appreciation by clicking
on the star below :cool:
------------------------------------------------------------------------
al_b's Profile: https://forums.netiq.com/member.php?userid=209
View this thread: https://forums.netiq.com/showthread.php?t=54330

0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver

On 9/22/2015 1:24 PM, al b wrote:
>
> May be we have to create "collective" request for product extension and
> put more voices under this request?
> More people will sign this petition - more chance that we will receive
> this extension for *query token*!


IDM User Group might be a way to push this? I could post a GDocs Poll
and ask people to fill in if this interests them, and how many user
licenses they represent. That might be helpful.

Want to think about other issues to request as well? 🙂

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Max number driver

Geoffrey, why not create a poll here? Probably a bit more-visible, and
you can send a link to the poll to the user group as well.

Other things to improve: Designer performance in just about any way.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver

On Tue, 22 Sep 2015 18:28:09 +0000, Geoffrey Carman wrote:

> Want to think about other issues to request as well? 🙂


http://www.novell.com/rms 😉



--
--------------------------------------------------------------------------
David Gersic dgersic_@_niu.edu
Knowledge Partner http://forums.microfocus.com

Please post questions in the forums. No support provided via email.
If you find this post helpful, please click on the star below.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver

al b wrote:

>
> May be we have to create "collective" request for product extension and
> put more voices under this request?
> More people will sign this petition - more chance that we will receive
> this extension for *query token*!


I'm all for this.

Another thing that might help is providing a suggested way to implement this that doesn't break any existing functionality.

Some things to consider.

What we want is to be able to supply a query that offers most of the LDAP style query functionality. Myself I don't really need it to be LDAP syntax, just similar functionality

1. nested AND + OR, the current design does not really adapt itself to this.
To get or, you need to send multiple <query /> tokens and any union is not optimized out.
Nesting is not supported either.

2. Negation. Return everything where the value is not this.

3. It should be pageable (like query-ex)

4. Only really guaranteed to support the query to IDvault. However how should this behave if querying an app that does support all these things? Maybe needs a driver param to indicate whether this is actually supported by each shim

5. This is a nice to have, but it should be possible to generate via a local variable or DirXML-Script both the query criteria and the attributes to return.

As for implementation, maybe:

<query-ex>
<search-class class-name="User"/>
<search-conditions>
<or>
<search-attr attr-name="Surname">
<value>Foo</value>
</search-attr>
<search-attr attr-name="Surname">
<value>Bar</value>
</search-attr>
</or>
</search-conditions>
<read-attr/>
</query-ex>
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Max number driver

On 09/22/2015 03:12 PM, Alex McHugh wrote:
> al b wrote:
>
>>
>> May be we have to create "collective" request for product extension and
>> put more voices under this request?
>> More people will sign this petition - more chance that we will receive
>> this extension for *query token*!

>
> I'm all for this.
>
> Another thing that might help is providing a suggested way to implement this that doesn't break any existing functionality.
>
> Some things to consider.
>
> What we want is to be able to supply a query that offers most of the LDAP style query functionality. Myself I don't really need it to be LDAP syntax, just similar functionality


Sure, the syntax will be XML just because, and that's fine.

> 1. nested AND + OR, the current design does not really adapt itself to this.
> To get or, you need to send multiple <query /> tokens and any union is not optimized out.
> Nesting is not supported either.
>
> 2. Negation. Return everything where the value is not this.
>
> 3. It should be pageable (like query-ex)
>
> 4. Only really guaranteed to support the query to IDvault. However how should this behave if querying an app that does support all these things? Maybe needs a driver param to indicate whether this is actually supported by each shim
>
> 5. This is a nice to have, but it should be possible to generate via a local variable or DirXML-Script both the query criteria and the attributes to return.


The biggest thing with WorkOrder that leads to a problem here is
understanding more about syntax and handling appropriately, for example
range queries on time attributes (where time >= this or <= that) which
LDAP does very well and is part of some packages that optimize the
WorkOrder driver through their use.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Knowledge Partner
Knowledge Partner

Re: Max number driver

On 9/22/2015 10:30 AM, David Gersic wrote:
> On Tue, 22 Sep 2015 13:57:39 +0000, ab wrote:
>
>> Things that matter in the realm of possibility:

>
> Additionally, the WorkOrder driver can be especially memory hungry due to
> the way it works.


Recall, I have a package that fixes this. 🙂

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.