Highlighted
Absent Member.
Absent Member.
282 views

Multithreading IDM Drivers


Is there a way to multi thread a driver, meaning driver should process
multiple events at the same time. e.g. A SOAP driver can send multiple
request to the end point web service at any given point of time.
Currently it processes only one event at any given time, basically works
on FIFO structure. Can some one please help.


--
lsheshadri
------------------------------------------------------------------------
lsheshadri's Profile: https://forums.netiq.com/member.php?userid=4415
View this thread: https://forums.netiq.com/showthread.php?t=46885

Labels (1)
0 Likes
5 Replies
Highlighted
Absent Member.
Absent Member.

Re: Multithreading IDM Drivers

On 20.02.2013 10:34, lsheshadri wrote:
>
> Is there a way to multi thread a driver, meaning driver should process
> multiple events at the same time. e.g. A SOAP driver can send multiple
> request to the end point web service at any given point of time.
> Currently it processes only one event at any given time, basically works
> on FIFO structure. Can some one please help.


This is contrary to the whole event driven design of IDM.
There are plenty of examples where it's vital that events arrive in a
connected system in a specific order (org structure must be created
before you can assign a user to a department for example)


It's possible for events to execute on both subscriber and publisher
channels simultaneously, but there is no threading within a channel.

Also out of band operations can pause the current event and will resume
it when completed.

--
----------------------------------------------------------------------
Alex McHugh
NetIQ Knowledge Partner http://forums.netiq.com

Please post questions in the forums. No support is provided via email.
0 Likes
Highlighted
Absent Member.
Absent Member.

Re: Multithreading IDM Drivers


Thanks Alex, this is what I wanted to know. But this
has one drawback also, overall time for processing the
event becomes too long if there are large number of
events in the cache. Correct me if my assumption is
wrong.


alexmchugh;225729 Wrote:
> On 20.02.2013 10:34, lsheshadri wrote:
> >
> > Is there a way to multi thread a driver, meaning driver should

> process
> > multiple events at the same time. e.g. A SOAP driver can send

> multiple
> > request to the end point web service at any given point of time.
> > Currently it processes only one event at any given time, basically

> works
> > on FIFO structure. Can some one please help.

>
> This is contrary to the whole event driven design of IDM.
> There are plenty of examples where it's vital that events arrive in a
> connected system in a specific order (org structure must be created
> before you can assign a user to a department for example)
>
>
> It's possible for events to execute on both subscriber and publisher
> channels simultaneously, but there is no threading within a channel.
>
> Also out of band operations can pause the current event and will resume
> it when completed.
>
> --
> ----------------------------------------------------------------------
> Alex McHugh
> NetIQ Knowledge Partner http://forums.netiq.com
>
> Please post questions in the forums. No support is provided via email.



--
lsheshadri
------------------------------------------------------------------------
lsheshadri's Profile: https://forums.netiq.com/member.php?userid=4415
View this thread: https://forums.netiq.com/showthread.php?t=46885

0 Likes
Highlighted
Absent Member.
Absent Member.

Re: Multithreading IDM Drivers

On 20.02.2013 11:54, lsheshadri wrote:
>
> Thanks Alex, this is what I wanted to know. But this
> has one drawback also, overall time for processing the
> event becomes too long if there are large number of
> events in the cache. Correct me if my assumption is
> wrong.



If it is possible to sub-divide your events that are sent to this web
services interface, then you could create several copies of the SOAP
driver with event transform scoping to ensure that events are only
processed on one SOAP driver.

Not really the most ideal threading solution, but it would work
(assuming you can segment your events (by object class, hierarchy or
some other attribute)

--
----------------------------------------------------------------------
Alex McHugh
NetIQ Knowledge Partner http://forums.netiq.com

Please post questions in the forums. No support is provided via email.
0 Likes
Highlighted
Absent Member.
Absent Member.

Re: Multithreading IDM Drivers

I might add that Identity Manager's designed task is to manage identities,
specifically the lifecycle of identities, and as a result order matters.
Many companies, when first implementing IDM, will do a bulk-load to start
things off if there are millions of identities involved simply because
while doing things one at a time works very well (basically is required)
for the long-term, this bulk-load situation can usually be handled better.

For example, bulk-loading a new eDirectory tree using an LDIF and the
ldif2dib utility is much, much faster than synchronizing with IDM when you
get into larger environments. Not worth it for less than a few thousand
users, but once you have the process setup it will save a lot of time.
The same could apply for any interface that can take multiple events at a
time. IDM does one event at a time typically, and you can force it to do
multiples if you really want to, some applications can handle multiples in
a single transaction (JDBC, SOAP, etc.) so a quick export, mangling via
scripts, and then import can save a lot of time. Once done either
pre-populate the association values or just let IDM move on from there,
automatically "matching" whenever an event comes through for the affected
object.

Good luck.
0 Likes
Highlighted
Absent Member.
Absent Member.

Re: Multithreading IDM Drivers

On Wed, 20 Feb 2013 09:34:02 +0000, lsheshadri wrote:

> Is there a way to multi thread a driver, meaning driver should process
> multiple events at the same time.


Currently, no. You can, however, run multiple drivers in parallel,
assuming that the connected system can handle the load. I run 12 eDir
drivers in parallel, for example, with very careful scoping rules so that
they don't have any event crossover between them. I know of a site that
runs dozens of Google drivers in parallel in an attempt to make the
Google API perform reasonably well (it takes 7-10 seconds to create an
object in Google, so having one driver create a million objects takes a
very long time, but having 100 drivers create a million objects takes
somewhat less time, though it's not linear on Google's side).


> e.g. A SOAP driver can send multiple
> request to the end point web service at any given point of time.


But the replies coming back also have to be handled, and could, depending
on the reply, have side effects.


> Currently it processes only one event at any given time, basically works
> on FIFO structure. Can some one please help.


Correct. IDM, by design, does serial IO on each driver thread. Each
driver has two threads (publisher, subscriber).


--
--------------------------------------------------------------------------
David Gersic dgersic_@_niu.edu
Knowledge Partner http://forums.netiq.com

Please post questions in the forums. No support provided via email.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.