Anonymous_User Absent Member.
Absent Member.
327 views

Any way to speed up RRSD?

Hello,
IDM 402 Patch C.

We have about 1500 users that had roles and resources that expired 31/3.
That caused a bunch of events in the queue of the RRSD driver that it's
still processing and from the looks of it it's going to take 2 more days.

Each user takes several minutes to process.

Any tips on speeding it up?
I can see in DSTrace that a bunch of queries are being performed for
roles and resources. (Using +RECM).

All driver trace levels are 0.
Labels (1)
0 Likes
10 Replies
Anonymous_User Absent Member.
Absent Member.

Re: Any way to speed up RRSD?

On 2014-04-02 17:01, alekz wrote:
> Hello,
> IDM 402 Patch C.
>
> We have about 1500 users that had roles and resources that expired 31/3.
> That caused a bunch of events in the queue of the RRSD driver that it's
> still processing and from the looks of it it's going to take 2 more days.
>
> Each user takes several minutes to process.
>
> Any tips on speeding it up?
> I can see in DSTrace that a bunch of queries are being performed for
> roles and resources. (Using +RECM).
>
> All driver trace levels are 0.
>

We have about 46000 objects in the ResourceRequests container.
Requests container has 16000 objects.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Any way to speed up RRSD?

On 04/02/2014 11:03 AM, alekz wrote:
> On 2014-04-02 17:01, alekz wrote:
>> Hello,
>> IDM 402 Patch C.
>>
>> We have about 1500 users that had roles and resources that expired 31/3.
>> That caused a bunch of events in the queue of the RRSD driver that it's
>> still processing and from the looks of it it's going to take 2 more days.
>>
>> Each user takes several minutes to process.
>>
>> Any tips on speeding it up?
>> I can see in DSTrace that a bunch of queries are being performed for
>> roles and resources. (Using +RECM).
>>
>> All driver trace levels are 0.
>>

> We have about 46000 objects in the ResourceRequests container.
> Requests container has 16000 objects.
>

Greetings,
Are you sure that you did the special instructions for Patch C.
There have been fixes to the RRSD and if you did not manually replace
the jar (on each Vault server) then you do not have fixes.
I have also seen the driver be slow depending upon indexes and the
actual HD.

--

Sincerely,
Steven Williams
Lead Software Engineer
NetIQ
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Any way to speed up RRSD?

On 2014-04-02 19:50, Steven Williams wrote:
> On 04/02/2014 11:03 AM, alekz wrote:
>> On 2014-04-02 17:01, alekz wrote:
>>> Hello,
>>> IDM 402 Patch C.
>>>
>>> We have about 1500 users that had roles and resources that expired 31/3.
>>> That caused a bunch of events in the queue of the RRSD driver that it's
>>> still processing and from the looks of it it's going to take 2 more
>>> days.
>>>
>>> Each user takes several minutes to process.
>>>
>>> Any tips on speeding it up?
>>> I can see in DSTrace that a bunch of queries are being performed for
>>> roles and resources. (Using +RECM).
>>>
>>> All driver trace levels are 0.
>>>

>> We have about 46000 objects in the ResourceRequests container.
>> Requests container has 16000 objects.
>>

> Greetings,
> Are you sure that you did the special instructions for Patch C. There
> have been fixes to the RRSD and if you did not manually replace the jar
> (on each Vault server) then you do not have fixes.
> I have also seen the driver be slow depending upon indexes and the
> actual HD.
>

Hello,

I'll check with the people who did the patching, I was not involved.
What should the driver version be for nrfdriver.jar in patch C?

I can't find anything regarding performance fixes in the patch notes.
Are there fixes for nrfdriver.jar that are not documented in the notes?

Regarding indexes, I have checked that they have all the regular
nrf-indexes. In ndstrace, the queries that are performed in regards to
the nrf-attributes, I don't see any mention of "no index used". But I'm
seeing alot of NOT filters used in the queries (!(nrfStatus...., if I
remember correctly there is a TID mentioning that such searches don't
use indexes at all, but that TID talks about LDAP searches and not the
JClient stuff that rrsd seem to use so it may not apply to this situation.

I think that the writing is the real bottleneck here, since ndsd rarely
goes above 100% cpu utilization, it is pretty stable on 100-115% which
should indicate that alot of writing is going on which makes sense since
it must clean up different DN references I guess.

iostat also indicates that it's mostly writes going to the volume
containing the DIB.

Unfortunately the server is virtual, I'll try to get info on the disk
subsystem.

Thanks Steven.

-alekz










0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Any way to speed up RRSD?

On Wed, 02 Apr 2014 19:42:48 +0000, alekz wrote:

> I think that the writing is the real bottleneck here


If so, the iMonitor page that shows the writer threads waiting for the DIB
lock would show this pretty clearly.


--
--------------------------------------------------------------------------
David Gersic dgersic_@_niu.edu
Knowledge Partner http://forums.netiq.com

Please post questions in the forums. No support provided via email.
If you find this post helpful, please click on the star below.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Any way to speed up RRSD?

On 2014-04-02 22:30, David Gersic wrote:
> On Wed, 02 Apr 2014 19:42:48 +0000, alekz wrote:
>
>> I think that the writing is the real bottleneck here

>
> If so, the iMonitor page that shows the writer threads waiting for the DIB
> lock would show this pretty clearly.
>
>

Hmm there is some writing going but not as much I thought, not for the
entire time it takes RRSD to process 1 event. I guess it has to check
each role/resource the user has before it can safely remove a role/resource.

It has problems processing removal of roles/resource, those take much
more time than assigning roles/resources, minutes, sometimes it "works"
on an event for 5-10 minutes.
Each processed user caused a bunch of new requests to be created by the
RRSD and written to the directory.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Any way to speed up RRSD?

On 4/3/2014 11:34 AM, alekz wrote:
> On 2014-04-02 22:30, David Gersic wrote:
>> On Wed, 02 Apr 2014 19:42:48 +0000, alekz wrote:
>>
>>> I think that the writing is the real bottleneck here

>>
>> If so, the iMonitor page that shows the writer threads waiting for the DIB
>> lock would show this pretty clearly.
>>
>>

> Hmm there is some writing going but not as much I thought, not for the
> entire time it takes RRSD to process 1 event. I guess it has to check
> each role/resource the user has before it can safely remove a role/resource.


I originally wondered how it worked, since it did not look very event
driven.

I have an article written, waiting for publication, going through trace
samples of all the things the RRSD driver does.

Basically ANY event on a user, group, OU that looks role related causes
the engine to send a command, instead of an event (per se) to the
driver. Then it looks like the driver reads all the info about the
user, looks at its current state and tries to top it up. That is,
re-evaluate the current state of the object that changed, make sure it
is properly reconciled at the end of the event.

It is very Oracle IDM like, which I think is a bad thing. However, I
think that the inherent design flaw in how Roles are implemented by
basically ignoring the fact that eDir is a directory and not a database
leaves few other options. I truly wish that Roles used real
inheritance, which would make things so much easier.

> It has problems processing removal of roles/resource, those take much
> more time than assigning roles/resources, minutes, sometimes it "works"
> on an event for 5-10 minutes.
> Each processed user caused a bunch of new requests to be created by the
> RRSD and written to the directory.


As others have noted, they use Jclient in the shim itself, which in
principle should be the fastest way to do directory operations.

But it writes direct to eDir, it does not send events back to the engine
to write to eDir. That is, it is very un-IDM like. Sort of odd
approach. Would love to hear the back story on that one day. I assume
this approach is probably faster than doing it via the engine, but
clearly not enough to succeed at being fast enough.




0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Any way to speed up RRSD?

On 04/03/2014 11:34 AM, alekz wrote:
> On 2014-04-02 22:30, David Gersic wrote:
>> On Wed, 02 Apr 2014 19:42:48 +0000, alekz wrote:
>>
>>> I think that the writing is the real bottleneck here

>>
>> If so, the iMonitor page that shows the writer threads waiting for the DIB
>> lock would show this pretty clearly.
>>
>>

> Hmm there is some writing going but not as much I thought, not for the
> entire time it takes RRSD to process 1 event. I guess it has to check
> each role/resource the user has before it can safely remove a role/resource.
>
> It has problems processing removal of roles/resource, those take much
> more time than assigning roles/resources, minutes, sometimes it "works"
> on an event for 5-10 minutes.
> Each processed user caused a bunch of new requests to be created by the
> RRSD and written to the directory.
>

Greetings,
During a Revocation, we do have to double check if the user should
indeed be removed from the Role/Resource. Depending upon your structure
and if the same resource can be assigned from different Roles and/or
with different values, it can add time.


--

Sincerely,
Steven Williams
Lead Software Engineer
NetIQ
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Any way to speed up RRSD?

On 2014-04-03 21:48, Steven Williams wrote:
> On 04/03/2014 11:34 AM, alekz wrote:
>> On 2014-04-02 22:30, David Gersic wrote:
>>> On Wed, 02 Apr 2014 19:42:48 +0000, alekz wrote:
>>>
>>>> I think that the writing is the real bottleneck here
>>>
>>> If so, the iMonitor page that shows the writer threads waiting for
>>> the DIB
>>> lock would show this pretty clearly.
>>>
>>>

>> Hmm there is some writing going but not as much I thought, not for the
>> entire time it takes RRSD to process 1 event. I guess it has to check
>> each role/resource the user has before it can safely remove a
>> role/resource.
>>
>> It has problems processing removal of roles/resource, those take much
>> more time than assigning roles/resources, minutes, sometimes it "works"
>> on an event for 5-10 minutes.
>> Each processed user caused a bunch of new requests to be created by the
>> RRSD and written to the directory.
>>

> Greetings,
> During a Revocation, we do have to double check if the user should
> indeed be removed from the Role/Resource. Depending upon your structure
> and if the same resource can be assigned from different Roles and/or
> with different values, it can add time.
>
>

Probably the amount of expirations will spread out over a longer period
of time next year so that the system doesn't have to process all
revocations at once as it did now.
This deployment also has *many* resources that have dynamic values and
they can be assigned by different roles as you commented.

-alekz
0 Likes
Knowledge Partner
Knowledge Partner

Re: Any way to speed up RRSD?

> Are you sure that you did the special instructions for Patch C. There
> have been fixes to the RRSD and if you did not manually replace the jar
> (on each Vault server) then you do not have fixes.
> I have also seen the driver be slow depending upon indexes and the
> actual HD.


Can you recommend specific attributes to index, that would be helpful
for RRSD performance?



0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Any way to speed up RRSD?


We have the nrfdriver.jar from patch C, v4.0.0.6430


--
alekz
------------------------------------------------------------------------
alekz's Profile: https://forums.netiq.com/member.php?userid=974
View this thread: https://forums.netiq.com/showthread.php?t=50431

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.