Which web application service is used? x86_32 or x86_64?

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

The JBoss that comes with IDM 4.0.2 and 64-bit Java 7.

On 2014-11-06 23:00, ab wrote:
> Which web application service is used? x86_32 or x86_64?
>

Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
do you also happen to have another message mentioning PermGen (a problem
with the permanent generation specifically) or GCOverheadException (a
setting in the garbage collector that throws the OOM error in certain
cases when garbage collection is not recovering much memory for the time
taken)?

Any ulimit settings apply to JBoss's java? Post the output of the
following, substituting the PID with the proper java process PID:

cat /proc/PID/limits

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

On 11/07/2014 08:00 AM, ab wrote:
> Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
> do you also happen to have another message mentioning PermGen (a problem
> with the permanent generation specifically) or GCOverheadException (a
> setting in the garbage collector that throws the OOM error in certain
> cases when garbage collection is not recovering much memory for the time
> taken)?
>
> Any ulimit settings apply to JBoss's java? Post the output of the
> following, substituting the PID with the proper java process PID:
>
>

> cat /proc/PID/limits
> 

>
Greetings,
I have seen this before from a few customers who have workflows that
will return all users in a certain role via a DAL Query. Some of these
Roles have many thousands of users assigned.



--

Sincerely,
Steven Williams
Lead Software Engineer
NetIQ

On 11/07/2014 09:35 PM, Steven Williams wrote:
> On 11/07/2014 08:00 AM, ab wrote:
>> Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
>> do you also happen to have another message mentioning PermGen (a problem
>> with the permanent generation specifically) or GCOverheadException (a
>> setting in the garbage collector that throws the OOM error in certain
>> cases when garbage collection is not recovering much memory for the time
>> taken)?
>>
>> Any ulimit settings apply to JBoss's java? Post the output of the
>> following, substituting the PID with the proper java process PID:
>>
>>

>> cat /proc/PID/limits
>> 

>>
> Greetings,
> I have seen this before from a few customers who have workflows that
> will return all users in a certain role via a DAL Query. Some of these
> Roles have many thousands of users assigned.
>
>
>
Greetings,
If the case above is what you are seeing then you will need to limit
the results on the Entity in the DAL.

But I would suggest that you track what is going on to better
confirm what is going on. Also, apply the latest Public Patch for 4.0.2

--

Sincerely,
Steven Williams
Lead Software Engineer
NetIQ

Hi ab,
I can check the ulimit setting in a couple of days...

No hs_err files, I think they only get created when the JVM crashes
which is not the case here.

On 2014-11-07 14:00, ab wrote:
> Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
> do you also happen to have another message mentioning PermGen (a problem
> with the permanent generation specifically) or GCOverheadException (a
> setting in the garbage collector that throws the OOM error in certain
> cases when garbage collection is not recovering much memory for the time
> taken)?
>
> Any ulimit settings apply to JBoss's java? Post the output of the
> following, substituting the PID with the proper java process PID:
>
>

> cat /proc/PID/limits
> 

>

Thanks Steven,
I will start to go through the PRD's looking for getRole* calls, to try
to identify the cause.

Do you know if it's possible from a memory dump to see maybe the logged
in users at the time, or to see the PRD that called the getRole API?



On 2014-11-08 03:41, Steven Williams wrote:
> On 11/07/2014 09:35 PM, Steven Williams wrote:
>> On 11/07/2014 08:00 AM, ab wrote:
>>> Do you have the hs_err*.log file from the OOM? Is it a plain old
>>> OOM, or
>>> do you also happen to have another message mentioning PermGen (a problem
>>> with the permanent generation specifically) or GCOverheadException (a
>>> setting in the garbage collector that throws the OOM error in certain
>>> cases when garbage collection is not recovering much memory for the time
>>> taken)?
>>>
>>> Any ulimit settings apply to JBoss's java? Post the output of the
>>> following, substituting the PID with the proper java process PID:
>>>
>>>

>>> cat /proc/PID/limits
>>> 

>>>
>> Greetings,
>> I have seen this before from a few customers who have workflows that
>> will return all users in a certain role via a DAL Query. Some of these
>> Roles have many thousands of users assigned.
>>
>>
>>
> Greetings,
> If the case above is what you are seeing then you will need to limit
> the results on the Entity in the DAL.
>
> But I would suggest that you track what is going on to better confirm
> what is going on. Also, apply the latest Public Patch for 4.0.2
>

Hello

A small update, if anyone is interested :D

Further investigation of the heap dumps reveals that the VirtualEntity
objects contain VirtualEntityDefinition objects that point to the entity
"sys-nrf-user" which is not editable/visible in Designer. I can see it
in LDAP but I'm guessing that setting a query limit by editing the XML
is not supported.

On the bright side the analysis of the dumps shows which LDAP queries
were running at the time by looking at the
com.sun.jndi.ldap.LdapSearchEnumeration objects.
The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
points to a role containing 16000 members.
Also in the LDAP query it's asking for a number of attributes which
contain several hundred values each (nrfAssignedResources for example).

No PRDs use the sys-nrf-user entity so I'm guessing it must come from
the Roles and Resource tab itself, maybe someone is trying to view the
large role.



[1] java.lang.String "nrfAssignedResources"
[4] java.lang.String "nrfInheritedRoles"
[6] java.lang.String "nrfContainerRoles"
[11] java.lang.String "srvprvHideAttributes"
[5] java.lang.String "nrfGroupRoles"
[8] java.lang.String "nrfAssignedRoles"
[10] java.lang.String "srvprvHideUser"
[12] java.lang.String "modifyTimeStamp"
[7] java.lang.String "givenName"
[9] java.lang.String "nrfMemberOf"
[13] java.lang.String "objectClass"
[0] java.lang.String "mail"
[2] java.lang.String "sn"
[3] java.lang.String "cn"


--
alekz
------------------------------------------------------------------------
alekz's Profile: https://forums.netiq.com/member.php?userid=974
View this thread: https://forums.netiq.com/showthread.php?t=52124

On 11/12/2014 09:55 AM, alekz wrote:
>
> Hello
>
> A small update, if anyone is interested :D
>
> Further investigation of the heap dumps reveals that the VirtualEntity
> objects contain VirtualEntityDefinition objects that point to the entity
> "sys-nrf-user" which is not editable/visible in Designer. I can see it
> in LDAP but I'm guessing that setting a query limit by editing the XML
> is not supported.
>
> On the bright side the analysis of the dumps shows which LDAP queries
> were running at the time by looking at the
> com.sun.jndi.ldap.LdapSearchEnumeration objects.
> The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
> points to a role containing 16000 members.
> Also in the LDAP query it's asking for a number of attributes which
> contain several hundred values each (nrfAssignedResources for example).
>
> No PRDs use the sys-nrf-user entity so I'm guessing it must come from
> the Roles and Resource tab itself, maybe someone is trying to view the
> large role.
>
>
>
> [1] java.lang.String "nrfAssignedResources"
> [4] java.lang.String "nrfInheritedRoles"
> [6] java.lang.String "nrfContainerRoles"
> [11] java.lang.String "srvprvHideAttributes"
> [5] java.lang.String "nrfGroupRoles"
> [8] java.lang.String "nrfAssignedRoles"
> [10] java.lang.String "srvprvHideUser"
> [12] java.lang.String "modifyTimeStamp"
> [7] java.lang.String "givenName"
> [9] java.lang.String "nrfMemberOf"
> [13] java.lang.String "objectClass"
> [0] java.lang.String "mail"
> [2] java.lang.String "sn"
> [3] java.lang.String "cn"
>
>
Greetings,
I would strongly suggest that you upgrade to Patch "E" for 4.0.2.
First of all there are ten (10) Security issues that have been fixed
with this patch. Also, for this specific bug fix that is listed in the
README (and on the Patch page for problems resolved:

Bug 873715 - Field Patch (402) Role and Resource Assignments tabs do not
work correctly with thousands of users assigned



Here is the direct link to Patch "E" for 4.0.2

https://dl.netiq.com/Download?buildid=MsOUtQILyLA~






--

Sincerely,
Steven Williams
Lead Software Engineer
NetIQ

Thanks Steven, hopefully the patch will be applied in January.

We managed to locate the user that was listing the roles and causing the
OutOfMemory error.


On 2014-11-12 19:01, Steven Williams wrote:
> On 11/12/2014 09:55 AM, alekz wrote:
>>
>> Hello
>>
>> A small update, if anyone is interested :D
>>
>> Further investigation of the heap dumps reveals that the VirtualEntity
>> objects contain VirtualEntityDefinition objects that point to the entity
>> "sys-nrf-user" which is not editable/visible in Designer. I can see it
>> in LDAP but I'm guessing that setting a query limit by editing the XML
>> is not supported.
>>
>> On the bright side the analysis of the dumps shows which LDAP queries
>> were running at the time by looking at the
>> com.sun.jndi.ldap.LdapSearchEnumeration objects.
>> The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
>> points to a role containing 16000 members.
>> Also in the LDAP query it's asking for a number of attributes which
>> contain several hundred values each (nrfAssignedResources for example).
>>
>> No PRDs use the sys-nrf-user entity so I'm guessing it must come from
>> the Roles and Resource tab itself, maybe someone is trying to view the
>> large role.
>>
>>
>>
>> [1] java.lang.String "nrfAssignedResources"
>> [4] java.lang.String "nrfInheritedRoles"
>> [6] java.lang.String "nrfContainerRoles"
>> [11] java.lang.String "srvprvHideAttributes"
>> [5] java.lang.String "nrfGroupRoles"
>> [8] java.lang.String "nrfAssignedRoles"
>> [10] java.lang.String "srvprvHideUser"
>> [12] java.lang.String "modifyTimeStamp"
>> [7] java.lang.String "givenName"
>> [9] java.lang.String "nrfMemberOf"
>> [13] java.lang.String "objectClass"
>> [0] java.lang.String "mail"
>> [2] java.lang.String "sn"
>> [3] java.lang.String "cn"
>>
>>
> Greetings,
> I would strongly suggest that you upgrade to Patch "E" for 4.0.2.
> First of all there are ten (10) Security issues that have been fixed
> with this patch. Also, for this specific bug fix that is listed in the
> README (and on the Patch page for problems resolved:
>
> Bug 873715 - Field Patch (402) Role and Resource Assignments tabs do not
> work correctly with thousands of users assigned
>
>
>
> Here is the direct link to Patch "E" for 4.0.2
>
> https://dl.netiq.com/Download?buildid=MsOUtQILyLA~
>
>
>
>
>
>

Thanks Steven, hopefully the patch will be applied in January.

We managed to locate the user that was listing the roles and causing the
OutOfMemory error.


On 2014-11-12 19:01, Steven Williams wrote:
> On 11/12/2014 09:55 AM, alekz wrote:
>>
>> Hello
>>
>> A small update, if anyone is interested :D
>>
>> Further investigation of the heap dumps reveals that the VirtualEntity
>> objects contain VirtualEntityDefinition objects that point to the entity
>> "sys-nrf-user" which is not editable/visible in Designer. I can see it
>> in LDAP but I'm guessing that setting a query limit by editing the XML
>> is not supported.
>>
>> On the bright side the analysis of the dumps shows which LDAP queries
>> were running at the time by looking at the
>> com.sun.jndi.ldap.LdapSearchEnumeration objects.
>> The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
>> points to a role containing 16000 members.
>> Also in the LDAP query it's asking for a number of attributes which
>> contain several hundred values each (nrfAssignedResources for example).
>>
>> No PRDs use the sys-nrf-user entity so I'm guessing it must come from
>> the Roles and Resource tab itself, maybe someone is trying to view the
>> large role.
>>
>>
>>
>> [1] java.lang.String "nrfAssignedResources"
>> [4] java.lang.String "nrfInheritedRoles"
>> [6] java.lang.String "nrfContainerRoles"
>> [11] java.lang.String "srvprvHideAttributes"
>> [5] java.lang.String "nrfGroupRoles"
>> [8] java.lang.String "nrfAssignedRoles"
>> [10] java.lang.String "srvprvHideUser"
>> [12] java.lang.String "modifyTimeStamp"
>> [7] java.lang.String "givenName"
>> [9] java.lang.String "nrfMemberOf"
>> [13] java.lang.String "objectClass"
>> [0] java.lang.String "mail"
>> [2] java.lang.String "sn"
>> [3] java.lang.String "cn"
>>
>>
> Greetings,
> I would strongly suggest that you upgrade to Patch "E" for 4.0.2.
> First of all there are ten (10) Security issues that have been fixed
> with this patch. Also, for this specific bug fix that is listed in the
> README (and on the Patch page for problems resolved:
>
> Bug 873715 - Field Patch (402) Role and Resource Assignments tabs do not
> work correctly with thousands of users assigned
>
>
>
> Here is the direct link to Patch "E" for 4.0.2
>
> https://dl.netiq.com/Download?buildid=MsOUtQILyLA~
>
>
>
>
>
>