Anonymous_User Absent Member.
Absent Member.
228 views

Narrowing down OutOfMemory error


UA 4.0.2 patch D on Linux x64
I know E is out but customer is afraid of patching...

I'm troubleshooting an OOM error and I have enabled memory dumps on
OutOfMemory

errors.

The customer has a max heap size of 1.8GB according to recommendations
by NetIQ

support.

Looking through those dump I can see a large number of objects that take
up most

of the heap space when the OOM error occurs.

com.novell.srvprv.impl.vdata.model.VirtualEntity
com.novell.srvprv.impl.vdata.model.VirtualEntityAttribute

The VirtualEntity objects contain the VirtualEntityAttribute objects.

For example there are 8 398 VirtualEntity objects and 66 752

VirtualEntityAttribute objects that take over 1,2GB of heap.

The VirtualEntityAttribute objects that take up most of the space
contain

AssignedResources, InheritedRoles, MemberOf.

I'm trying to identify what's causing it to list *all* role/resource
memberships

for all those users.

Any ideas?

Would looking at a role/resource assignement tab in the role catalog
cause to

retrieve *all* values for a User?

Would using any of the built-in reports in UA cause this?

Can I put a limit on the query so that it doesn't try to list so many
users?

I have two stack traces here:

http://paste.opensuse.org/872f3169


http://paste.opensuse.org/76a8eabb


--
alekz
------------------------------------------------------------------------
alekz's Profile: https://forums.netiq.com/member.php?userid=974
View this thread: https://forums.netiq.com/showthread.php?t=52124

Labels (1)
0 Likes
12 Replies
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

Which web application service is used? x86_32 or x86_64?

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

The JBoss that comes with IDM 4.0.2 and 64-bit Java 7.

On 2014-11-06 23:00, ab wrote:
> Which web application service is used? x86_32 or x86_64?
>

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
do you also happen to have another message mentioning PermGen (a problem
with the permanent generation specifically) or GCOverheadException (a
setting in the garbage collector that throws the OOM error in certain
cases when garbage collection is not recovering much memory for the time
taken)?

Any ulimit settings apply to JBoss's java? Post the output of the
following, substituting the PID with the proper java process PID:


cat /proc/PID/limits


--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

On 11/07/2014 08:00 AM, ab wrote:
> Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
> do you also happen to have another message mentioning PermGen (a problem
> with the permanent generation specifically) or GCOverheadException (a
> setting in the garbage collector that throws the OOM error in certain
> cases when garbage collection is not recovering much memory for the time
> taken)?
>
> Any ulimit settings apply to JBoss's java? Post the output of the
> following, substituting the PID with the proper java process PID:
>
>

> cat /proc/PID/limits
>

>

Greetings,
I have seen this before from a few customers who have workflows that
will return all users in a certain role via a DAL Query. Some of these
Roles have many thousands of users assigned.



--

Sincerely,
Steven Williams
Lead Software Engineer
NetIQ
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

On 11/07/2014 09:35 PM, Steven Williams wrote:
> On 11/07/2014 08:00 AM, ab wrote:
>> Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
>> do you also happen to have another message mentioning PermGen (a problem
>> with the permanent generation specifically) or GCOverheadException (a
>> setting in the garbage collector that throws the OOM error in certain
>> cases when garbage collection is not recovering much memory for the time
>> taken)?
>>
>> Any ulimit settings apply to JBoss's java? Post the output of the
>> following, substituting the PID with the proper java process PID:
>>
>>

>> cat /proc/PID/limits
>>

>>

> Greetings,
> I have seen this before from a few customers who have workflows that
> will return all users in a certain role via a DAL Query. Some of these
> Roles have many thousands of users assigned.
>
>
>

Greetings,
If the case above is what you are seeing then you will need to limit
the results on the Entity in the DAL.

But I would suggest that you track what is going on to better
confirm what is going on. Also, apply the latest Public Patch for 4.0.2

--

Sincerely,
Steven Williams
Lead Software Engineer
NetIQ
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

Thanks Steven,
I will start to go through the PRD's looking for getRole* calls, to try
to identify the cause.

Do you know if it's possible from a memory dump to see maybe the logged
in users at the time, or to see the PRD that called the getRole API?



On 2014-11-08 03:41, Steven Williams wrote:
> On 11/07/2014 09:35 PM, Steven Williams wrote:
>> On 11/07/2014 08:00 AM, ab wrote:
>>> Do you have the hs_err*.log file from the OOM? Is it a plain old
>>> OOM, or
>>> do you also happen to have another message mentioning PermGen (a problem
>>> with the permanent generation specifically) or GCOverheadException (a
>>> setting in the garbage collector that throws the OOM error in certain
>>> cases when garbage collection is not recovering much memory for the time
>>> taken)?
>>>
>>> Any ulimit settings apply to JBoss's java? Post the output of the
>>> following, substituting the PID with the proper java process PID:
>>>
>>>

>>> cat /proc/PID/limits
>>>

>>>

>> Greetings,
>> I have seen this before from a few customers who have workflows that
>> will return all users in a certain role via a DAL Query. Some of these
>> Roles have many thousands of users assigned.
>>
>>
>>

> Greetings,
> If the case above is what you are seeing then you will need to limit
> the results on the Entity in the DAL.
>
> But I would suggest that you track what is going on to better confirm
> what is going on. Also, apply the latest Public Patch for 4.0.2
>

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error


Hello

A small update, if anyone is interested 😄

Further investigation of the heap dumps reveals that the VirtualEntity
objects contain VirtualEntityDefinition objects that point to the entity
"sys-nrf-user" which is not editable/visible in Designer. I can see it
in LDAP but I'm guessing that setting a query limit by editing the XML
is not supported.

On the bright side the analysis of the dumps shows which LDAP queries
were running at the time by looking at the
com.sun.jndi.ldap.LdapSearchEnumeration objects.
The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
points to a role containing 16000 members.
Also in the LDAP query it's asking for a number of attributes which
contain several hundred values each (nrfAssignedResources for example).

No PRDs use the sys-nrf-user entity so I'm guessing it must come from
the Roles and Resource tab itself, maybe someone is trying to view the
large role.



[1] java.lang.String "nrfAssignedResources"
[4] java.lang.String "nrfInheritedRoles"
[6] java.lang.String "nrfContainerRoles"
[11] java.lang.String "srvprvHideAttributes"
[5] java.lang.String "nrfGroupRoles"
[8] java.lang.String "nrfAssignedRoles"
[10] java.lang.String "srvprvHideUser"
[12] java.lang.String "modifyTimeStamp"
[7] java.lang.String "givenName"
[9] java.lang.String "nrfMemberOf"
[13] java.lang.String "objectClass"
[0] java.lang.String "mail"
[2] java.lang.String "sn"
[3] java.lang.String "cn"


--
alekz
------------------------------------------------------------------------
alekz's Profile: https://forums.netiq.com/member.php?userid=974
View this thread: https://forums.netiq.com/showthread.php?t=52124

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

On 11/12/2014 09:55 AM, alekz wrote:
>
> Hello
>
> A small update, if anyone is interested 😄
>
> Further investigation of the heap dumps reveals that the VirtualEntity
> objects contain VirtualEntityDefinition objects that point to the entity
> "sys-nrf-user" which is not editable/visible in Designer. I can see it
> in LDAP but I'm guessing that setting a query limit by editing the XML
> is not supported.
>
> On the bright side the analysis of the dumps shows which LDAP queries
> were running at the time by looking at the
> com.sun.jndi.ldap.LdapSearchEnumeration objects.
> The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
> points to a role containing 16000 members.
> Also in the LDAP query it's asking for a number of attributes which
> contain several hundred values each (nrfAssignedResources for example).
>
> No PRDs use the sys-nrf-user entity so I'm guessing it must come from
> the Roles and Resource tab itself, maybe someone is trying to view the
> large role.
>
>
>
> [1] java.lang.String "nrfAssignedResources"
> [4] java.lang.String "nrfInheritedRoles"
> [6] java.lang.String "nrfContainerRoles"
> [11] java.lang.String "srvprvHideAttributes"
> [5] java.lang.String "nrfGroupRoles"
> [8] java.lang.String "nrfAssignedRoles"
> [10] java.lang.String "srvprvHideUser"
> [12] java.lang.String "modifyTimeStamp"
> [7] java.lang.String "givenName"
> [9] java.lang.String "nrfMemberOf"
> [13] java.lang.String "objectClass"
> [0] java.lang.String "mail"
> [2] java.lang.String "sn"
> [3] java.lang.String "cn"
>
>

Greetings,
I would strongly suggest that you upgrade to Patch "E" for 4.0.2.
First of all there are ten (10) Security issues that have been fixed
with this patch. Also, for this specific bug fix that is listed in the
README (and on the Patch page for problems resolved:

Bug 873715 - Field Patch (402) Role and Resource Assignments tabs do not
work correctly with thousands of users assigned



Here is the direct link to Patch "E" for 4.0.2

https://dl.netiq.com/Download?buildid=MsOUtQILyLA~






--

Sincerely,
Steven Williams
Lead Software Engineer
NetIQ
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

Thanks Steven, hopefully the patch will be applied in January.

We managed to locate the user that was listing the roles and causing the
OutOfMemory error.


On 2014-11-12 19:01, Steven Williams wrote:
> On 11/12/2014 09:55 AM, alekz wrote:
>>
>> Hello
>>
>> A small update, if anyone is interested 😄
>>
>> Further investigation of the heap dumps reveals that the VirtualEntity
>> objects contain VirtualEntityDefinition objects that point to the entity
>> "sys-nrf-user" which is not editable/visible in Designer. I can see it
>> in LDAP but I'm guessing that setting a query limit by editing the XML
>> is not supported.
>>
>> On the bright side the analysis of the dumps shows which LDAP queries
>> were running at the time by looking at the
>> com.sun.jndi.ldap.LdapSearchEnumeration objects.
>> The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
>> points to a role containing 16000 members.
>> Also in the LDAP query it's asking for a number of attributes which
>> contain several hundred values each (nrfAssignedResources for example).
>>
>> No PRDs use the sys-nrf-user entity so I'm guessing it must come from
>> the Roles and Resource tab itself, maybe someone is trying to view the
>> large role.
>>
>>
>>
>> [1] java.lang.String "nrfAssignedResources"
>> [4] java.lang.String "nrfInheritedRoles"
>> [6] java.lang.String "nrfContainerRoles"
>> [11] java.lang.String "srvprvHideAttributes"
>> [5] java.lang.String "nrfGroupRoles"
>> [8] java.lang.String "nrfAssignedRoles"
>> [10] java.lang.String "srvprvHideUser"
>> [12] java.lang.String "modifyTimeStamp"
>> [7] java.lang.String "givenName"
>> [9] java.lang.String "nrfMemberOf"
>> [13] java.lang.String "objectClass"
>> [0] java.lang.String "mail"
>> [2] java.lang.String "sn"
>> [3] java.lang.String "cn"
>>
>>

> Greetings,
> I would strongly suggest that you upgrade to Patch "E" for 4.0.2.
> First of all there are ten (10) Security issues that have been fixed
> with this patch. Also, for this specific bug fix that is listed in the
> README (and on the Patch page for problems resolved:
>
> Bug 873715 - Field Patch (402) Role and Resource Assignments tabs do not
> work correctly with thousands of users assigned
>
>
>
> Here is the direct link to Patch "E" for 4.0.2
>
> https://dl.netiq.com/Download?buildid=MsOUtQILyLA~
>
>
>
>
>
>

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

Hi ab,
I can check the ulimit setting in a couple of days...

No hs_err files, I think they only get created when the JVM crashes
which is not the case here.

On 2014-11-07 14:00, ab wrote:
> Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
> do you also happen to have another message mentioning PermGen (a problem
> with the permanent generation specifically) or GCOverheadException (a
> setting in the garbage collector that throws the OOM error in certain
> cases when garbage collection is not recovering much memory for the time
> taken)?
>
> Any ulimit settings apply to JBoss's java? Post the output of the
> following, substituting the PID with the proper java process PID:
>
>

> cat /proc/PID/limits
>

>

0 Likes
jtl1 Absent Member.
Absent Member.

Re: Narrowing down OutOfMemory error

I've seen memory problems when "Perform Automatic Query" is set on the entity properties and attributes with a large number of
values is in the entity (or any of its DN lookup entities). I think it was because memberOf was included on role and/or group in
a combination of automatic query on the user entity which had a DN Lookup showing role and/or group...

Best regards,
Tobias

On 2014-11-06 22:20, alekz wrote:
>
> UA 4.0.2 patch D on Linux x64
> I know E is out but customer is afraid of patching...
>
> I'm troubleshooting an OOM error and I have enabled memory dumps on
> OutOfMemory
>
> errors.
>
> The customer has a max heap size of 1.8GB according to recommendations
> by NetIQ
>
> support.
>
> Looking through those dump I can see a large number of objects that take
> up most
>
> of the heap space when the OOM error occurs.
>
> com.novell.srvprv.impl.vdata.model.VirtualEntity
> com.novell.srvprv.impl.vdata.model.VirtualEntityAttribute
>
> The VirtualEntity objects contain the VirtualEntityAttribute objects.
>
> For example there are 8 398 VirtualEntity objects and 66 752
>
> VirtualEntityAttribute objects that take over 1,2GB of heap.
>
> The VirtualEntityAttribute objects that take up most of the space
> contain
>
> AssignedResources, InheritedRoles, MemberOf.
>
> I'm trying to identify what's causing it to list *all* role/resource
> memberships
>
> for all those users.
>
> Any ideas?
>
> Would looking at a role/resource assignement tab in the role catalog
> cause to
>
> retrieve *all* values for a User?
>
> Would using any of the built-in reports in UA cause this?
>
> Can I put a limit on the query so that it doesn't try to list so many
> users?
>
> I have two stack traces here:
>
> http://paste.opensuse.org/872f3169
>
>
> http://paste.opensuse.org/76a8eabb
>
>


0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.