Narrowing down OutOfMemory error


UA 4.0.2 patch D on Linux x64
I know E is out but customer is afraid of patching...

I'm troubleshooting an OOM error and I have enabled memory dumps on
OutOfMemory

errors.

The customer has a max heap size of 1.8GB according to recommendations
by NetIQ

support.

Looking through those dump I can see a large number of objects that take
up most

of the heap space when the OOM error occurs.

com.novell.srvprv.impl.vdata.model.VirtualEntity
com.novell.srvprv.impl.vdata.model.VirtualEntityAttribute

The VirtualEntity objects contain the VirtualEntityAttribute objects.

For example there are 8 398 VirtualEntity objects and 66 752

VirtualEntityAttribute objects that take over 1,2GB of heap.

The VirtualEntityAttribute objects that take up most of the space
contain

AssignedResources, InheritedRoles, MemberOf.

I'm trying to identify what's causing it to list *all* role/resource
memberships

for all those users.

Any ideas?

Would looking at a role/resource assignement tab in the role catalog
cause to

retrieve *all* values for a User?

Would using any of the built-in reports in UA cause this?

Can I put a limit on the query so that it doesn't try to list so many
users?

I have two stack traces here:

http://paste.opensuse.org/872f3169


http://paste.opensuse.org/76a8eabb


--
alekz
------------------------------------------------------------------------
alekz's Profile: https://forums.netiq.com/member.php?userid=974
View this thread: https://forums.netiq.com/showthread.php?t=52124

Parents
  • Which web application service is used? x86_32 or x86_64?

    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...
  • The JBoss that comes with IDM 4.0.2 and 64-bit Java 7.

    On 2014-11-06 23:00, ab wrote:
    > Which web application service is used? x86_32 or x86_64?
    >

  • Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
    do you also happen to have another message mentioning PermGen (a problem
    with the permanent generation specifically) or GCOverheadException (a
    setting in the garbage collector that throws the OOM error in certain
    cases when garbage collection is not recovering much memory for the time
    taken)?

    Any ulimit settings apply to JBoss's java? Post the output of the
    following, substituting the PID with the proper java process PID:


    cat /proc/PID/limits


    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...
  • On 11/07/2014 08:00 AM, ab wrote:
    > Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
    > do you also happen to have another message mentioning PermGen (a problem
    > with the permanent generation specifically) or GCOverheadException (a
    > setting in the garbage collector that throws the OOM error in certain
    > cases when garbage collection is not recovering much memory for the time
    > taken)?
    >
    > Any ulimit settings apply to JBoss's java? Post the output of the
    > following, substituting the PID with the proper java process PID:
    >
    >

    > cat /proc/PID/limits
    >

    >

    Greetings,
    I have seen this before from a few customers who have workflows that
    will return all users in a certain role via a DAL Query. Some of these
    Roles have many thousands of users assigned.



    --

    Sincerely,
    Steven Williams
    Lead Software Engineer
    NetIQ
  • On 11/07/2014 09:35 PM, Steven Williams wrote:
    > On 11/07/2014 08:00 AM, ab wrote:
    >> Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
    >> do you also happen to have another message mentioning PermGen (a problem
    >> with the permanent generation specifically) or GCOverheadException (a
    >> setting in the garbage collector that throws the OOM error in certain
    >> cases when garbage collection is not recovering much memory for the time
    >> taken)?
    >>
    >> Any ulimit settings apply to JBoss's java? Post the output of the
    >> following, substituting the PID with the proper java process PID:
    >>
    >>

    >> cat /proc/PID/limits
    >>

    >>

    > Greetings,
    > I have seen this before from a few customers who have workflows that
    > will return all users in a certain role via a DAL Query. Some of these
    > Roles have many thousands of users assigned.
    >
    >
    >

    Greetings,
    If the case above is what you are seeing then you will need to limit
    the results on the Entity in the DAL.

    But I would suggest that you track what is going on to better
    confirm what is going on. Also, apply the latest Public Patch for 4.0.2

    --

    Sincerely,
    Steven Williams
    Lead Software Engineer
    NetIQ
  • Hi ab,
    I can check the ulimit setting in a couple of days...

    No hs_err files, I think they only get created when the JVM crashes
    which is not the case here.

    On 2014-11-07 14:00, ab wrote:
    > Do you have the hs_err*.log file from the OOM? Is it a plain old OOM, or
    > do you also happen to have another message mentioning PermGen (a problem
    > with the permanent generation specifically) or GCOverheadException (a
    > setting in the garbage collector that throws the OOM error in certain
    > cases when garbage collection is not recovering much memory for the time
    > taken)?
    >
    > Any ulimit settings apply to JBoss's java? Post the output of the
    > following, substituting the PID with the proper java process PID:
    >
    >

    > cat /proc/PID/limits
    >

    >

  • Thanks Steven,
    I will start to go through the PRD's looking for getRole* calls, to try
    to identify the cause.

    Do you know if it's possible from a memory dump to see maybe the logged
    in users at the time, or to see the PRD that called the getRole API?



    On 2014-11-08 03:41, Steven Williams wrote:
    > On 11/07/2014 09:35 PM, Steven Williams wrote:
    >> On 11/07/2014 08:00 AM, ab wrote:
    >>> Do you have the hs_err*.log file from the OOM? Is it a plain old
    >>> OOM, or
    >>> do you also happen to have another message mentioning PermGen (a problem
    >>> with the permanent generation specifically) or GCOverheadException (a
    >>> setting in the garbage collector that throws the OOM error in certain
    >>> cases when garbage collection is not recovering much memory for the time
    >>> taken)?
    >>>
    >>> Any ulimit settings apply to JBoss's java? Post the output of the
    >>> following, substituting the PID with the proper java process PID:
    >>>
    >>>

    >>> cat /proc/PID/limits
    >>>

    >>>

    >> Greetings,
    >> I have seen this before from a few customers who have workflows that
    >> will return all users in a certain role via a DAL Query. Some of these
    >> Roles have many thousands of users assigned.
    >>
    >>
    >>

    > Greetings,
    > If the case above is what you are seeing then you will need to limit
    > the results on the Entity in the DAL.
    >
    > But I would suggest that you track what is going on to better confirm
    > what is going on. Also, apply the latest Public Patch for 4.0.2
    >


  • Hello

    A small update, if anyone is interested :D

    Further investigation of the heap dumps reveals that the VirtualEntity
    objects contain VirtualEntityDefinition objects that point to the entity
    "sys-nrf-user" which is not editable/visible in Designer. I can see it
    in LDAP but I'm guessing that setting a query limit by editing the XML
    is not supported.

    On the bright side the analysis of the dumps shows which LDAP queries
    were running at the time by looking at the
    com.sun.jndi.ldap.LdapSearchEnumeration objects.
    The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
    points to a role containing 16000 members.
    Also in the LDAP query it's asking for a number of attributes which
    contain several hundred values each (nrfAssignedResources for example).

    No PRDs use the sys-nrf-user entity so I'm guessing it must come from
    the Roles and Resource tab itself, maybe someone is trying to view the
    large role.



    [1] java.lang.String "nrfAssignedResources"
    [4] java.lang.String "nrfInheritedRoles"
    [6] java.lang.String "nrfContainerRoles"
    [11] java.lang.String "srvprvHideAttributes"
    [5] java.lang.String "nrfGroupRoles"
    [8] java.lang.String "nrfAssignedRoles"
    [10] java.lang.String "srvprvHideUser"
    [12] java.lang.String "modifyTimeStamp"
    [7] java.lang.String "givenName"
    [9] java.lang.String "nrfMemberOf"
    [13] java.lang.String "objectClass"
    [0] java.lang.String "mail"
    [2] java.lang.String "sn"
    [3] java.lang.String "cn"


    --
    alekz
    ------------------------------------------------------------------------
    alekz's Profile: https://forums.netiq.com/member.php?userid=974
    View this thread: https://forums.netiq.com/showthread.php?t=52124

  • On 11/12/2014 09:55 AM, alekz wrote:
    >
    > Hello
    >
    > A small update, if anyone is interested :D
    >
    > Further investigation of the heap dumps reveals that the VirtualEntity
    > objects contain VirtualEntityDefinition objects that point to the entity
    > "sys-nrf-user" which is not editable/visible in Designer. I can see it
    > in LDAP but I'm guessing that setting a query limit by editing the XML
    > is not supported.
    >
    > On the bright side the analysis of the dumps shows which LDAP queries
    > were running at the time by looking at the
    > com.sun.jndi.ldap.LdapSearchEnumeration objects.
    > The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
    > points to a role containing 16000 members.
    > Also in the LDAP query it's asking for a number of attributes which
    > contain several hundred values each (nrfAssignedResources for example).
    >
    > No PRDs use the sys-nrf-user entity so I'm guessing it must come from
    > the Roles and Resource tab itself, maybe someone is trying to view the
    > large role.
    >
    >
    >
    > [1] java.lang.String "nrfAssignedResources"
    > [4] java.lang.String "nrfInheritedRoles"
    > [6] java.lang.String "nrfContainerRoles"
    > [11] java.lang.String "srvprvHideAttributes"
    > [5] java.lang.String "nrfGroupRoles"
    > [8] java.lang.String "nrfAssignedRoles"
    > [10] java.lang.String "srvprvHideUser"
    > [12] java.lang.String "modifyTimeStamp"
    > [7] java.lang.String "givenName"
    > [9] java.lang.String "nrfMemberOf"
    > [13] java.lang.String "objectClass"
    > [0] java.lang.String "mail"
    > [2] java.lang.String "sn"
    > [3] java.lang.String "cn"
    >
    >

    Greetings,
    I would strongly suggest that you upgrade to Patch "E" for 4.0.2.
    First of all there are ten (10) Security issues that have been fixed
    with this patch. Also, for this specific bug fix that is listed in the
    README (and on the Patch page for problems resolved:

    Bug 873715 - Field Patch (402) Role and Resource Assignments tabs do not
    work correctly with thousands of users assigned



    Here is the direct link to Patch "E" for 4.0.2

    https://dl.netiq.com/Download?buildid=MsOUtQILyLA~






    --

    Sincerely,
    Steven Williams
    Lead Software Engineer
    NetIQ
Reply
  • On 11/12/2014 09:55 AM, alekz wrote:
    >
    > Hello
    >
    > A small update, if anyone is interested :D
    >
    > Further investigation of the heap dumps reveals that the VirtualEntity
    > objects contain VirtualEntityDefinition objects that point to the entity
    > "sys-nrf-user" which is not editable/visible in Designer. I can see it
    > in LDAP but I'm guessing that setting a query limit by editing the XML
    > is not supported.
    >
    > On the bright side the analysis of the dumps shows which LDAP queries
    > were running at the time by looking at the
    > com.sun.jndi.ldap.LdapSearchEnumeration objects.
    > The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
    > points to a role containing 16000 members.
    > Also in the LDAP query it's asking for a number of attributes which
    > contain several hundred values each (nrfAssignedResources for example).
    >
    > No PRDs use the sys-nrf-user entity so I'm guessing it must come from
    > the Roles and Resource tab itself, maybe someone is trying to view the
    > large role.
    >
    >
    >
    > [1] java.lang.String "nrfAssignedResources"
    > [4] java.lang.String "nrfInheritedRoles"
    > [6] java.lang.String "nrfContainerRoles"
    > [11] java.lang.String "srvprvHideAttributes"
    > [5] java.lang.String "nrfGroupRoles"
    > [8] java.lang.String "nrfAssignedRoles"
    > [10] java.lang.String "srvprvHideUser"
    > [12] java.lang.String "modifyTimeStamp"
    > [7] java.lang.String "givenName"
    > [9] java.lang.String "nrfMemberOf"
    > [13] java.lang.String "objectClass"
    > [0] java.lang.String "mail"
    > [2] java.lang.String "sn"
    > [3] java.lang.String "cn"
    >
    >

    Greetings,
    I would strongly suggest that you upgrade to Patch "E" for 4.0.2.
    First of all there are ten (10) Security issues that have been fixed
    with this patch. Also, for this specific bug fix that is listed in the
    README (and on the Patch page for problems resolved:

    Bug 873715 - Field Patch (402) Role and Resource Assignments tabs do not
    work correctly with thousands of users assigned



    Here is the direct link to Patch "E" for 4.0.2

    https://dl.netiq.com/Download?buildid=MsOUtQILyLA~






    --

    Sincerely,
    Steven Williams
    Lead Software Engineer
    NetIQ
Children
  • Thanks Steven, hopefully the patch will be applied in January.

    We managed to locate the user that was listing the roles and causing the
    OutOfMemory error.


    On 2014-11-12 19:01, Steven Williams wrote:
    > On 11/12/2014 09:55 AM, alekz wrote:
    >>
    >> Hello
    >>
    >> A small update, if anyone is interested :D
    >>
    >> Further investigation of the heap dumps reveals that the VirtualEntity
    >> objects contain VirtualEntityDefinition objects that point to the entity
    >> "sys-nrf-user" which is not editable/visible in Designer. I can see it
    >> in LDAP but I'm guessing that setting a query limit by editing the XML
    >> is not supported.
    >>
    >> On the bright side the analysis of the dumps shows which LDAP queries
    >> were running at the time by looking at the
    >> com.sun.jndi.ldap.LdapSearchEnumeration objects.
    >> The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
    >> points to a role containing 16000 members.
    >> Also in the LDAP query it's asking for a number of attributes which
    >> contain several hundred values each (nrfAssignedResources for example).
    >>
    >> No PRDs use the sys-nrf-user entity so I'm guessing it must come from
    >> the Roles and Resource tab itself, maybe someone is trying to view the
    >> large role.
    >>
    >>
    >>
    >> [1] java.lang.String "nrfAssignedResources"
    >> [4] java.lang.String "nrfInheritedRoles"
    >> [6] java.lang.String "nrfContainerRoles"
    >> [11] java.lang.String "srvprvHideAttributes"
    >> [5] java.lang.String "nrfGroupRoles"
    >> [8] java.lang.String "nrfAssignedRoles"
    >> [10] java.lang.String "srvprvHideUser"
    >> [12] java.lang.String "modifyTimeStamp"
    >> [7] java.lang.String "givenName"
    >> [9] java.lang.String "nrfMemberOf"
    >> [13] java.lang.String "objectClass"
    >> [0] java.lang.String "mail"
    >> [2] java.lang.String "sn"
    >> [3] java.lang.String "cn"
    >>
    >>

    > Greetings,
    > I would strongly suggest that you upgrade to Patch "E" for 4.0.2.
    > First of all there are ten (10) Security issues that have been fixed
    > with this patch. Also, for this specific bug fix that is listed in the
    > README (and on the Patch page for problems resolved:
    >
    > Bug 873715 - Field Patch (402) Role and Resource Assignments tabs do not
    > work correctly with thousands of users assigned
    >
    >
    >
    > Here is the direct link to Patch "E" for 4.0.2
    >
    > https://dl.netiq.com/Download?buildid=MsOUtQILyLA~
    >
    >
    >
    >
    >
    >

  • Thanks Steven, hopefully the patch will be applied in January.

    We managed to locate the user that was listing the roles and causing the
    OutOfMemory error.


    On 2014-11-12 19:01, Steven Williams wrote:
    > On 11/12/2014 09:55 AM, alekz wrote:
    >>
    >> Hello
    >>
    >> A small update, if anyone is interested :D
    >>
    >> Further investigation of the heap dumps reveals that the VirtualEntity
    >> objects contain VirtualEntityDefinition objects that point to the entity
    >> "sys-nrf-user" which is not editable/visible in Designer. I can see it
    >> in LDAP but I'm guessing that setting a query limit by editing the XML
    >> is not supported.
    >>
    >> On the bright side the analysis of the dumps shows which LDAP queries
    >> were running at the time by looking at the
    >> com.sun.jndi.ldap.LdapSearchEnumeration objects.
    >> The LDAP filter point to inetOrgPerson objects with nrfMemberOf that
    >> points to a role containing 16000 members.
    >> Also in the LDAP query it's asking for a number of attributes which
    >> contain several hundred values each (nrfAssignedResources for example).
    >>
    >> No PRDs use the sys-nrf-user entity so I'm guessing it must come from
    >> the Roles and Resource tab itself, maybe someone is trying to view the
    >> large role.
    >>
    >>
    >>
    >> [1] java.lang.String "nrfAssignedResources"
    >> [4] java.lang.String "nrfInheritedRoles"
    >> [6] java.lang.String "nrfContainerRoles"
    >> [11] java.lang.String "srvprvHideAttributes"
    >> [5] java.lang.String "nrfGroupRoles"
    >> [8] java.lang.String "nrfAssignedRoles"
    >> [10] java.lang.String "srvprvHideUser"
    >> [12] java.lang.String "modifyTimeStamp"
    >> [7] java.lang.String "givenName"
    >> [9] java.lang.String "nrfMemberOf"
    >> [13] java.lang.String "objectClass"
    >> [0] java.lang.String "mail"
    >> [2] java.lang.String "sn"
    >> [3] java.lang.String "cn"
    >>
    >>

    > Greetings,
    > I would strongly suggest that you upgrade to Patch "E" for 4.0.2.
    > First of all there are ten (10) Security issues that have been fixed
    > with this patch. Also, for this specific bug fix that is listed in the
    > README (and on the Patch page for problems resolved:
    >
    > Bug 873715 - Field Patch (402) Role and Resource Assignments tabs do not
    > work correctly with thousands of users assigned
    >
    >
    >
    > Here is the direct link to Patch "E" for 4.0.2
    >
    > https://dl.netiq.com/Download?buildid=MsOUtQILyLA~
    >
    >
    >
    >
    >
    >