Error saving driver persistent data

Hi,

IDM 4.7.3 on SLES 12.

In ndstrace with dvrs I see this error, any tips on what to do?

 

3063748352 DVRS: ENG ET:
DirXML Log Event -------------------
Status: Error
Message: Code(-9203) Error saving driver persistent data: Code(-9010) An exception occurred: com.novell.nds.io.JCIOException: openStream: -166
at com.novell.nds.io.NDSStreamAttributeOutputStream.<init>(NDSStreamAttributeOutputStream.java:111)
at com.novell.nds.dirxml.engine.ConfigAbstraction.setAttributeStream(ConfigAbstraction.java:1646)
at com.novell.nds.dirxml.engine.ConfigAbstraction.setAttributeXML(ConfigAbstraction.java:1790)
at com.novell.nds.dirxml.engine.ConfigAbstraction.savePersistentData(ConfigAbstraction.java:1406)
at com.novell.nds.dirxml.engine.ConfigAbstraction.savePersistentData(ConfigAbstraction.java:1357)
at com.novell.nds.dirxml.engine.ConfigAbstraction.shutdown(ConfigAbstraction.java:621)
at com.novell.nds.dirxml.engine.GCVDefInfo.close(GCVDefInfo.java:145)
at com.novell.nds.dirxml.engine.verb.GetDriverGCV.processSubVerb(GetDriverGCV.java:127)
at com.novell.nds.dirxml.engine.verb.DirXMLVerbs$GetVerbHandler.processVerb(DirXMLVerbs.java:711)
at com.novell.nds.dhutil.VerbProcessor$HandlerThread.run(VerbProcessor.java:507)
at java.lang.Thread.run(Thread.java:748)
Caused by: novell.jclient.JCException: openStream -166 DSERR_ALREADY_IN_USE
at novell.jclient.JClient.openStream(Native Method)
at com.novell.nds.io.NDSStreamAttributeInputStream.openStream(NDSStreamAttributeInputStream.java:237)
at com.novell.nds.io.NDSStreamAttributeOutputStream.<init>(NDSStreamAttributeOutputStream.java:99)
... 10 more

  • Hi Alekz,
    It looks like a relatively old and known issue with driver persistent data "corruption".
    It described in the next TID
    https://support.microfocus.com/kb/doc.php?id=7004982

    Resolution
    Delete the attribute DirXML-PersistentData from the problem driver. This can be done via LDAP or ConsoleOne, iManager 2.7.3 does not provide access to the attribute. After that the driver should start normally.
    Additional Information
    DirXML-PersistentData stores counters for health monitoring. If the attribute XML content get corrupted, the errors mentioned in this TID will occur.

    Since it is is recreated when it does not exist and the driver is started, there is no loss of driver configuration or logic in deleting the DirXML-PersistentData attribute.
  • Also I am pretty sure that DirXML-PersistentData is per replica, which is why it is hard to delete with iManager. So make sure to clear it on any replicas that the driver could run on.

    Side note: Alekz - any chance when working with a Per-Replica attribute in Console2 do you think you could provide a UI to show all possible values on different replica?  That would be super duper handy!

  • Is this in relation to RRSD driver by any chance? I ask because I have seen this on a box that pretty much just runs RRSD and IDM Apps.

  • Specifically, since they added parallel processing support to RRSD, the driver uses the DirXML-DriverStorage attribute (a per replica attribute) to keep track of  things. This is documented here https://www.netiq.com/documentation/identity-manager-48/identity_apps_admin/data/multi-thread-rrs-troubleshooting.html

    When all is well, the attribute is updated once the processing is completed.

    What I suspect is we are seeing some sort of conflict or locking failure in updating this attribute.

    Running RRSD 4.7.3.0 IDM 4.7.2.0 AE  eDir 9.1.4 on windows where I see this issue.
    Interested if anyone else can report as to what versions they run.

  • We have the same problem after updating from 4.7.4 -> 4.8.2 HF1. On 4.7.4 I could not observe this...

    Our environment is: eDir 9.2.2, IDM 4.8.2 HF1 and RRSD 4.8.2.1

  • To delete DirXML-PersistentData on the RRSD driver will not help to resolve this issue.

  • Have you checked if there's an event in the cache that's triggering it?

    Does it work if you:

    - stop driver

    - dump cache (dxcmd or sneakycat)

    - clear cache

    - disable parallelization/multi-threading

    - remove .tao file

    - start driver

    - re-inject cache

  • You can get it it with iMonitor (advanced options enabled).

  • This is not a problem of a corrupt event in the queue. The driver is running for a while. Sometimes longer, sometimes not so long. Then he suddenly hangs up. In this state you can not start or stop the driver. You see only the following message in the driver log. No event processing anymore. When you take a deeper look with the Java Mission Control Center into the Java Threads you will see that there is no activity for the RRSD driver. Only a restart of the ndsd process will help here. After a restart the processing starts without problem and the queued events will be processed. This shows me that it can't be a corrupt event in the queue. We have opened a ticket at support. I am currently in the state to collect traces...

    [02/11/21 22:42:02.117]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:42:12.118]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:42:22.118]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:42:32.118]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:42:42.119]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:42:52.119]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:43:02.119]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:43:12.120]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:43:22.120]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:43:32.120]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:43:42.121]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:43:52.122]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:44:02.123]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
    [02/11/21 22:44:12.123]:RoleResourceServiceDriver ST:: Thread ID:258 Waiting for the completion of the already in progress requests
  • It could look like a dead thread which the main tread is waiting for. At this point I would suggest to contact support to get someone to look at this.