Correct way to store last-update in REST driver

To all:

I am trying to maintain some XML inside DirXML-DriverStorage to manage polling within the REST driver. I am trying to figure out the “right” way to do this but the docs are incomplete (what a surprise) and the code is opaque so I don’t know exactly where to start.

I started to code this and found that the author of this driver seems to have had the same idea. With only one class (“Company”) defined so far, there was an XML fragment in the DirXML-DriverStorage attribute:

<init-params>
  <publisher-state>
    <Company_last-updated></Company_last-updated>
  </publisher-state>
</init-params>

Cool. But probably written there as text since a serialization would have written <Company_last-updated/>. Moving on.

In order to overcome the shortcoming of the polling mechanism…allow me to detour here:
  • I was happy to see that I could define a polling query right in the driver configuration. Seems totally logical.
  • I was unhappy to find that there is no way to put anything variable into that string. So I can’t do something like ?condition=lastUpdated>$lastPollTime
  • To work around that shortcoming, I poll for something that returns a short response consistently (user count) and then transform that into a nonsensical <sync> which is caught and stripped in the pub etp and bounced back as the query I should have been able to make right in the poll query had it been thought through by someone who actually builds IDM drivers.
Detour ends.

So what I need to do is to poll my end-point to get a list of Company objects. The list is large (>10K) so it has to be done in pages. Should be no-problem, except that the driver doesn’t support query-ex either. The API I am connecting to supports paged queries by just passing in a page number, so I can just increment the page numbers until I get back an empty JSON object (“[]”) and then terminate the loop. But since I have no query-ex support each query is going to look like a separate query, and with 15K of values will occasionally take more than the poll time. So another poll could come through while we are still processing the prior one.

My solution to that is to store the starting poll time in DirXML-DriverStorage as an attribute of the Company_last-update element and use that as a flag to ignore polls until the current one is finished processing (eventually there will be 4-5 classes read so I need to store this flag tied to the application class name). My intent was to copy the value from in-process to the text node so I have the last-update for the next poll, and set the in-process flag to false. So that all got coded and all was happy until I tried to write it back to DirXML-DriverStorage.

I read the XML into a local nodeset and manipulated the DOM to add what I want (I can see it did what I wish in trace). But when I try to write it back I get this:

[01/08/22 17:41:02.174]:ConnectWise PT: Evaluating selection criteria for rule 'setDriverStorage(lvDriverStorage);'.
[01/08/22 17:41:02.174]:ConnectWise PT: Rule selected.
[01/08/22 17:41:02.174]:ConnectWise PT: Applying rule 'setDriverStorage(lvDriverStorage);'.
[01/08/22 17:41:02.175]:ConnectWise PT: Action: do-trace-message(color="yellow",level="20",token-xml-serialize(token-local-variable("lvDriverStorage"))).
[01/08/22 17:41:02.175]:ConnectWise PT: arg-string(token-xml-serialize(token-local-variable("lvDriverStorage")))
[01/08/22 17:41:02.175]:ConnectWise PT: token-xml-serialize(token-local-variable("lvDriverStorage"))
[01/08/22 17:41:02.176]:ConnectWise PT: token-xml-serialize(token-local-variable("lvDriverStorage"))
[01/08/22 17:41:02.176]:ConnectWise PT: token-local-variable("lvDriverStorage")
[01/08/22 17:41:02.176]:ConnectWise PT: Token Value: {/}.
[01/08/22 17:41:02.176]:ConnectWise PT: Arg Value: {/}.
[01/08/22 17:41:02.177]:ConnectWise PT: Token Value: "<init-params>
<publisher-state>
<Company_last-updated in-progress="1641681662"/>
</publisher-state>
</init-params>".
[01/08/22 17:41:02.177]:ConnectWise PT: Arg Value: "<init-params>
<publisher-state>
<Company_last-updated in-progress="1641681662"/>
</publisher-state>
</init-params>".
[01/08/22 17:41:02.177]:ConnectWise PT:<init-params>
<publisher-state>
<Company_last-updated in-progress="1641681662"/>
</publisher-state>
</init-params>
[01/08/22 17:41:02.178]:ConnectWise PT: Action: do-trace-message(color="yellow",level="20",token-base64-encode(token-xml-serialize(token-local-variable("lvDriverStorage")))).
[01/08/22 17:41:02.178]:ConnectWise PT: arg-string(token-base64-encode(token-xml-serialize(token-local-variable("lvDriverStorage"))))
[01/08/22 17:41:02.179]:ConnectWise PT: token-base64-encode(token-xml-serialize(token-local-variable("lvDriverStorage")))
[01/08/22 17:41:02.179]:ConnectWise PT: token-base64-encode(token-xml-serialize(token-local-variable("lvDriverStorage")))
[01/08/22 17:41:02.180]:ConnectWise PT: token-xml-serialize(token-local-variable("lvDriverStorage"))
[01/08/22 17:41:02.180]:ConnectWise PT: token-xml-serialize(token-local-variable("lvDriverStorage"))
[01/08/22 17:41:02.180]:ConnectWise PT: token-local-variable("lvDriverStorage")
[01/08/22 17:41:02.180]:ConnectWise PT: Token Value: {/}.
[01/08/22 17:41:02.181]:ConnectWise PT: Arg Value: {/}.
[01/08/22 17:41:02.181]:ConnectWise PT: Token Value: "<init-params>
<publisher-state>
<Company_last-updated in-progress="1641681662"/>
</publisher-state>
</init-params>".
[01/08/22 17:41:02.181]:ConnectWise PT: Arg Value: "<init-params>
<publisher-state>
<Company_last-updated in-progress="1641681662"/>
</publisher-state>
</init-params>".
[01/08/22 17:41:02.182]:ConnectWise PT: Token Value: "PGluaXQtcGFyYW1zPgoJPHB1Ymxpc2hlci1zdGF0ZT4KCQk8Q29tcGFueV9sYXN0LXVwZGF0ZWQgaW4tcHJvZ3Jlc3M9IjE2NDE2ODE2NjIiLz4KCTwvcHVibGlzaGVyLXN0YXRlPgo8L2luaXQtcGFyYW1zPg==".
[01/08/22 17:41:02.182]:ConnectWise PT: Arg Value: "PGluaXQtcGFyYW1zPgoJPHB1Ymxpc2hlci1zdGF0ZT4KCQk8Q29tcGFueV9sYXN0LXVwZGF0ZWQgaW4tcHJvZ3Jlc3M9IjE2NDE2ODE2NjIiLz4KCTwvcHVibGlzaGVyLXN0YXRlPgo8L2luaXQtcGFyYW1zPg==".
[01/08/22 17:41:02.183]:ConnectWise PT:PGluaXQtcGFyYW1zPgoJPHB1Ymxpc2hlci1zdGF0ZT4KCQk8Q29tcGFueV9sYXN0LXVwZGF0ZWQgaW4tcHJvZ3Jlc3M9IjE2NDE2ODE2NjIiLz4KCTwvcHVibGlzaGVyLXN0YXRlPgo8L2luaXQtcGFyYW1zPg==
[01/08/22 17:41:02.184]:ConnectWise PT: Action: do-set-dest-attr-value("DirXML-DriverStorage",class-name="DirXML-Driver",direct="true",arg-dn(token-global-variable("dirxml.auto.driverdn")),token-base64-encode(token-xml-serialize(token-local-variable("lvDriverStorage")))).
[01/08/22 17:41:02.184]:ConnectWise PT: arg-dn(token-global-variable("dirxml.auto.driverdn"))
[01/08/22 17:41:02.185]:ConnectWise PT: token-global-variable("dirxml.auto.driverdn")
[01/08/22 17:41:02.185]:ConnectWise PT: Token Value: "\STUMP\system\Driver Set\CW Driver".
[01/08/22 17:41:02.185]:ConnectWise PT: Arg Value: "\STUMP\system\Driver Set\CW Driver".
[01/08/22 17:41:02.186]:ConnectWise PT:
DirXML Log Event -------------------
Driver: \STUMP\system\Driver Set\CW Driver
Channel: Publisher
Status: Error
Message: Code(-9010) An exception occurred: java.lang.NullPointerException
at com.novell.nds.dirxml.engine.rules.RuleUtil.findOrCreateModOperation(RuleUtil.java:1426)
at com.novell.nds.dirxml.engine.rules.DoAddDestAttrValue.apply(DoAddDestAttrValue.java:189)
at com.novell.nds.dirxml.engine.rules.ActionSet.apply(ActionSet.java:189)
at com.novell.nds.dirxml.engine.rules.DirXMLScriptProcessor.applyRules(DirXMLScriptProcessor.java:310)
at com.novell.nds.dirxml.engine.rules.DirXMLScriptProcessor.applyRules(DirXMLScriptProcessor.java:436)
at com.novell.nds.dirxml.engine.Transformer.applyInputTransformation(Transformer.java:333)
at com.novell.nds.dirxml.engine.Publisher.execute(Publisher.java:344)
at com.novell.nds.dirxml.driver.rest.RESTPublicationShim.poll(RESTPublicationShim.java:652)
at com.novell.nds.dirxml.driver.rest.RESTPublicationShim.start(RESTPublicationShim.java:292)
at com.novell.nds.dirxml.engine.Publisher.run(Publisher.java:607)
at java.lang.Thread.run(Thread.java:748)

I banged my head up against this one for a while and then finally decided it’s either a shim or engine bug, or it’s an undocumented limitation. It wasn’t a rights issue because I could go in to iDamager and update the attribute manually, so I wrote this little bit of ECMAScript to workaround this.

importPackage(Packages.com.novell.ldap);
importClass(java.security.Security);

var tracer = new Packages.com.novell.nds.dirxml.driver.Trace("ECMAScript");
const TRACE = 20;

function ldapModify(host, port, user, password, dn, attr, value)
{
tracer.trace("LDAPModifyWithTLS()",3);
var nodeSet = new Packages.com.novell.xml.xpath.NodeSet();
var document = Packages.com.novell.xml.dom.DocumentFactory.newDocument();
var ndsElement = document.createElement("nds");
document.appendChild(ndsElement);
ndsElement.setAttributeNS(null, "dtdversion", "3.5");
var outputElement = document.createElement("output");
ndsElement.appendChild(outputElement);

var error_p = null;

try
{

var lc = new LDAPConnection(new LDAPJSSEStartTLSFactory());
tracer.trace("create connection object",20);
lc.connect( host, port );
tracer.trace("bind to the server",20);
lc.bind( LDAPConnection.LDAP_V3, user, new java.lang.String(password).getBytes("UTF8") );
tracer.trace("Send modification",20);
lc.modify(dn, new LDAPModification(LDAPModification.REPLACE, new LDAPAttribute(attr,value)));
tracer.trace("return status",20);
var statusElement = document.createElement("status");
statusElement.setAttributeNS(null, "level", "success");
statusElement.appendChild(document.createTextNode("Written to "+attr+" attribute of "+dn));
outputElement.appendChild(statusElement);
nodeSet.add(statusElement);


}
catch(e)
{
//store the primary error in a variable
var error_p = e;
var statusElement = document.createElement("status");
statusElement.setAttributeNS(null, "level", "error");
statusElement.appendChild(document.createTextNode(e.toString()));
outputElement.appendChild(statusElement);
nodeSet.add(statusElement);
}

finally
{
try
{

//Stop the TLS protocol
lc.stopTLS();
// disconnect with the server
lc.disconnect();
}
catch(e)
{
if(error_p==null)
{
var statusElement = document.createElement("status");
statusElement.setAttributeNS(null, "level", "error");
statusElement.appendChild(document.createTextNode(e.toString()));
outputElement.appendChild(statusElement);
nodeSet.add(statusElement);
}
}

}

return nodeSet;
}

I verified this worked in iManager. I actually screwed it up, I forgot to remove the B64 encoding but was happy for a days work complete and shut the driver down.

The next morning before starting the driver I looked at the attribute and saw that something (probably during driver shutdown) replaced my B64 with blank XML. So chances are this approach is not going to work either; it’s going to get overwritten after each driver shutdown.

Which brings me finally to the core question here: How is this supposed to be done in the REST shim? How does it record the last-update time? I can figure out other workarounds but I would prefer to not replace delivered functionality if there was only some documentation that could tell me how to use it.

Any enlightenment would be greatly appreciated.
Thanks,
Rob