Recently, one of my clients asked me to create a custom administration capability to replace portlets which they had created in IDM User Application version 3.7 and had been working in 4.0.2. Under 4.6, these portlets no longer functioned. I created a PRD which consolidated seven of their portlets into a single management application which provided all the necessary functionality and leveraged the same mechanism they used for granting access (which were eDirectory trustee rights).

They loved it, except for one thing: The built-in search capability in a DN field is akin to the basic search in the portlet. They needed an equivalent of the "advanced search". The standard search allowed you to search on a single attribute but did not enable searches with more complex combinations of attributes. Advanced Search in the portlets allows this.

My first attempt was borrowed from an earlier client who had built something similar using the DAL Global Query. Their implementation was to use multiple variables, and multiple query attributes in the global query to represent seven or so different attributes they might want to look up. Using the LDAP syntax of a preceding asterisk meaning ends with, a following asterisk meaning starts with and both meaning contains, they could query quite flexibly. Any attribute that was not being searched on was matched with an asterisk, and each of these were and-ed together, so the result was that only the attributes where you specified a value came up in the search.

Worked great...but…

As I was testing this something bizarre was happening. After searching for a user, and modifying that user using the PRD, the user would no longer come up in the search. I picked a user that would search, exported their object as an LDIF, modified them with the PRD, and created another LDIF. The comparison showed that five attributes had been lost after submitting the form. I put back the attributes manually and the user could be searched once again.

After correcting the broken workflow, I set my sights on the search limitation. What I realized was that if any of the searched attributes were unvalued, the object did not appear in the search results. Now if you have a very reliable and complete authoritative source that is populating every single value you wish to search on within every single object in your directory, then this approach works fine. My latest client, however, populates their directory only when source data is available (which comes from Workday.com).

I tried for a bit to craft a better query strategy using the IDVault.GlobalQuery() which would serve my needs. I kept thinking I needed to include each attribute grouped in an or clause which was that the attribute equals * or the attribute does not equal * which takes care of the attributes we don't want to filter on, but for the ones we do, attribute equals foo* and attribute not equals a nonce. I tried this, but it was slow, and did not result in a reliable way to not include attributes I didn't want to search on on this query, nor a way to create an even more complex query that would result in the correct response.

So I knuckled down, and started working this angle: Query the VDX directly. Immediately there were some things to learn:

• The VDX Query Language

• The SOAP Request for the VDX query

• The SOAP Response for a VDX query

But before we get into that, let's talk enabling technologies.

Enabling Technologies

The first thing to understand here is that you need to formulate a SOAP XML message to send to the VDX service. Fortunately, most modern brower's javascript interpreter includes XML support. You can use this within the form code. You need to instantiate a number of XML objects within the browser.

You do this by adding the following code to the form onload:

/** enable XML support **/
 function createXMLHttpRequest() {
   try { return new XMLHttpRequest(); } catch(e) {}
   try { return new ActiveXObject("Msxml2.XMLHTTP"); } catch (e) {}
   form.alert("XMLHttpRequest not supported by this browser");
   return null;
 }

xmlhttp = createXMLHttpRequest();

SOAP is built on XML and is a mechanism for programs to communicate with other programs via an exchange of documents. SOAP only defines the standard for the top level XML envelope of the message. The contents of the message depend on the application. These contents are known as "vocabularies". The names of most of these end in "ML", for example: SPML, DSML, SAML, etc.

Many applications also define their own vocabularies. The services within the IDM User Application fall into that category. In order to know what the request format is for messages, you need yet one more XML document known as the "WSDL", or "Web Services Description Language". Fortunately, most web services have this available somewhere.

In the case of the VDX service within IDM, you can get to the WSDL via the following URL:

http://myserver:8080/IDMProv/vdx/service?wsdl. Once you have the WSDL, you need to extract the sample requests from it. I use a tool from SmartBear called SoapUI for this purpose. You can simply create a new SOAP project, specify the URL of the WSDL, and it will provide sample request documents.



You can then manually test sending requests by editing the XML by hand, and pressing the run (right facing green arrow) button. The response will be provided in the right pane.

Note that you may have to put in credentials in the panel at the lower left.



There are other ways to test SOAP, such as the command line tool "curl", but this is a great tool for interpreting the WSDL and testing your messages.

Working with XML can be challenging. Sure, it's just text, but the correct way to deal with it is to leverage the Document Object Model ("DOM"). DOM provides a set of methods and properties to allow you to easily manipulate XML documents.

DOM is confusing and esoteric, written by some clever programmers who wrapped their brains around a way to represent and manipulate an XML document in a generic way from multiple object oriented programming languages. More power to them. It took me a while, but I know enough about it now to recognize that there are several things I do frequently. I have created a library of functions which I include in the script tab to simplify my XML manipulation. Here are some of them:

function xmlParse(XML)
{
var oParser = new DOMParser();
var oDOM = oParser.parseFromString(XML, "text/xml");
return oDOM;
}

function xmlSetText(DOM, elementName, value, dex)
{
if (undef(dex)) dex = 0;
var LMent = DOM.getElementsByTagName(elementName)[dex]
var textNode = xmlGetTextNode(LMent)
if (undef(textNode))
	{
	textNode = 	DOM.createTextNode(value),
	LMent.appendChild(textNode);
	}
else
	textNode.nodeValue = value;
}

function xmlSetAttribute(DOM, elementName, attributeName, value, dex)
{
if (undef(dex)) dex = 0;
var attrNode = DOM.getElementsByTagName(elementName)[dex].getAttributeNode(attributeName);
attrNode.nodeValue = value;
}

function xmlGetText(elem)
{

var textNode = xmlGetTextNode(elem);
if (textNode)
	{
	return textNode.nodeValue;
	}
else
	{
	return "";
	}
}


function xmlGetTextNode(elem) {
// NOT GENERIC. assumes only child node within the element is the text node
return elem.firstChild;
}

function xmlSerialize(DOM)
{
return new XMLSerializer().serializeToString(DOM);
}

function getXMLResponse(form,field,xmlhttp,url,domRequest,callbackFunction)
{

txtRequest = xmlSerialize(domRequest);

xmlhttp.open("POST",url,true);
xmlhttp.withCredentials = true;
xmlhttp.setRequestHeader("Content-type", "text/xml");
xmlhttp.send(txtRequest);
xmlhttp.onreadystatechange= callbackFunction;
}


function xmlGetAttribute(elem, attributeName)
{
if (elem.hasAttribute(attributeName))
	return elem.getAttribute(attributeName)
else
	return "";
}

function xmlGetTextByAttribute(DOM, elementName, attributeName, value)
{
var textNodes = DOM.getElementsByTagName(elementName)
for (var i = 0; i < textNodes.length; i  )
	{
	var attrNode = textNodes[i].getAttributeNode(attributeName);
	if (attrNode.nodeValue == value)
		{
		return textNodes[i].childNodes[0]
		}
	}
return "";	
}

That being said, rather than build the document from scratch, I do often start with a blank sample document in text, which makes it easier to manipulate. This came from SoapUI but I removed the comments to keep it clean:

function queryDoc() 
{
var Q='<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://www.novell.com/vdx/service">'
   Q ='<soapenv:Header/>'
   Q ='   <soapenv:Body>'
   Q ='      <ser:queryRequest>'
   Q ='         <ser:arg0></ser:arg0>'
   Q ='         <ser:arg1></ser:arg1>'
   Q ='         <ser:arg2></ser:arg2>'
   Q ='      </ser:queryRequest>'
   Q ='   </soapenv:Body>'
   Q ='</soapenv:Envelope>'
return Q;
}

The next thing to understand is how to send an XML request and receive and XML response within a form. Like most things in IDM, it starts with an event. In this case, though, it is a javascript event within the browser. One way to generate an event is to add a button to the form. This too is not standardized, but I have my own preferred way to do it. I add the following to the scripts tab:

/* adds a button to a field. When clicked the button publishes an event named "onclick-fieldname-buttonname" */
function addButton(field,name,label,fieldName)
{
if (undef(fieldName))
	fieldName = field.getName();
	
var ctrl = JUICE.UICtrlUtil.getControl(field.getName());
var btn = JUICE.UICtrlUtil.addButton(ctrl, name , label, name, function() {field.fireEvent("onclick-" fieldName "-" name,field.getName())}); 
}

Then I add something like this to the onload of a field:

/* field ADVSRCH_mode_advanced */
addButton(field,"search","Search")

When this button is clicked, it fired an event named onclick-ADVSRCH_mode_advanced-search. So now let's next examine how we prepare the XML document to be transmitted, and how we retrieve data from the XML document. In that event handler we need to place code to transmit our XML document. In this example, domRequest is a DOM object containing the XML request:

getXMLResponse(form,field,xmlhttp,"/IDMProv/vdx/service",domRequest, OnStateChange);

It should be noted that you cannot use anything but a relative URL to the user application
server in this call. The reason for this is known as the "Same Origin Policy". See https://en.wikipedia.org/wiki/Same-origin_policy for more details.

The other important parameter in this example is "OnStateChange". This is the name of a javascript function. The function is also included in the same script:

function OnStateChange() {
            if (xmlhttp.readyState==4) {
		    field.fireEvent("onreceive-ADVSRCH_mode_advanced",xmlhttp.responseText)
			}
            }

When our XML response is received, it will be in the form of a string, in the responseText property of the xmlhttp object. All this function does is to take that value and fire another event, where the received response will be processed.

Finally, let's discuss how to prepare our request and to retrieve the results. This particular bit of code prepares a shell of the request document by filling in the arguments of the calls:

function vdxQuery(entityKey, queryString)
{

ATTRS = ["LastName", "FirstName", "CN", "ACMELastSSN","workforceID","L", "employeeStatus","PasswordExpirationTime","LockedByIntruder","ACMEAuthLockout", "Email","manager"]

var domRequest = xmlParse(queryDoc());
xmlSetText(domRequest, "ser:arg0", entityKey)
xmlSetText(domRequest, "ser:arg2", queryString)

ATTRS = ["LastName", "FirstName", "CN", "ACMELastSSN","workforceID","L", "employeeStatus","PasswordExpirationTime","LockedByIntruder","ACMEAuthLockout", "Email","manager"]

var serArg1 = domRequest.getElementsByTagName("ser:arg1")[0]; 
for (var i = 0; i < ATTRS.length; i  )
	{
	var serStringText = domRequest.createTextNode(ATTRS[i]);
	var serString = domRequest.createElement("ser:string");
	serString.appendChild(serStringText);
	serArg1.appendChild(serString);
	}

getXMLResponse(form,field,xmlhttp,"/IDMProv/vdx/service",domRequest, OnStateChange);

}

It first calls queryDoc() which returns the shell request document as a string, and passes it in to xmlParse() to turn that into a DOM object. Using my xmlSetText() function, we search the document for specific elements and insert the appropriate parameter values.

I need the query to return a series of attributes within the response. I could have just hard coded that into the source document, but instead I put it into an array and iterate through it to populate the XML. The first thing the code does is to get the element object representing where in the document the attribute should land. Next, we iterate through the array, and create two objects to insert the text, and then an element to support the items of the attribute list.

Once that is done, the XML document is transmitted to the VDX service. Now we need to parse the response. When our OnStateChange is run, it fires an event and passes the response in as the custom data of the event. To convert this into a DOM so it could be parsed, pass it into xmlParse():

var domResponse = xmlParse(event.getCustomData());

There are three main things you may want to do with the resulting DOM:

Selecting an element

<myElem myAttribute="value">text</myElem>

Retrieving text under an element

<myElem myAttribute="value">text</myElem>

Retrieving attribute value.

<myElem myAttribute="value">text</myElem>

• var instances = domResponse.getElementsByTagName("entry")

This method returns an array of element objects representing every element named "entry" in the document. You can count how many entries are returned by checking the property instances.length.

• var dn = thisElement.getElementsByTagName("key")[0].textContent;

You can retrieve the text using the textContent property. It is important, however, to make sure you are selecting an element object. In this example, we are appending the index [0] to the end of the method call to getElementsByTagName because this method returns a NodeList (which in javascript is represented as an array). That selects the first element returned by this method.

• var unvalued = (xmlGetAttribute(thisAttr, "xsi:nil") == "1")

Finally, to retrieve an attribute value, I have created a function called xmlGetAttribute(). The reason I use a function here is that it allowed me to encapsulate the error handling for the case when there was no such attribute on the element. In that case it returns a blank string.

Finally, on to the VDX

With this XML background handled, let's move on to using VDX. VDX is a SOAP web service which is available on the IDM User Application server at the relative URL /IDMProv/vdx/service. From within a Provisioning Request Definition ("PRD"), the URL is relative to the User Application and is contained within the UA's security domain, so it can be accessed with the rights based on your logged in context (it can also be called from any application using basic authentication passing a DN and password). Additionally, a utility is provided to create stub bindings to call the VDX from a java application. But I digress.

When I first tried to use the VDX, I foolishly tried to use LDAP query syntax and LDAP attribute names. VDX is the SOAP exposure of the DAL, so the attributes and entities are identified by their DAL key names, and the query language is simpler than LDAP but it is proprietary.

The VDX Query Language

VDX is a simple expression language which supports relational expressions which can be groups together by parenthesis as well as joined logically by AND or OR. The relational operators supported include:

> , < , >= , <= , = , != , !< , !> , !<= , !>= , STARTWITH, !STARTWITH, IN , !IN , PRESENT, !PRESENT

Attributes must be represented by their DAL key, and literals must be quoted. For example, to query for all users whose first name starts with J, last name is Smith or user name contains "guest", you might put a query together looking like this:

((FirstName STARTWITH 'J') AND (LastName = 'Smith')) OR (CN IN 'guest')

The SOAP Request for the VDX query

The following is a sample SOAP request for a VDX query to return the manager, PasswordExpirationTime, FirstName and LastName of all users whose first name starts with an M and last name starts with a C.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://www.novell.com/vdx/service">
   <soapenv:Header/>
   <soapenv:Body>
      <ser:queryRequest>
         <!--Optional:-->
         <ser:arg0>user</ser:arg0>
         <!--Optional:-->
         <ser:arg1>
            <!--Zero or more repetitions:-->
            <ser:string>manager</ser:string>
            <ser:string>PasswordExpirationTime</ser:string>
            <ser:string>LastName</ser:string>
            <ser:string>FirstName</ser:string>
         </ser:arg1>
         <!--Optional:-->
         <ser:arg2>(FirstName STARTWITH 'M') AND (LastName STARTWITH 'C') </ser:arg2>
      </ser:queryRequest>
   </soapenv:Body>
</soapenv:Envelope

A sample SOAP Response for such a VDX query

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <SOAP-ENV:Body>
      <ns1:queryResponse xmlns="http://www.novell.com/vdx/service" xmlns:ns1="http://www.novell.com/vdx/service">
         <result>
            <entries>
               <entry>
                  <key>cn=fcgd4675,ou=users,o=acme</key>
                  <values>
                     <attribute>
                        <binaries/>
                        <booleans/>
                        <dates/>
                        <integers/>
                        <strings>
                           <string>cn=blyc3668,ou=users,o=acme</string>
                        </strings>
                        <type>DN</type>
                     </attribute>
                     <attribute>
                        <binaries/>
                        <booleans/>
                        <dates>
                           <datetime>2016-09-09T15:10:36</datetime>
                        </dates>
                        <integers/>
                        <strings/>
                        <type>Time</type>
                     </attribute>
                     <attribute>
                        <binaries/>
                        <booleans/>
                        <dates/>
                        <integers/>
                        <strings>
                           <string>Cencer</string>
                        </strings>
                        <type>String</type>
                     </attribute>
                     <attribute>
                        <binaries/>
                        <booleans/>
                        <dates/>
                        <integers/>
                        <strings>
                           <string>Marla</string>
                        </strings>
                        <type>String</type>
                     </attribute>
                  </values>
               </entry>
               <entry>
                  <key>cn=dhgb2806,ou=users,o=acme</key>
                  <values>
                     <attribute xsi:nil="1"/>
                     <attribute>
                        <binaries/>
                        <booleans/>
                        <dates>
                           <datetime>2016-09-09T14:42:17</datetime>
                        </dates>
                        <integers/>
                        <strings/>
                        <type>Time</type>
                     </attribute>
                     <attribute>
                        <binaries/>
                        <booleans/>
                        <dates/>
                        <integers/>
                        <strings>
                           <string>Castro</string>
                        </strings>
                        <type>String</type>
                     </attribute>
                     <attribute>
                        <binaries/>
                        <booleans/>
                        <dates/>
                        <integers/>
                        <strings>
                           <string>Marcos</string>
                        </strings>
                        <type>String</type>
                     </attribute>
                  </values>
               </entry>
            </entries>
         </result>
      </ns1:queryResponse>
   </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

My onreceive event handler was used to leverage DOM to parse the contents of the XML document and produce a multi-dimensional array in a global variable. Once this is populated, it fires another event, which evaluates the global array containing the query result to determine which of three courses of action to take.

No objects returned

Display a "not found" dialog

One object returned

Select that item and populate the form

Multiple objects returned

Fire an event to signal an HTML control to render the array as a table

Since this was intended to replace the advanced search within the portlet, I simply selected to view the source of the resulting table of a query and used that XML as the model for how to present the result table in this PRD.

As I iterate through the rows and columns of the form, I build a conventional HTML table using the table, tr and td elements. Within the HTML, I put in calls on each cell to call a form API to generate an event on click with the DN of the object which is being represented. This allows the user to click anywhere on the form data and select the object they wish to edit.

The event fired receives the DN to be sent as custom data. This is received by the "recipient" field, setting its value to the DN. This in turn triggers the "recipient-changed" event, causing every control to populate with the appropriate value from that object.

Conclusion

The ability to make a VDX query within a workflow form enables direct access to a powerful facility, which, among other things, can be used to create highly targeted administrative solutions, closely aligning business and technical requirements.