With Novell Identity Manager 4.0 there are a number of new features available. You can read more about those features in these articles:

• What has changed between Identity Manager versions - Part 1

• What has changed between Identity Manager versions - Part 2

• What has changed between Identity Manager versions - Part 3

• More on what's new in IDM 4

There are four new drivers, two are for new connected system supported (Salesforce.com and Sharepoint) and two are used as service drivers that are needed for the Reporting module.

These two drivers are the Managed System Gateway (MSG) driver, and Data Collection Service (DCS) driver, the first of which you can read about in this series of articles:

• Trying to understand the Managed System Gateway driver in IDM 4 - Part 1

• Trying to understand the Managed System Gateway driver in IDM 4 - Part 2

• Trying to understand the Managed System Gateway driver in IDM 4 - Part 3

• Trying to understand the Managed System Gateway driver in IDM 4 - Part 4

• Trying to understand the Managed System Gateway driver in IDM 4 - Part 5

• Trying to understand the Managed System Gateway driver in IDM 4 - Part 6

The Data Collection Service driver, is the second half of that pairing, and will be the topic of this series. Both these drivers are meant to enable the Reporting module to get enough information about the system to report upon it. The MSG driver is focused more on providing information about how the drivers are configured, heck it even tries to infer the matching rule criteria by reading the rules out of the objects, and the DCS driver is focused more on collecting events about objects for storage in the Reporting database.

• 
  
  • In the first article Data Collection Service Driver Walkthrough - Part 1 I started looking at how it builds a cache variable and got through most of the work it does to get the correct IP Address of the server running the Managed System Gateway driver.
  
  
• In the second article Data Collection Service Driver Walkthrough - Part 2 I finished working through how the cache is built.

• In the third article Data Collection Service Driver Walkthrough - Part 3 I look at how queries out of the cache are handled, and what the filter for this driver sends to the shim.

• In the fourth article Data Collection Service Driver Walkthrough - Part 4 I finished up one last rule in the Input Transform that handles some error cases and started discussing the Subscriber channel.

• In the fifth article Data Collection Service Driver Walkthrough - Part 5 I worked through more of the Subscriber Event Transformation policies, finishing the Event Transformation and getting through the Creation Policy set.

• In the sixth article Data Collection Service Driver Walkthrough - Part 6

I worked through the Subscriber Command Transform Policy set, making it all the way into the Output Transform policies.

In this article I will work through the rest of the Output Transformation policy set.

We have three rules left in the Output Transformation. A rule to reformat the member query, and then two more to handle format conversions, where one is done in XSLT.

First up is the policy object NOVLIDMDCSB-otp-ReformatMemberQuery. The single condition test is if operational attribute memberQueryURL is changing. This is mapped to eDirectory memberQuery, which is used to define a dynamic group. The older Roles Based Entitlements driver used this attribute to store the search filter that defined the dynamic group to define who deserved a particular entitlement. So if the filter was something like (&(objectClass=User)(acmeADFlag=true)) then all users who had acmeADFlag set to true were returned as results of the query, were thus members of the dynamic group, and had the entitlement added by the RBE driver. (It added the attribute DirXML-EntitlementRef which is a path syntax attribute, where the nameSpace component, a 32 bit integer is usually 0 or 1 for revoked or granted, the volume component is a DN of the Entitlement being referenced, and the path component which interestingly shows up as path.xml is an XML document with some information about what agent granted it, and some time stamps).

I am not entirely sure what else the Reporting module is going to do with a dynamic group, but I imagine it would be useful to know who should be members, based on the query filter, so a report could be generated to basically implement that filter against the data in the database instead.

At a high level, what this policy does seems odd. It removes the memberQueryURL attribute from the event document with a strip operational attribute token. Then it builds up the infrastructure needed to build an LDAP query via the LDAP query ECMA functions included in the lib-AJC ECMA object.

It gets the DN of the object in the event document, and queries via LDAP for the filter attribute. Finally if it gets a value, it adds it back to the event document.

All that is clear and obvious from what the code is doing, but here we have the perfect example of needing an explanation for WHY is the driver doing this? What is wrong with the memberQueryURL data that makes it out of the vault and through all the policies to here, that it needs to be retrieved again, specifically via LDAP as opposed to using a Query token to read it more trivially out of eDirectory.

Without an example in trace to see, and lacking comments in the driver, my guess is that the formatting is somehow mangled when IDM handles the event. I have nothing to base that on, except that they went to a fairly complex approach to getting the value back, when there is a single token (Source Attribute) that would have done exactly the same thing for them. Its 170 lines of XML vs 3 lines of XML if they used the Source Attribute token.

So how do they do the actual dirty work? Well this is a nice example of how you might use LDAP in a driver of your own, so it is worth dissecting so others can use it themselves later.

First up is a check for a local variable, dcs-user-password since if it is not found, it will be set as a driver scoped variable. Thus if this is the first time through this rule, it gets set if not, use the value in memory already. Set the variable from the Named Password.

If you are familiar with Lothar Haegers Password Expiration Notification driver, you will have seen this all before. He takes a very unique approach of trying to avoid a lot of this work, and look at the driver objects Security Equals attribute, then gets the nspmDistributionPassword of that object, and now we have a DN and a password to make administrative level LDAP queries with.

Here they use the configured password and user from the driver. Both ways work. Lothars is just more elegant.

Next the driver needs to get the IP address of the LDAP server and its port. Now since these are not hard coded values, it is going to derive them from the values in the Network Address of the current servers NCP Server object. Again, if the driver scoped variables for LDAP IP and LDAP port are not available we run through the code to get them. If they are, we move on.

Get the Network Address values into a nodeset variable called ncpServer using the new Global Configuration Variable (GCV) in IDM 4 dirxml.auto.localserverdn, that returns the LDAP formatted DN of the server object the current driver is running on. This needs to be ParseDN'ed back to backslash format for IDM to use in the Query token, which then reads back the Network Address attribute values.

Now the Network Address attribute (Shown in ConsoleOne in the General tab of a NCP Server object) has lots and lots of values these days. There are the NCP address, the LDAP address, HTTP and HTTPS addresses for iMonitor, and so on. Now multiply all that by possibly listening on multiple IP interfaces, and there is quite the mess in this attribute.

The good news is that Network Address's syntax is such that the netAddrType component can be used to at least lightly filter out some of the cruft.

Thus an XPATH is used to set the URLs variable to be a subset of the ncpServer variable. Both as node sets of course. The XPATH used is:
$ncpServer/attr[@attr-name='Network Address']/value[component[@name='netAddrType']/text()=13]/component[@name='netAddr']/text()

This looks at the ncpServer variable, for the <attr attr-name='Network Address'> attribute and under its <value> node it specifically looks for the component named netAddrType, and then only for the ones whose value is 13. This is the value[component[@name='netAddrType']/text()=13] part, that has two predicates in it. First is the [component[@name='netAddrType'] and then wrapped around that is the test for text()=13. Thus any nodeset that matches, is 'copied' via the set local variable call into the new URLs variable. In fact, it also only copies the netAddr component part of the nodeset, as a string, so that we actually get a flattened nodeset of just strings, which makes dealing with it much easier.

Then loop through the URLs variable, since type 13 I think is IP Address based network addresses (as opposed to NCP, AppleTalk, NFS, and other others) so we almost certainly have more than one.

Then decode the string value we see in each loop, with Base64 decode tokens, then use an ECMA functions from the lib-AJC library object, to get the protocol. This calls the function getProtocolFromURL, which is defined as:

/** 
 * Get the protocol from a URL
 *
 * @param urlStr     URL string
 * @return intPort   port as an integer
 */
function getProtocolFromURL(urlStr)
{
	return urlStr.substring(0,urlStr.indexOf(':'));
}

This basically takes something in the format of http://www.google.com:2020/some/url and gets everything before the first colon (:). It substrings from the 0th position to the position of the colon.

This could trivially be done in XPATH or in policy, so I am not sure why they bothered with a reasonably unsubtle ECMA to do it. Probably because it was already available. The Lib-AJC is the Advanced Java Class library that Novell Consulting had been using for years, ported to run in ECMA, so that they could include it in a project as a policy object, (Well DirXML-Resource object with DirXML-ContentType set to be an ECMA resource) instead of distributing and versioning a JAR file.

Now to find the LDAP or LDAPS protocol instance, since its possible to have LDAP listen on a different IP from the NCP or HTTP listening addresses. There are some machinations to ensure we get the LDAP over SSL preferentially, and not one of the two common local host addresses, after which we set the protocol, IP, and Port into a set of driver scoped variables, using the ECMA functions getPortFromURL() and getIPFromURL() which I located and show below:

/** 
 * Get the port from a URL
 *
 * @param urlStr     URL string
 * @return intPort   port as an integer
 */
function getPortFromURL(urlStr)
{
	port = "";
	temp1 = urlStr.substring(urlStr.lastIndexOf(':') 1);
	if (temp1.indexOf('/') > 0)
	{
		port = temp1.substring(0, temp1.indexOf('/'));
	}
	else
	{
		port = temp1;
	}
	lastChar = java.lang.Character.getNumericValue(port.substring(port.length-1)) 
	if (lastChar > 9 || lastChar < 0)
	{
		port = port.substring(0, port.length-1);
	}
	return port;
}

/** 
 * Get the IP from a URL
 *
 * @param urlStr     URL string
 * @return dottedIP  IP in common dotted string format
 */
function getIPFromURL(urlStr)
{
	var ip = "";
	var temp1 = urlStr.substring(urlStr.lastIndexOf("://") 3);
	if (temp1.indexOf(":") > 0)
	{
		ip = temp1.substring(0, temp1.indexOf(":"));
	}
	else if (temp1.indexOf("/") > 0)
	{
		ip = temp1.substring(0, temp1.indexOf("/"));
	}
	else
	{
		ip = temp1;
	}
	return ip;
}

I am not sure these are the most subtle ways to pick apart a URL as a general solution, however, since the possible values are limited as they all come from the eDirectory code base which sets the values on the NCP Server object I am sure they suffice.

Now that we have LDAP configuration information, lets set the DN of our search base in the variable 'base' by using the qualified-src-dn XML attribute in the event node. Use ParseDN to get it into LDAP format.

Now we can make the LDAP search call via ECMA. In case you were wondering how it works, this should be a nice example:

es:ldapSearch($dcs-driver-ldap-ip, $dcs-driver-ldap-port, '~dcs-user-dn~', $dcs-user-password, $base, 'base', 'objectClass=*', 'memberQueryURL')

es:ldapSearch is how you call an ECMA function from a linked in ECMA Resource library. es:functionName.

The parameters it takes, are then IP, port, bindDN, bind Password, searchBase, scope value (sub, one, base), then a filter and finally what attributes to return.

In this case we have the DN of the object so do a base level query and return the memberQueryURL. This gets set into a nodeset local variable dynGroups, and then the XPATH to get the value we want out is:

$dynGroup/attr[@attr-name='memberQueryURL']/value/text()

That is, find the <attr attr-name='memberQueryURL'> node, and get the value out of it.

There is some final checking to be sure we actually got a value and not a null, which is nice to see. If there is no value returned, then clear the destination address, otherwise send it into the connected system.

Next up is the policy object NOVLIDMDCSB-otp-FormatConversions which basically flattens the Fax number attribute, which in eDirectory is a nicely structured attribute with bit rate and features components, but alas, no one but eDirectory cares about that info anymore. So a simple reformat operational attribute handles it for us, and sets the value in the document to just the value of the component whose name XML attribute is faxNumber.

The XPATH is: $current-value/component[@name='faxNumber'] that does that selection for us.

The last object in this driver is actually a XSLT Stylesheet and not a Policy object. It is named, NOVLIDMDCSB-ots-FormatConversions (note that it is -ots- in the name for Output Transform with an S for stylesheet, instead of -otp- which uses a P for policy.

A couple of things are done here. First the timestamp XML attribute is reformatted from CTIME to more LDAP'y time formatting of yyyyMMddHHmmss followed by a capital Z for Zulu time zone. I am quite certain we ought to be able to do this in Policy, easier even, since we have the Convert Time token available, but this works fine as well.

	<xsl:template match="@timestamp">
		<xsl:attribute name="timestamp">
			<xsl:value-of select="concat(es:getTime(substring-before(., '#'), 'yyyyMMddHHmmss'), 'Z')"/>
		</xsl:attribute>
	</xsl:template>

Then we have a touch up of the cached-time XML attribute, to change from yyyyMMddHHmmss.000 or the like to yyyyMMddHHmmssZ to indicate the time zone. (Z for Zulu/UTC instead of .000 for zeroth time zone).

	<!-- re-format @cached-time -->
	<xsl:template match="@cached-time">
		<xsl:attribute name="cached-time">
			<xsl:value-of select="concat(substring-before(., '.'), 'Z')"/>
		</xsl:attribute>
	</xsl:template>

There is a template to change all DN attributes from backslash format to LDAP format as shown here:

	<!-- re-format dn values -->
	<xsl:template match="value[@type='dn']|component[@name='volume']|component[@name='dn']">
		<xsl:copy>
			<xsl:apply-templates select="@*"/>
			<xsl:value-of select="es:lowerString(dncv:convert($dnConverter, ., 'qualified-slash', 'ldap'))"/>
		</xsl:copy>
	</xsl:template>

This finds any attribute that is likely to be a DN. type='dn' is the common for a standard attribute. component[@name='volume'] catches Path syntax attributes. I am not sure offhand which syntax type uses component[@name='dn'] but I am sure there is one.

The problem I see with this, is that the call to DNConverter, which is the class that ParseDN calls under the covers, is that the values are not coming in as qualified-slash, I am pretty sure they are coming to this rule in simple backslash format. I really need to get some more trace samples to be certain.

Finally there is a template to fix up any attribute of syntax time from CTIME to the more LDAP'y format time.

	<xsl:template match="value[@type='time']">
		<xsl:copy>
			<xsl:apply-templates select="@*"/>
			<xsl:value-of select="concat(es:getTime(text(), 'yyyyMMddHHmmss'), 'Z')"/>
		</xsl:copy>
	</xsl:template>

That just about wraps up the driver policies that are found in the driver configuration. I certainly learned some interesting things doing this.

I clearly need to see more of the inner workings of the Reporting module to better understand how this all fits together, but it is pretty clear that the basic data for tracking events in the Identity Vault is coming through this driver. I think it is the other sources of data into the Identity Warehouse that enrich it that will make this more understandable.