In Identity Manager 4.0 Novell has introduced a number of new features. There are four new driver configurations, two for applications (Salesforce.com and Sharepoint) and two for IDM itself to use, the Managed System Gateway driver and the Data Collection Service driver.

The Managed System Gateway driver is primarily used by the Reporting module to get information about users out of IDM and into the Reporting database. This is somewhat analogous to how the Identity Audit extension policies that were added to drivers are used to get Identity information into the Sentinel database.

As with many things in IDM 4, this is totally new stuff, and will take some time to get used too. You can read more about the changes between the various IDM versions in these articles:

• What has changed between Identity Manager versions - Part 1

• What has changed between Identity Manager versions - Part 2

• What has changed between Identity Manager versions - Part 3

• More on what's new in IDM 4

One of the main new features is Packages, which is critical for all this working, as there are packages that add support for the reporting module to each of the many driver configurations. In fact the same approach is used for the Identity audit extensions as well. This is different than in the past, where the policies were stored in Libraries, centrally and linked into each driver as needed. Now with packages the content is actually duplicated in many places, but with Packages, upgrades are made easier than the previous model. I have been working on a series on Packages in Designer 4, that you can use to gain some insight:

• Let's talk some more about Packages in Designer 4 - Part 1

• Let's talk some more about Packages in Designer 4 - Part 2

• Let's talk some more about Packages in Designer 4 - Part 3

• Let's talk some more about Packages in Designer 4 - Part 4

• Let's talk some more about Packages in Designer 4 - Part 5

• Let's talk some more about Packages in Designer 4 - Part 6

• Let's talk some more about Packages in Designer 4 - Part 7

• Let's talk some more about Packages in Designer 4 - Part 8

• Let's talk some more about Packages in Designer 4 - Part 9

The Managed System Gateway (MSG) driver is one interesting critter. It is doing all sorts of funky and interesting things that it is worth discussing the low level functionality. After all, if you do not know what it is supposed to be doing, how would you know what it is not doing, when it is not working. Most connected system drivers are pretty traditional, that is an event comes out of the application of eDirectory, as an XDS document (which is what the shims job is, convert the applications event into XDS and convert XDS into things the application understands) which is then processed in the flow.

• In the first article Trying to understand the Managed System Gateway driver in IDM 4 - Part 1 of this series I started looking at how it builds up a cache of various interesting things. It stores them as driver scoped local variables. That means they are available while the driver is running. On every restart they need to be recalculated. There is the SERVER_INTERFACE, and DRIVER_DN variables that we talked about. These are in the Input Transform, in a series of policy objects.

• In the second article Trying to understand the Managed System Gateway driver in IDM 4 - Part 2 I started looking at one of the most audacious things I have ever seen a driver policy try to do! Read back another drivers DirXML Script policy, and figure out what it is doing! I was deeply astonished when I realized what it was trying to do! In fact my first reaction was, no way are they that crazy. Well apparently they are. I got through how they detect that a User is being managed, and how they detect that attribute based matching is happening.

• In the third article Trying to understand the Managed System Gateway driver in IDM 4 - Part 3 I looked at how the Matching rule is read to infer what attributes of the User was being matched upon. A very interesting and audacious approach!

• In the fourth article Trying to understand the Managed System Gateway driver in IDM 4 - Part 4 I discussed some short comings of that approach, and moved on to how the queries are handled from the various cache variables.

In this article I will look at some more examples of queries that the driver performs out of the cache.

The driver has four different cache variables set up, and it supports queries out of all of them. At some level this is kind of boring, but at another it is useful to have written down, I think. The final query it supports is more interesting so stick with me as I get there.

In the previous article I looked at the SERVER_INTERFACE API query, and that returns the connection information for a system. Next up is a query into for the MANAGED_SYSTEM_INFORMATION. This is built in the cache from the MSysInfo Global Configuration Variable object in each driver. If your driver supports the Managed System Gateway (MSG) driver then it includes a package that adds at least the GCV object named MSysInfo. To build this cache, this driver queries for all Driver objects in this driver set, that have a MSysInfo GCV object directly under the driver node. This is then parsed, as discussed in the first article in this series, using an amusing XPATH that looks for all nodes have a name that starts with a string. I.e. Get me all GCV's named ms.info.* from that object. Which I never really thought of doing before. Neat idea, just requires that you name things with rigor.

Thus we see the query come in on the Publisher channel from the driver shim as the following XDS document:

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product build="4.0.0" instance="Managed System Gateway Driver" version="4.0.0">Identity Manager Managed System Gateway Driver</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query class-name="MANAGED_SYSTEM_INFORMATION" scope="subtree">
      <search-class class-name="MANAGED_SYSTEM_INFORMATION"/>
      <read-attr/>
    </query>
  </input>
</nds>

The Publisher Event Transformation returns all the instances in the query, since there is no specific GUID that the query is looking for. In our example trace, we only have a single <instance> in the cache since only one driver is enabled for this support.

The rule adds the cache data into the <operation-data> node as below:

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product build="4.0.0" instance="Managed System Gateway Driver" version="4.0.0">Identity Manager Managed System Gateway Driver</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query class-name="MANAGED_SYSTEM_INFORMATION" scope="subtree">
      <search-class class-name="MANAGED_SYSTEM_INFORMATION"/>
      <read-attr/>
      <operation-data api-name="MANAGED_SYSTEM_INFORMATION">
        <instance class-name="MANAGED_SYSTEM_INFORMATION" logical-instance="false" src-dn="cn=acme domain,cn=driverset1,o=system">
          <attr attr-name="msysinfo.dyn.ms.driverGuid">
            <value type="string">B097B0E7-A932-2540-63A7-B097B0E7A932</value>
          </attr>
          <attr attr-name="msysinfo.dyn.ms.hierarchical">
            <value type="state">true</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.name">
            <value type="string">ACME Domain Server</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.description">
            <value type="string">ACME Domain Server</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.location">
            <value type="string">Chicago</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.vendor">
            <value type="string">Microsoft</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.version">
            <value type="string">2003</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.businessOwner">
            <value type="dn">cn=cnano,ou=users,o=data</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.applicationOwner">
            <value type="dn">cn=tmellon,ou=users,o=data</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.classification">
            <value type="string">Vital</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.environment">
            <value type="string">Production</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.auto.generated.info.show">
            <value type="string">false</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.id">
            <value type="string">B097B0E7-A932-2540-63A7-B097B0E7A932</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.type">
            <value type="string">AD</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.auth.ip">
            <value type="string">app.idm.com</value>
          </attr>
          <attr attr-name="msysinfo.drv.ms.auth.port">
             <value type="string"/>
          </attr>
          <attr attr-name="msysinfo.drv.ms.auth.id">
            <value type="string">Administrator</value>
          </attr>
        </instance>
      </operation-data>
    </query>
  </input>
</nds>

The document gets submitted to eDirectory, but still does not show up in trace, regardless, the Output Transform rule gets it, sees that it is an api example and fixes it up to look like a proper response document.

This happens in the NOVLIDMMSGWB-otp-QueryResponse rule, which is quite simple. It looks for the presence of the operation property api-name, which you can see in the <operation-data api-name="MANAGED_SYSTEM_INFORMATION"> node. Then it uses Clone by XPATH with this XPATH statement as the source:

./operation-data/instance[@class-name=../@api-name]

This means, in the current document (the period (.)) in the <operation-data> node, select the <instance> node (and of course implicitly all its children) that has a class-name XML attribute, the same as the api-name. in the parent node (i.e. The Query doc). This is a nice touch, as it will only copy the requested results in as the class-name in the query needs to match the api-name in the operation-data.

The target for the clone by XPATH is simply dot dot (..), which is the parent of operation-data.

Thus we end up with a response document back to the driver shim that you might see coming out of any driver in the palette.

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="4.0.0">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <instance class-name="MANAGED_SYSTEM_INFORMATION" logical-instance="false" src-dn="cn=acme domain,cn=driverset1,o=system">
      <attr attr-name="GUID">
        <value type="string">B097B0E7-A932-2540-63A7-B097B0E7A932</value>
      </attr>
      <attr attr-name="Hierarchical">
        <value type="state">true</value>
      </attr>
      <attr attr-name="Name">
        <value type="string">ACME Domain Server</value>
      </attr>
      <attr attr-name="Description">
        <value type="string">ACME Domain Server</value>
      </attr>
      <attr attr-name="Location">
        <value type="string">Chicago</value>
      </attr>
      <attr attr-name="Vendor">
        <value type="string">Microsoft</value>
      </attr>
      <attr attr-name="Version">
        <value type="string">2003</value>
      </attr>
      <attr attr-name="BusinessOwner">
        <value type="dn">cn=cnano,ou=users,o=data</value>
      </attr>
      <attr attr-name="ApplicationOwner">
        <value type="dn">cn=tmellon,ou=users,o=data</value>
      </attr>
      <attr attr-name="Classification">
        <value type="string">Vital</value>
      </attr>
      <attr attr-name="Environment">
        <value type="string">Production</value>
      </attr>
      <attr attr-name="msysinfo.drv.ms.auto.generated.info.show">
        <value type="string">false</value>
      </attr>
      <attr attr-name="ID">
        <value type="string">B097B0E7-A932-2540-63A7-B097B0E7A932</value>
      </attr>
      <attr attr-name="Type">
        <value type="string">AD</value>
      </attr>
      <attr attr-name="AuthenticationIPAddress">
        <value type="string">app.idm.com</value>
      </attr>
      <attr attr-name="AuthenticationPort">
        <value type="string"/>
      </attr>
      <attr attr-name="AuthenticationID">
        <value type="string">Administrator</value>
      </attr>
    </instance>
  </output>
</nds>

What is kind of nice is that the driver also adds comments into the trace, when it does this, to make reading it a bit easier. For example, after it fixes up the queries we just talked about, it traces out a nice message that uses this XPATH:

count(./operation-data/instance[@class-name=../@api-name])

to tell how many results were returned. This is the same XPATH as used in the Clone By XPATH, but this time, it is wrapped inside a count() function call. Count() is a cool XPATH function. It counts nodes that are at the specified level. Thus this will return the number of <instance> nodes that are for this event. So you would know that 1, 2 or say 5 results were returned, in the trace. Classy touch. It can also be used as a true or false test, since if you query for something, and it is not there, the count() of the results is zero. There are a number of really neat XPATH tricks I have seen using count() that look like something like magic, when you read them.

There are some additional query docs that can be sent by the driver. For example, it can get information back about a specific driver in the following exchange of documents reading out of the DRVR_CACHE variable.

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product build="4.0.0" instance="Managed System Gateway Driver" version="4.0.0">Identity Manager Managed System Gateway Driver</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query class-name="DRIVER_DN" scope="subtree">
      <search-class class-name="DRIVER_DN"/>
      <association>B097B0E7-A932-2540-63A7-B097B0E7A932</association>
      <read-attr/>
    </query>
  </input>
</nds>

The cache data is selected, and added to the operation-data node.

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product build="4.0.0" instance="Managed System Gateway Driver" version="4.0.0">Identity Manager Managed System Gateway Driver</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query class-name="DRIVER_DN" scope="subtree">
      <search-class class-name="DRIVER_DN"/>
      <read-attr/>
      <operation-data api-name="DRIVER_DN">
        <instance class-name="DRIVER_DN" src-dn="o=system\cn=driverset1\cn=acme domain">
          <association>B097B0E7-A932-2540-63A7-B097B0E7A932</association>
        </instance>
      </operation-data>
    </query>
  </input>
</nds>

Then the output transform turns it back into a normal looking result document.

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="4.0.0">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <instance class-name="DRIVER_DN" src-dn="o=system\cn=driverset1\cn=acme domain">
      <association>B097B0E7-A932-2540-63A7-B097B0E7A932</association>
    </instance>
  </output>
</nds>

There is a query for MANAGED_SYSTEM_MATCHING that reads out of the RULE_CACHE variable with a query like this one, which includes the driver GUID:

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product build="4.0.0" instance="Managed System Gateway Driver" version="4.0.0">Identity Manager Managed System Gateway Driver</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query class-name="MANAGED_SYSTEM_MATCHING" scope="subtree">
      <search-class class-name="MANAGED_SYSTEM_MATCHING"/>
      <association>B097B0E7-A932-2540-63A7-B097B0E7A932</association>
      <read-attr/>
    </query>
  </input>
</nds>

The cache data is selected from the local variable via XPATH, and added to the operation-data node.

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product build="4.0.0" instance="Managed System Gateway Driver" version="4.0.0">Identity Manager Managed System Gateway Driver</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query class-name="MANAGED_SYSTEM_MATCHING" scope="subtree">
      <search-class class-name="MANAGED_SYSTEM_MATCHING"/>
      <read-attr/>
      <operation-data api-name="MANAGED_SYSTEM_MATCHING">
        <instance class-name="MANAGED_SYSTEM_MATCHING" src-dn="cn=msysinfo,cn=acme domain,cn=driverset1,o=system">
	<association>B097B0E7-A932-2540-63A7-B097B0E7A932</association>
	<attr attr-name="rule">
            <value type="structured">
              <component name="CN"/>
            </value>
            <value type="structured">
              <component name="CN"/>
            </value>
            <value type="structured">
              <component name="FULL_NAME"/>
            </value>
          </attr>
        </instance>
      </operation-data>
    </query>
  </input>
</nds>

Then the output transform converts the document to look like a regular response document.

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="4.0.0">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <instance class-name="MANAGED_SYSTEM_MATCHING" src-dn="cn=msysinfo,cn=acme domain,cn=driverset1,o=system">
	<association>B097B0E7-A932-2540-63A7-B097B0E7A932</association>
<attr attr-name="rule">
        <value type="structured">
          <component name="CN"/>
        </value>
        <value type="structured">
          <component name="CN"/>
        </value>
        <value type="structured">
          <component name="FULL_NAME"/>
        </value>
      </attr>
    </instance>
  </output>
</nds>

So those last two were kind of boring, but I still think worth including for the Google cache, so someone searching on an error with any of this code in it, might find the article.

One other neat thing that this driver does is that unlike other drivers, where a change to a GCV, a filter, a policy change, and other things requires a driver restart to take affect, this driver is monitoring other drivers, and building up a cache of data based on the current driver configurations.

But what happens if one of those drivers is changed, and this driver is not restarted? It would not go out and rebuild the cached data, and slowly fall out of sync with the world it is living in. This is exacerbated because this driver relies on the configuration of other drivers that are changing. Well the good news is that there is policy in the Subscriber channel, in the Event Transform policy set that is dedicated to detecting changes in other drivers, and if appropriate dirtying the cache.

This set of policies look for events from DirXML-Driver objects. This is sort of funny, since usually in Identity Manager, it is a driver looking for events upon users, groups, or other objects, but in this case, the driver is looking for changes to other drivers themselves.

Previously I had actually had a good reason to do this in a project. In the Provisioning side of Identity Manager there was prior to IDM 4, no easy way to get access to GCV's defined in the driver or driver set. In IDM 4, one of the enhancements is the ability of a GCV() function, that lets a Provisioning Request Definition reference a GCV from the Policy side of IDM. This is great, since it is much better defining variables and values globally, instead of having to redefine it. More importantly, when these are values that change between the development and production systems, it is a royal pain to have to change all hard coded references to them. The other new feature in IDM 4 is that ECMA resource objects, that when linked into a driver make the ECMA functions available, are now also available on the Provisioning side, another great change in IDM 4. It seems simple, but making it possible to share ECMA functions makes many things easier.

Back in the Identity Manager 3.5.1 days I worked on a project with a lot of GCV data and a lot of provisioning work. It was just as easy to implement a driver that watched for all other drivers GCV's changing, and to then record the data in a format more easily recovered in the directory so that it was easy to query on the Provisioning side. The problem was that GCV data is stored as an XML string, and it is quite hard to parse out in Provisioning. By having a driver translate it into sets of attribute pairs it makes life easier to recover. However, with the newer GCV types, like a structured GCV this model gets harder and harder to manage. So I am very glad for the additional functionality in IDM 4.

This driver takes it to another level, since it is handling more than just GCV values, and needs to watch for changes to all sorts of objects.

When there is a change on an NCP Server object, which the filter only lets through the Network Address attribute, then the Server cache variable needs to be dirtied, since this is tracking the network address to be able to know how to contact the server.

When there is a change on the DirXML-Driver, the filter lets through the DirXML-ApplicationSchema and DirXML-Policies attribute. I am not sure what is being done with DirXML-ApplicationSchema, but this is where the connected system schema, retrieved via the GetSchema() call that drivers are expected to implement is stored. You can read about an approach to using this data to manage single valued to multi-valued attribute management in the Active Directory driver in this article:
Generic Single-valued Schema Enforcement


Alas, there is not enough information in that attribute to make me happy, as I would like to see sizing info (max length, min length, etc), and not all drivers populate it, and at differing levels. Of course that makes sense, since some of the data does not make sense in all cases of connected systems.

However the DirXML-Policies node we know from the Rule Cache is used, to infer the matching rules in use by the various drivers. Now in the old linkage model there was a discrete attribute for the Matching rule, but of course you would have to walk the DirXML-NextTransformation attributes to be sure you had the entire chain of policies. In the current model, any change to the DirXML-Policies attribute on a driver means it should be completely recalculated.

Next up is the Managed System cache which is watching for changes in a DirXML-GlobalConfigDef object, the new GCV holding object from IDM 4. With the move to this new GCV module, it is not enough to look for changes on the DirXML-Driver for a change in the DirXML-ConfigValues attribute, now we have to look for any changes to a DirXML-GlobalConfigDef object, and read backs its DirXML-ConfigValues attribute. Not that much harder, but slightly different.

The Rule cache is dirtied, in a much more common case, alas, where any DirXML-Policies on any driver changes, as discussed before, and if any DirXML-Rule object changes, we cannot know in advance nor easily if it is a Matching rule or otherwise. Thus it is simpler to just dirty the cache and recalculate the cache values.

As you can see this really is a fun driver to work through!