Changes that do not require a driver restart, and errors on starting an Identity Manager driver

Novell Identity Manager uses several components that all need to be working together for the functionality to be available.

There is the engine, that needs to be on a working eDirectory instance, since it runs as a Java application inside the eDirectory application space. (The major consequence is that eDirectory and Identity Manager share the same memory pool, which on a 32 bit operating system is usually 2 giga bytes, so be careful to not over-allocate either component).

There are the driver shims that are either running locally or on a Remote Loader instance, that can be Java based (JDBC, LDAP, etc) or native code (Active Directory's addriver.dll).

Finally there are the rules themselves, that define the driver configuration. These are stored as objects in eDirectory. There is an object for each driver set (DirXML-DriverSet object class) and under that would be each Driver included in the DriverSet (DirXML-Driver object class).

Inside a driver object there are a number of rules, things like the Schema Map, Filter, Input Transform, Output Transform, Mapping tables, Job definitions, and more. There are two containers, (CN= actually, not OU= as we are used to in eDirectory. Same for Driver Sets and Driver objects) Subscriber and Publisher (DirXML-Publisher object class) that contains the Rule objects for each channel.

Inside a Rule object (DirXML-Rule object class), there is an attribute called XmlData that holds the actual XML of the rule. It is of syntax Stream, so the size can get about as big as it can ever need to be. (Stream files are stored as external files from the database DIB itself and probably are limited by the maximum size of a file in that file system. If you look in your DIB directory (sys:_netware for Netware, for Windows and Unix installs you would have to look where you specified it).

When a change is made, the XmlData contents are updated, either live by iManager, or offline by Designer (the change is written to eDirectory when you do a Compare or Deploy of the driver), and do not take affect until the driver is reloaded.

ECMAScript objects are much the same.

Some changes take affect without a driver restart.

Changes to the trace settings of a Driver or Driver Set object take affect live. This includes things like trace level, trace file name, trace file size. You can watch this for example if you have a driver on a system where you can "tail -f" the trace file. Tail is a Unix utility that shows the last few lines of a file, and with the -f switch show any changes that are appended to the file. Of course it is meant for simple text files, which dstrace output to a file is a great example of.

If you run "tail -f /path/to/the/tracefile" on a normally busy driver you will see the normal flow of events being processed according to the trace level set. Level 3 is the traditional basic debugging level that shows most things needed (rules processing and whatnot) to see what happened and what went wrong or right.

While you have that running in a terminal session flashing changes, bring up an iManager instance in a web browser, get to the Properties page of the Driver or Driver Set object, then onto the Misc tab. Finally watch what happens when you change the 3 to a 0, and click apply.

This is a nice feature, because if something seems to be going on or wrong with a driver, usually you want to see what is happening to try and diagnose the problem (catching it in the act, so to speak), but restarting the driver in the middle for such a change to take affect would likely interrupt the problem at hand. (Possibly resolve the issue and possibly harm, so best to see what is going on first, before making any changes).

Mapping tables, if updated, do not require a driver restart. Mostly because they are read live each time. I am not sure if they are cached at all to boost performance but they seem to perform nicely with the current design.

Email templates and email template collections should be read each time they are called, so that changes take affect the next time they are used, and should not require a driver restart.

I am curious if people have run across more examples of things that you can change without requiring a driver restart to take affect! Please comment on this article so we can have it all together in one place, instead of scattered throughout the documentation.

Having discussed that aspect of changes you can make to a driver, there is a flip side to this discussion in that there are a number of things you can do just in driver rules and configuration settings, that will cause a driver to fail to start up.

Probably the most common I have seen is an overzealous Veto command. There is a series of articles about the indiscriminate use of the Veto command, and the end result is that the documents the driver shim needs to exchange with the engine in order to complete the
startup process get vetoed.

Usually the administrator wants to block events in one direction and adds a rule that does a Veto without sufficient constraints. I usually recommend specifying a class-name equals User test, or operation equals add type test, rather than something "IF Source DN not in subtree ACME\Users then Veto". In that case, the startup documents will not complete as they do not have a src-dn attribute with the correct information to get past the rule. If you added a second condition, AND class-name=User then the driver startup documents would not get blocked.

This is a harder one to troubleshoot, since it requires that you recognize what a proper driver startup should look like, watch the broken driver startup, and notice that it is shorter and incomplete. Usually one of the last events is a do-veto of an operation, but it is possible that it gets buried in trace and not be obvious. If it is the last event in the trace, it is usually quite obvious, but it is not always the case.

Another pretty common occurance is when the driver was installed but not licensed, and then the ninety day evaluation period has expired. In that case, you get a nice error message in the trace telling you that the evaluation period has expired, so the driver is stopping.

The next most common startup issue I seem to run into is a bad XPATH statement. XPATH expressions are validated somewhat at driver startup. I say somewhat because it is still possible to submit useless or incorrect XPATH statements. The engine just catches some classes of syntax errors. I would love to see what exactly it is validating, but I have not been able to find that information yet. Usually the error is that the driver is expecting a node test and instead gets something else. For more information on the difference between a node test (used to select) and other XPATH functions, check out this article:
Some thoughts on XPATH in Novell Identity Manager

A call to an incorrect or invalid function can lead to the driver throwing an error and failing to startup. The good news is the error will identify the Rule object and the line within the XmlData attribute that the error occurred in. To get the driver up and running fastest, disable the entire rule, or disable the XPATH statement until you are able to debug it.

A similar error case would be an invalid Regular Expression. In most compares of values (If Local Variable equal, if source attribute equal, if destination attribute equal and so on) you have a choice as to the compare type. There is case sensitive, case insensitive (Very useful!), and others. One of those others is Regular expressions (commonly referred to as regex). If you enter an invalid regex value, you can cause a driver startup failure, that looks pretty much the same as bad XPATH startup error.

Variable expansion is a pretty useful feature, and for Global Configuration Values (GCVs) you have two options. In Argument and String Builders there is a token-global-configuration-value available, and for fields that support it (most of them! Designer now shows a widget that lets you select a local variable or GCV on every field the DTD suggests support variable expansion) you can use ~GCV-Name~ to specify a GCV. At driver start time, all the GCV's referenced between tilde characters are expanded as the driver loads, and if a GCV is not actually defined anywhere, the driver will fail to load until either reference is removed or the GCV is defined. See this article (Discussing GCV's in Comments field in Identity Manager) about using ~GCVName~ in the Comments section of a driver, causing it to fail to start. The trace is quiet clear in what the error is, reporting that the GCV GCVName is not defined.

Specific to the eDirectory to eDirectory driver, there is an interesting failure case. The default driver configuration ships with a filter that allows Users, Organization, Organizational Unit, Country, Domain, and Group objects to synchronize. However, most people will clean up the filter and remove object classes not in use, or that they do not want synchronized. Why process events when they are not needed, it just eats up resources. Say for example, all but User was removed from the filter, but at a later date you realize you need to synchronize Group objects. No problem, edit the filter, add class for Group, add CN, Owner, Member, and Description, all the attributes you are interested in. If you try and restart the driver it will fail to load. The eDirectory driver has a requirement that GUID be available in the filter. The driver uses the value of GUID as the Association value. (eDir uses the eDir GUID, Active Directory uses the GUID in Active Directory which is in a different format, Lotus Notes uses the Universal ID (UNID), AS400 and Unix drivers uses the object name, since that must be globally unique. See article: Open Call - IDM Association Values for eDirectory Objects for a list of association keys.

During an upgrade of a driver shim, it may be that the new shim expects certain specific parameters defined at start time. For example the JDBC driver added some new configuration settings in the upgrade of the driver shim from Identity Manager 3.0 to 3.5 levels, and if they were not added to the driver properties, the driver would not start. I have not found a good reference detailing this, but since trace shows the error very clearly, with the specific missing configuration setting it is looking for, it is quite simple to troubleshoot. The easiest way I can think of is to import a new driver of the correct vintage into Designer or into a test Driver Set with basically nonsense connection values (I.e. So that it can never even possibly connect) and look in the XML of the configuration for the setting that trace pointed at. Copy it into your driver and try again. Rinse and repeat until all the mandatory configuration attributes are provided for a successful driver start. After all, just overwriting your current configuration with the default imported one would probably be terrible as it could overwrite any rules and settings you had modified to support your specific environment and implementation.

Continuing to more esoteric examples, a missing JAR or library file will cause a driver start failure, and again the trace will be pretty conclusive reporting it cannot find a class or file to load. An example is in the SAP HR driver, if the remote loader does not have the SAP Jconnect libraries properly configured (see: Using the SAP HR Driver for IDM on a 64-bit OS) then the driver will not correctly start up.

Just like with the changes to a driver that do not require a restart to take effect, if you have any more examples of things you can do to a driver that will cause it to fail to start, please comment on this article so we can have it as a reference point. Especially since it will show up in Google's web index and help others when they are searching help with these sorts of problems.


How To-Best Practice
Comment List
  • Looks like no one took me up on my offer of finding more examples, but I recently ran into one:

    DirXML Log Event -------------------
    Channel: Subscriber
    Status: Error
    Message: Code(-9075) Shutting down because DirXML engine evaluation period has expired.

    If the license on the installation has expired, then you cannot start a driver either. In this case, we had started with the 90 day trial license under IDM 3.01, then installed the real license and all was good.

    Then we updated to IDM 3.5.1 and forgot to put in the 3.5.1 license, and after 90 days we started getting this error.
  • Looks like no one took me up on my offer of finding more examples, but I recently ran into one:

    DirXML Log Event -------------------
    Channel: Subscriber
    Status: Error
    Message: Code(-9075) Shutting down because DirXML engine evaluation period has expired.

    If the license on the installation has expired, then you cannot start a driver either. In this case, we had started with the 90 day trial license under IDM 3.01, then installed the real license and all was good.

    Then we updated to IDM 3.5.1 and forgot to put in the 3.5.1 license, and after 90 days we started getting this error.
No Data