Delving into and beyond the current-op – Part 1
This article is the first part in a series that will delve into the way the NetIQ Identity Manager engine juggles atomic operations.
The first article, will give an overview of how the engine actually handles the current operation and outline key technical details related to this concept. Subsequent articles will delve deeper into how to access and modify operations.
The series was inspired by a problem I had where I needed some information I needed was split across two operations. (In my case, this was a status and an add-association).
My searches turned up nothing. So I asked on the Micro Focus Identity Manager Support Forums. There I got some clues (from Shon Vella, who is one of the authors of the core engine code) about how this actually worked. Thanks to these clues, it became clearer to me why nothing I tried actually worked. In the end, for that particular problem, I found a way to get everything I wanted from the current operation but this nagged at me until I found the time to dig in deeper.
The basics: XDS documents, operations, policies
After spending any time at all with NetIQ Identity Manager, it becomes clear that the "current operation" is central to the way policy operates.
As a quick refresher, an Identity Manager XDS document consists of operations (generally, an event, command or status but not exclusively limited to those categories). In the case of some drivers (like the Delimited Text driver reading an XML file), one can easily end up with thousands (or more) of operations within a single XDS document.
As the engine applies DirXML-Script policy to each operation in turn, that operation becomes the current operation, the object represented by the operation becomes the current object and any tokens that operate on the current object now refer to this object/operation.
Reviewing the trace, shows that once the entire policy has executed against all operations (in the document), any changes the policy has made only now are in the resultant XDS document.
While convenient (and far easier to follow when writing policies), one is limited to looking at the input (or output) node’s child nodes (operations) in sequential order. Once policy processes an operation, there appears to be no way to backtrack or compare it to another operation in the same document.
This approach differs dramatically from that used for XSLT stylesheets.
XLST stylesheets allows access to the entire document and offers flexibility in exactly how (and in what order, for example via sorting or selecting via XPath) the document is processed. XSLT can also operate on other documents than XDS. Well-written XSLT is easy to understand (once you learn the syntax).
The trade-offs with XSLT stylesheets are that it can be somewhat hard to troubleshoot and it is far harder to visualize an XSLT stylesheet in a GUI.
Before DirXML-Script was introduced in NSure Identity Manager 2.0x (As IDM was known back then), XSLT stylesheets were the only available option.
One can still use XSLT stylesheets in the current version of Identity Manager (4.0.2 at the time of writing) but they have found a niche in places where they are better suited. For example, in input/output transforms for Delimited Text and SOAP drivers. In addition, when working with package prompts the only option is XSLT stylesheets.
So far, this is all straightforward. However, if you have ever tried to do something a little more complex with DirXML-Script (like introduce a couple of extra direct add operations), you might start wondering about how this all works behind the scenes?
- How it possible that one can modify/append a few attributes to one of these direct adds before it is sent to the engine?
- What is the context in at this point?
- What has happened to the actual current-object?
- Is there a way to know if the direct write was successful?
- Is there a way to transform an existing regular operation into a direct one?
- Where are all the other operations in the XDS document, can they be accessed and even changed?
The documentation mentions the current-op local variable; the documentation on direct write is confusing. For example:
direct="true" is deprecated in favour of when="direct"
Yet, NetIQ tools like Designer still set the depreciated style (direct="true") which just makes it more difficult for NetIQ to remove this depreciated syntax.
There are scant hints in Support Forum posts, mostly from Shon. However little to no documentation exists for this. So, let us try to fill in the gaps in the documentation and answer some of these questions:
"current operation" node set
As mentioned earlier, when the engine starts processing a policy, it sequentially extracts each operation and applies the rules defined in the policy
While this is happening, one can use XPath to get at any part of the current operation and the operation itself is the default context for such XPath expressions.
For example, the Source DN token will return the current object's Source DN; the equivalent XPath expression is
What is less well known is that there can be multiple element siblings in the "current operation" node set. The engine automatically generates additional operations under certain situations. Where necessary, the engine will append (or preface) such operations as element siblings to the "current operation" node set.
In general, any token that instructs the engine to operate on another object than the "current object", or using the before, after or direct attributes (in tokens that support the aforementioned attributes) causes the engine to add (or modify) another element sibling in the "current operation" node set other than the node currently considered to be the default context node.
In addition, the various XPath and XML tokens can append, remove or modify sibling nodes in the "current operation" node set.
Regardless of how many sibling elements exist. The default context for any XPath expressions (and tokens that operate on the current operation/object) remains the original operation extracted from the XDS document. That is unless you change this default context.
To access these other nodes, one can use the XPath parent axis. For example the XPath expression:
will select the last modify node in the "current operation" node set.
A common use case is adding an extra attribute to a sibling node.
<do-add-dest-attr-value class-name="Group" name="Member">
<do-set-xml-attr expression="../modify[last()]" name="src-dn">
One can use any normal XPath expression against the "current operation" node set, however one cannot navigate further than one level up the parent axis.
It is quite common (but somewhat tedious) to use the various append/set xml tokens to construct completely new operations as siblings of the original operation.
Once all the rules in the policy are applied, the following occurs:
- Sibling nodes marked with a "direct" attribute are processed as out of band operations. I suspect they are actually cloned/stripped out as a separate node set. The trace indicates that the engine saves/suspends the current operation node set. Then the engine switches to process these out of band operation(s). Then the engine restores the saved current operation node set
- The remaining nodes in the "current operation" node set are copied back into the XDS document, taking the same position as the original extracted operation.
- The engine then takes the next operation from the original XDS document and repeats the process with this next operation.
Common misconceptions related to this process are:
- Any changes made against prior "current operations" are not accessible to the Engine whilst it processes the current operation
- It is not possible to manipulate the other operations in the XDS document whilst the Engine is processing the current operation
- There is no way to control the execution of direct/out of band operations - for example to detect if they succeeded or failed, or to trigger another direct operation prior
The current-op variable
This is mentioned in the documentation and there is even a useful example, but it is something I have seen rarely used (mostly I think because people do not really understand its purpose).
Quite simply, this is a policy scoped, default variable (of type node set) which contains a linked copy of a node from within the "current operation" node set. At any time, the contents of current-op are the actual current operation/context.
It only really becomes useful when one has added sibling nodes to the current operation node set, then one can set the current-op variable to the contents of a sibling node and then all the tokens that refer to the current object contained in the updated current-op.
The example earlier (setting source-dn on a sibling modify) could be achieved without resorting to the set/add xml tokens.
<do-add-dest-attr-value class-name="Group" name="Member">
<do-set-local-variable name="saved-op" scope="policy">
<do-set-local-variable name="current-op" scope="policy">
<do-set-local-variable name="current-op" scope="policy">
As one can see there is somewhat of an overhead with switching current-op contexts. There are several steps involved.
- Save a copy of the original current-op context.
- Clone/create/append a sibling node.
- Set the current-op context to the newly created node.
- Perform required logic against this new context.
- Set the current-op context back to the saved context.
If the required logic is located at the end of the policy, then it appears to be okay to skip the last step (restoring the saved context). The engine still copies the entire “current operation” node set back to the main XDS document and continues processing with the next operation from the document.
Despite the overhead, this can make code a lot more readable (though there is an increased risk of confusion as to which object/operation is the current one). Instead of setting many variables and constructing long XPath expressions, one can rely on the built-in tokens that offer extra features like caching, case insensitivity and regular expressions.
Accessing the remainder of the XDS document
This part eluded me for quite some time.
The entire XDS document exists in what appears to be a separate node set, which is accessible in XPath by specifying an absolute path from the root.
So, one accesses the other operations with the XPath expression
//input | //output
What was initially confusing is that the current operation node is not present in this entire document node set.
This makes it not so straightforward to determine the positional context of the current operation node with respect to the entire document. The engine knows exactly where it extracted the current operation from, but I cannot find a way to determine this.
My testing indicates that within this document node set, element nodes under the input or output node are writable, and the engine uses the current version of this document node set each time it starts applying the current policy to the next operation in this node set.
As an extreme example: there might be 1000 operations in an XDS document, after processing say 5, the policy determines that there is no point applying the remainder of this complex policy to the remaining operations (for example, these objects are sub-departments of the current department object which has been vetoed).
A potential solution might be to peek ahead and add a "skip" attribute to the remaining operations indicating that the remaining nodes do not need to be processed. The policy would also need to a rule that detects this skip attribute and break/veto as required. Thus, not applying the remainder of the long/complex policy (which could significantly improve processing time).
I have noticed that the engine does have some limited capability to reorder operations (for example when dealing with a DN-syntax attribute on an object, which references another object that is also synchronised). However, from what I have seen this occurs very early in the event flow
The next post in this series will outline some toolbox rules and general best practices designed to overcome some of the above outlined limitations. For example: determining the position of the current operation with respect to the overall XDS document or counting how many operations are actually in the XDS document. In addition, the post will delve into further detail about manipulating operations, coupled with real world examples of manipulating the XDS document node set.