The first article in this series covered the basics aspects related to the current operation, including an overview of how the engine actually handles the current operation the key technical details related to the current operation.

The second article in this series expanded further on some technical details, outlined general best practices related to working directly with nodes other than the current-op within the "current operation". Finally, it also presented some toolbox rules.

This, the third article in the series deals exclusively with direct operations, it will outline the technical details, review general best practices related to working with direct operations. It will outline how to manipulate and transform direct operations and provide some useful toolbox rules related to direct operations.

Finally, this article will specifically address and answer the following questions asked in the first article:

• How it possible that one can modify/append a few attributes to a direct add operation before sending this to the engine?
• Is there a way to know if a direct write was successful?
• Is there a way to transform an existing regular operation into a direct one?

Direct (out of band) Operations

Categories

As mentioned in the first article in this series, it appears that the Engine processes direct operations added by policy in one of the following ways:

Direct operations that do not modify the destination or source (in other words queries for objects, attributes etc.) are processed immediately and the response is returned to the rule / token which initiated the direct operation.

Direct operations that modify the destination or source system appear to remain as part of the “current operation” up until the point where the engine has finished applying all rules against the current policy object. Then the engine removes each direct operation from the "current operation" and “executes” the direct operations of the current operation back into the main XDS document.

There are other tokens which do not affect the current operation but which take effect immediately. These tokens modify either external systems or other Identity Manager components.

Read-Only (Driver Queries)

These build upon the Java class XdsQueryProcessor, which is an interface that allows the engine to query either the source or destination.

The engine initialises two instances of this class and makes them available to policies as a local variable of type object: srcQueryProcessor and destQueryProcessor with a driver scope.

Additionally, the engine implicitly defines a namespace: "xmlns:query="http://www.novell.com/nxsl/java/com.novell.nds.dirxml.driver.XdsQueryProcessor" which is available to policy.

One can either specify a properly formed XDS query document or define the query based on specific parameters (association, distinguished name, attributes to match/retrieve) and let the XdsQueryProcessor generate the XDS query document based on the supplied parameters.

Many IDM tokens, for example: query token, source, destination attribute tokens, unique name token etc. all make use of this class under the covers. They are generally far superior to calling the java class directly via XPath as the tokens bundle in value-added features like caching and coalescing/pre-fetch to reduce the number of queries executed.

From within the same policy, there does not appear to be a way to intercept, modify or tag these read-only out of band operations (in other words, manipulate the returned queries and the instance documents).

However, a workaround does exist - only when executing queries against the application. NetIQ uses this workaround in various pre-configured drivers. For example Active Directory and Office 365. It addresses the requirement of some components with valued entitlements that only support Query based entitlements (for example Role Mapping Administrator).

The general gist of this approach is:

1. 
   Entitlement object is configured with query-based entitlement that searches for a fictive object class
2. 
   Add an “Intercept outbound queries for fictive-object-class” rule in the output transform. This matches on a query for this fake object class and transforms this into a driver activation ping. In addition it appends some operation data to the query (which allows the response to be identified and tagged)
3. 
   Add an “intercept fictive-object-class tagged query response & status” rule which looks for the returned instance/status operations which strips off the actual response (which is a predictable response to the driver ping) and generates a new fake response.

This is really just a workaround in the driver so that the driver can answer a query generated by either Reporting or User App code map refresh. Instead of responding with an error, the driver can respond with a single - fixed, predetermined value that appears as if it actually originated from the connected system.

One could conceivably use the same technique for other purposes than the one outlined above (returning a single faked instance from a tagged query).

Channel Write (aka Channel Write-Back)

Introduced in Identity Manager Version 1.1

However, as an aside, at the time the product's name was actually DirXML 1.1

Upon introduction, they called this “Channel Write-Back”. The terms “direct write” or “write directly to source/destination data store” are preferred nowadays.

Direct write relies on the Java class XdsCommandProcessor, which is an interface that allows the engine to execute an XDS command against either the source or destination.

The engine initialises two instances of this class and makes them available to policies as a local variable of type object: destCommandProcessor and srcCommandProcessor with a driver scope.

Additionally, the engine implicitly defines a namespace "xmlns:cmd="http://www.novell.com/nxsl/java/com.novell.nds.dirxml.driver.XdsCommandProcessor" which is available to policy.

One must supply a properly formed XDS operation document to the XdsCommandProcessor class. It will then execute this operation against the source or destination (you need to specify which by including destCommandProcessor or destCommandProcessor along with the XDS operation document). Once the command has executed, the class returns the result of the command (As a standard XDS status operation document). Once can examine this result to determine if the command was successful or not.

Of the policy-sets, only Schema mapping and input or output transforms may be applied. This only applies in one specific scenario:

• Where the direct write crosses from application to eDirectory namespace. (Or vice versa)

This means that write-back to connected system in the input transform does not pass through the output transform (which can present a problem for drivers like the SOAP driver).

In essence, the direct write takes the shortest possible route to the source/destination. As a result, the direct write skips over much of the built-in engine functionality.

1. On the publisher channel the engine will not identify and add automatically add auxiliary classes for an auxiliary attribute.
2. Additionally, publisher optimize modify is not performed.
3. Finally, direct writes also bypass the driver filter (for both channels)

Direct operations within the current operation node set are first tagged with a direct="src" or direct="dest" attribute. As a result, direct operations are accessible (read-write) via XPath.

This appears to be undocumented (this originated via a tip from Shon Vella in the NetIQ Support forums). The tip also noted that this was only supported in version 3.6.1 or higher of Identity Manager.

This enhancement offers substantially more control of exactly when and how the engine processes a direct write. This allows significantly enhanced functionality. Particularly when combined with the Java XdsCommandProcessor objects (which as mentioned earlier, the engine automatically makes available to policy).

External Read/Write (with respect to the Driver)

By definition, these actions modify either external systems or other IDM components.

It might seem odd that things like add role are considered external but if one takes the case of Roles and Resources, the engine simply passes the request to a SOAP endpoint on the User Application/RBPM server and that handles the rest.

The following external direct actions set an error variable, which can be accessed/inspected for success/fail.

• 
  Add Resource
• 
  Remove Resource
• 
  Add Role
• 
  Remove Role
• 
  Start Workflow
• 
  Send Email
• 
  Send Email from Template
• 
  Clear SSO Credential
• 
  Set SSO Credential
• 
  Set SSO Passphrase

Another vector for external actions is calling arbitrary java code from policy. Best practices suggest using restraint with this capability and avoiding it as much as possible, as it can decrease the stability and reliability of the engine.

Best Practices for Direct Operations

• 
  Reserve the techniques explained in this series for only when you absolutely cannot find a way to solve your problem via the standard engine functionality. The engine really is well designed for most use-cases.
• 
  When using verified direct write, place this as the last rule in your policy.
• 
  Always define an action (send email, driver retry or fatal) for a failure of a verified direct operation. Otherwise, just use a regular (unverified) direct operation.
• 
  For an object, always consider actions within same “current operation” that apply to the same object prior (before) a direct operation. Will the direct operation fail without these?
• 
  When transforming a regular operation to a direct write, try to place the policy as close to the last command transform policy as possible.
• 
  Try to avoid overuse of the do-append-xml-element, do-append-xml-text and do-set-xml-attr. Instead, ask yourself, how can I achieve this with the existing object/attribute tokens? This can often make for more legible and resilient code. It also allows the engine to be smarter with caching and optimizations.
• 
  Add a comment or trace to indicate when switching to another current-op. This avoids confusion.

Selecting direct operations

Select all direct operations from within the “current operation”

Use the following XPath expression.

../*[@direct="dest"] | ../*[@direct="src"]

It might be sufficent to just match on the direct attribute, regardless of its value, however to err on the side of safety, I would suggest the expression above.

Toolbox Rule: Transform a regular operation to a direct operation.

Again, I cannot claim credit for discovering this. However, little to no documentation exists related to this. This technique is so incredibly simple, but effective. Including it here as it logically belongs in this list of generic direct-write toolbox rules.

This can be effective in various scenarios:

1. 
   Where you have conflicting authoritative requirements between for example add and modify. Say, you have an attribute that you want the driver filter to enforce as authoritative in the destination system (so you have the filter set to "reset" for this attribute in this direction) for all scenarios except initial add.
2. 
   You identify a criteria where you need to ensure that the attributes within the current modify operation should ignore the "publisher optimize modify" option specified in the driver filter.
3. 
   The automatic addition of aux classes is undesirable.

Place the following as the last policy of your publisher command transformation, and adjust the condition criteria as required.

<rule>
    <description>Transform a regular operation to a direct operation.</description>
    <conditions>
        <and>
            <if-operation op="equal">add</if-operation>
            <if-dest-dn op="available"/>
        </and>
    </conditions>
    <actions>
        <do-set-xml-attr expression="." name="direct">
            <arg-string>
                <token-text>dest</token-text>
            </arg-string>
        </do-set-xml-attr>
    </actions>
</rule>

The engine will then process this operation as if it was always a direct operation once it has applied all other rules in defined in the current policy to this current operation.

Toolbox Rule: Generic validated direct-write

The following rule derives from a snippet posted by Shon Vella in the NetIQ Identity Manager Support Forums.

It emulates the engine behaviour described above, but gives access to the result/response. Thus enabling the creation of policy that can better handle failures or errors that occurred because of the actions performed in the direct write.

<rule>
    <description>Generic validated direct-write</description>
    <comment xml:space="preserve">Supports direct operations in both directions.
Shuts down the driver if the validated direct write fails.
If less severe, change the "fatal" status to a send notification email.
Best placed as the last rule in a policy, can also be used as an include.</comment>
    <conditions>
        <and/>
    </conditions>
    <actions>
        <do-set-local-variable name="lineSeparator" notrace="true" scope="policy">
            <arg-string>
                <token-xpath expression="java.lang.System:getProperty('line.separator')" notrace="true"/>
            </arg-string>
        </do-set-local-variable>
        <do-for-each>
            <arg-node-set>
                <token-xpath expression="../*[@direct=&quot;dest&quot;] | ../*[@direct=&quot;src&quot;]
"/>
            </arg-node-set>
            <arg-actions>
                <do-if>
                    <arg-conditions>
                        <and>
                            <if-xpath op="true">$current-node/@direct="src"</if-xpath>
                        </and>
                    </arg-conditions>
                    <arg-actions>
                        <do-set-local-variable name="dirCommandProcessor" scope="policy">
                            <arg-object>
                                <token-local-variable name="srcCommandProcessor"/>
                            </arg-object>
                        </do-set-local-variable>
                    </arg-actions>
                </do-if>
                <do-if>
                    <arg-conditions>
                        <and>
                            <if-xpath op="true">$current-node/@direct="dest"</if-xpath>
                        </and>
                    </arg-conditions>
                    <arg-actions>
                        <do-set-local-variable name="dirCommandProcessor" scope="policy">
                            <arg-object>
                                <token-local-variable name="destCommandProcessor"/>
                            </arg-object>
                        </do-set-local-variable>
                    </arg-actions>
                </do-if>
                <do-set-local-variable name="result" scope="policy">
                    <arg-node-set>
                        <token-xpath expression="cmd:execute($dirCommandProcessor, $current-node)//status"/>
                    </arg-node-set>
                </do-set-local-variable>
                <do-if>
                    <arg-conditions>
                        <and>
                            <if-xpath op="true">$result//@level = 'error'</if-xpath>
                        </and>
                    </arg-conditions>
                    <arg-actions>
                        <do-trace-message color="red" level="0">
                            <arg-string>
                                <token-text notrace="true" xml:space="preserve">Error processing direct command, aborting further processing. Error details</token-text>
                                <token-local-variable name="lineSeparator" notrace="true"/>
                                <token-xml-serialize notrace="true">
                                    <token-xml-serialize notrace="true">
                                        <token-replace-all notrace="true" regex="[\n\r]" replace-with="$lineSeparator$">
                                            <token-xpath expression="$result/node()" notrace="true"/>
                                        </token-replace-all>
                                    </token-xml-serialize>
                                </token-xml-serialize>
                            </arg-string>
                        </do-trace-message>
                        <do-trace-message color="yellow" level="1" notrace="true">
                            <arg-string>
                                <token-text notrace="true" xml:space="preserve">Direct command details:</token-text>
                                <token-xml-serialize notrace="true">
                                    <token-local-variable name="current-node" notrace="true"/>
                                </token-xml-serialize>
                            </arg-string>
                        </do-trace-message>
                        <do-strip-xpath expression="$current-node"/>
                        <do-status level="fatal">
                            <arg-string>
                                <token-text xml:space="preserve">Direct operation could not be completed.</token-text>
                            </arg-string>
                        </do-status>
                        <do-break/>
                    </arg-actions>
                    <arg-actions/>
                </do-if>
                <do-strip-xpath expression="$current-node"/>
            </arg-actions>
        </do-for-each>
    </actions>
</rule>

Toolkit Rule: Convert Regular add to validated direct write empty modify

This rule comes in handy if one has additional requirements that are only possible once the object is already present in the destination system, yet the current operation is an add operation. In other words, a "chicken and the egg" scenario. It builds upon the previous toolkit rules outlined above and works best in combination with the previous toolkit rule “Generic validated direct-write”.

Placement wise, this rule should located immediately prior to the previous toolkit rule “Generic validated direct-write” within the same policy. I often have “Generic validated direct-write” as a library object and reference it via an include. This works well if the required action taken on failure is the same.

This essentially takes the add operation, constructs an empty modify from it (which can then be optionally reacted to by subsequent policies). It then converts the add operation to a direct add. If “Generic validated direct-write” is a subsequent rule in the same policy, then the direct add is processed in a validated fashion and logic can be added to react accordingly if the write fails.

This is flexible enough that it could also be adapted to other scenarios.

In my testing the empty modify is processed without issue when the destination is the Identity Vault, if used in the opposite direction, your results may vary depending on which driver shim will consume the modify. In principle, though it should be harmless as it is empty.

If the destination is Identity Manager, it is usually considered better practice to simply write a trigger attribute and then use a business logic (Null) driver to perform the “modify” actions based on this trigger. The reasons that this is best are threefold.

1. 
   This ensures a cached event, thus the business logic driver could be safely stopped (but not set to disabled) and the action will still be performed when the business logic driver is restarted.
2. 
   Achieves logical separation between business logic and data transport, which makes for simpler drivers that are easier to test and maintain.
3. 
   Ensures that the same business logic is applied regardless which source generated the trigger event.

However there are definitely scenarios where it may be an absolute requirement that the post-add actions only happen as part of the input (publisher) process - or not at all.

An alternative approach is highlighted in the Active Directory driver. In that approach, a subscriber policy tags add events with an operation property. Subsequently, when the add operation is successful, an add-association is generated and the operation data copied from the add operation. An input transform policy can detect the combination of add-association and operation property tag.

The downsides of that approach are:

1. 
   Only works on subscriber channel
2. 
   Only works where the engine restores input operation data to any related output operations (status, add-association etc.)
3. 
   Only triggers on add events
4. 
   Write-back to connected system can be tricky with drivers where the shim does not translate XDS (SOAP for example).

In contrast, the "Convert Regular add to validated direct write empty modify" offers solutions to all of the above limitations.

<rule>
    <description>Convert add to direct add   empty modify trigger</description>
    <comment xml:space="preserve">Add a cloned/empty modify event (inherits all attributes and operation data from add)
Convert the the add event to a direct add (so it will write directly to the destination system)
Schedule this rule immediately prior to the Toolkit Rule: "Generic validated direct-write".
Once the add event has been written directly, the empty modify event can then be used in subsequent policies to trigger any actions which might require the object to already exist in destination system.</comment>
    <conditions>
        <and>
            <if-operation op="equal">add</if-operation>
            <if-dest-dn op="available"/>
            <if-association op="available"/>
        </and>
    </conditions>
    <actions>
        <do-set-dest-attr-value name="¤dummy¤" when="after">
            <arg-value type="string"/>
        </do-set-dest-attr-value>
        <do-strip-xpath expression="../modify[last()]/modify-attr[@attr-name='¤dummy¤']"/>
        <do-set-op-property name="from-direct-add">
            <arg-string>
                <token-text xml:space="preserve">true</token-text>
            </arg-string>
        </do-set-op-property>
        <do-clone-xpath dest-expression="../modify[last()]" src-expression="operation-data"/>
        <do-clear-op-property name="from-direct-add"/>
        <do-set-xml-attr expression="." name="direct">
            <arg-string>
                <token-text>dest</token-text>
            </arg-string>
        </do-set-xml-attr>
    </actions>
</rule>

Note that in the above rule, I try to adhere to the best practice guideline "Try to avoid overuse of the do-append-xml-element, do-append-xml-text and do-set-xml-attr". Only the last rule uses one of these tokens.

Contrast this against an earlier draft of the same code (where I ignored my own guidelines) - which is listed below. This code also worked, but was far longer, harder to understand and needlessly replicated built-in features offered by the engine.

<rule>
	<description>Trigger creation of object performed as a validated direct write (early draft)</description>
	<conditions>
		<and>
			<if-operation op="equal">add</if-operation>
			<if-dest-dn op="available"/>
		</and>
	</conditions>
	<actions>
		<do-append-xml-element expression=".." name="modify"/>
		<do-set-local-variable name="attributes" scope="policy">
			<arg-node-set>
				<token-split csv="true" delimiter=",">
					<token-text xml:space="preserve">class-name,dest-dn,dest-entry-id,event-id,qualified-src-dn,src-dn,src-entry-id</token-text>
				</token-split>
			</arg-node-set>
		</do-set-local-variable>
		<do-for-each>
			<arg-node-set>
				<token-xpath expression="self::add/@*[name()=$attributes]"/>
			</arg-node-set>
			<arg-actions>
				<do-clone-xpath dest-expression="../modify[last()]" src-expression="$current-node"/>
			</arg-actions>
		</do-for-each>
		<do-clone-xpath dest-expression="../modify[last()]" src-expression="self::add/association"/>
		<do-set-xml-attr expression="../modify[last()]/association" name="state">
			<arg-string>
				<token-text xml:space="preserve">associated</token-text>
			</arg-string>
		</do-set-xml-attr>
		<do-append-xml-element expression="../modify[last()]" name="operation-data"/>
		<do-clone-xpath dest-expression="../modify[last()]/operation-data" src-expression="operation-data/node()"/>
		<do-clone-xpath dest-expression="../modify[last()]/operation-data" src-expression="operation-data/@*"/>
		<do-set-xml-attr expression="../modify[last()]/operation-data" name="from-direct-add">
			<arg-string>
				<token-text xml:space="preserve">true</token-text>
			</arg-string>
		</do-set-xml-attr>
		<do-set-xml-attr expression="." name="direct">
			<arg-string>
				<token-text>dest</token-text>
			</arg-string>
		</do-set-xml-attr>
	</actions>
</rule>

Quickly, comparing the two versions of this policy:

1. 
   Create an empty sibling operation of type modify. In contrast, the newer code lets the engine implicitly perform this by selecting when="after" in the do-set-dest-attr-value token.
2. 
   Create a node-set containing common attribute names found in operations. Then loop through any attributes in the current-op that are present in the variable above.
3. 
   ​If present in the current-op, clone the attribute to the sibling modify operation created in the first step. Again the engine implicitly handles this as part of selecting when="after" in the do-set-dest-attr-value token. It is also smart enough to avoid copying some attributes like "timestamp" which do not belong in the newly created sibling operation.

• 
  Clone the association and association state to new sibling operation. Yet again the engine implicitly handles this as part of selecting when="after" in the do-set-dest-attr-value token.
• 
  Create an operation-data child node of the new sibling operation, copy the current-op's operation-data contents (if any). Finally tag the empty modify with an operation property indicating its relation to the current-op. The engine does not handle this automatically, so we must copy this manually either way. However, when following best practices, I used three rules instead of four and the resulting rules are arguably more legible.
• 
  Remove the dummy attribute; this is only necessary in the best practices version. One could argue that this looks a bit clunky (add a dummy attribute, to trigger some engine behaviour, finally strip the dummy attribute). However, I still think such policy is easier to follow and less prone to errors