Cleaning up NULL Association values - Delimited Text Driver


While troubleshooting an issue on yet another customer system, one of the first things I noticed in the driver traces was that the Association value was not being set correctly and left as a NULL value. Looking at the driver trace it will look something like: (note the association is null <association/>)


<nds dtdversion="1.1" ndsversion="8.6" xml:space="default"> <input> <add class-name="User" src-dn=""> <association/> <add-attr attr-name="UserID"> <value type="string">123456</value> </add-attr> <add-attr attr-name="Given"> <value type="string">Paul</value> </add-attr> <add-attr attr-name="Surname"> <value type="string">Jones</value> </add-attr> </add> </input> </nds>


I have seen this now on a few different systems. I suspect this occurs because someone was getting started using the Delimited Text Driver and did not fully appreciate the association value setting. Out of the box the Delimited Text Driver is configured to use the input value of Email as the default association value. As the out of the box default driver is seldom (if ever) used in that configuration, it gets modified to match the needs of the data being input. When the driver is reconfigured to match the actual input fields, the setting for the Association may or may not be changed to match the input fields. At least that seems to be the likely root cause for the situations I have encountered so far. (In one case the association was left at "Email", but the input field name was "Mail".)

As a result of not having the Delimited Text Driver configured to use an available input field correctly for the Association, the driver sets it to a Null value. In this situation an Add event will result in the Association being set to Null. If a Match is found with an existing target object the event is sent through the Merge Optimization process and will update any changed attribute values.

If the Delimited Text Driver configuration is not changed, every event coming in (always an Add) will be considered as not associated and then sent through the the Matching Policies, where it will find the matching object once more, and then be sent to the Merge Optimization, yet again. If there had been a valid association in place, the steps through the Match and Merge cycles would be skipped and processing would go directly to the Modify Optimization instead.

Functionally the extra steps would not cause much more than wasted processor cycles and some time. As a result, if this situation is not identified early and change, it is likely to be ignored as everything continues to work, just with a bit of unnecessary extra processing. There is also the extra overhead in any trace generated that adds to the file space consumed and the extra noise when troubleshooting.

If the configuration oversight (error) is caught and corrected, the next time an input match is found it results in an error as the matching object is technically associated, even though the value is null. This results in a failed input/update and can lead to other issues that require more work to fix after the fact.

So what do you do once the driver has been running and creating objects for some time before this situation is identified?

As always in Identity Manager there are a few different options to choose when making changes. The following options show some different ways I have used to resolve this scenario, from a brute force one time approach to a nice clean option that works quietly in the background with no fanfare.

The first approach is to fix everyone at once using a Subscriber Policy and a Trigger Job all Associated objects. The following rule checks that the existing association value matches the desired attribute value and if it does not, remove the existing association and replace it with the desired association value. This can also be used when the association was incorrectly set to a wrong attribute value. A little modification and you can spruce it up to your specific needs.


<rule> <description>Fix Invalid Association Value</description> <comment xml:space="preserve">If the Asscociation value does not match the User ID value, it is invalid.</comment> <conditions> <and> <if-class-name mode="nocase" op="equal">user</if-class-name> </and> </conditions> <actions> <do-set-local-variable name="lv-userID" scope="policy"> <arg-string> <token-attr name="UserID"/> </arg-string> </do-set-local-variable> <do-if> <arg-conditions> <and> <if-association mode="nocase" op="not-equal">$lv-UserID$</if-association> </and> </arg-conditions> <arg-actions> <do-remove-src-attr-value name="DirXML-Associations"> <arg-value type="structured"> <arg-component name="nameSpace"> <token-text xml:space="preserve">1</token-text> </arg-component> <arg-component name="volume"> <token-global-variable name=""/> </arg-component> <arg-component name="path"> <token-association/> </arg-component> </arg-value> </do-remove-src-attr-value> <do-add-association when="after"> <arg-dn> <token-src-dn/> </arg-dn> <arg-association> <token-local-variable name="lv-userID"/> </arg-association> </do-add-association> <do-strip-xpath expression="association/text()"/> <do-append-xml-text expression="association"> <arg-string> <token-local-variable name="lv-userID"/> </arg-string> </do-append-xml-text> </arg-actions> <arg-actions/> </do-if> </actions> </rule>


This first sample rule would need to be wrapped in additional rules to process correctly for a Trigger Job, which also needs to be setup and configured. As this method is a one shot process, the driver should not be processing any input documents while this is being run and the driver also has to be reconfigured to set the correct association value after the fix processing has completed.

A second option is to check for the existence of a Null association value on the matching object in the Publisher Command Transform during normal input processing and update it prior to the association check that will direct the event to the Matching Policies. This method uses the fact that the input event will never be declared associated as the matching object has a Null association value. Using this approach the driver can be reconfigured and process input files normally. Another advantage is the additional policy only executes for the event if it is not identified as being associated. Any null association will be corrected with the next update with new input for the object allowing the Null associations to be cleaned up over time.

I am providing two methods of to check and clean up the Null association. The first option could also be modified for use in the Subscriber by adjusting the target (source or destination) accordingly. It can also be modified to work for another driver if you provide the target driver's DN separately than using the auto generated GSV. This method relies on reading the DirXML-Associations attribute and then iterating through the values to find the check and update one matching the running driver.

The first two Rules scope out what is permitted to be processed in this Policy. (I am a big fan of scoping rules at the start of every policy - unless there is a single rule that is self scoping - to quickly exit further processing if the event does not match the processing required. I find this greatly cleans up traces making troubleshooting a lot easier.) The condition on the third rule is also a safeguard in case the Association has not been configured correctly on the driver. If the driver configuration has resulted in a Null value for the input association, then skip further processing and trying to clean up associations. This condition also allows you to put this code in place and change the association configuration later.


<policy> <rule> <description>Break if Not a User Add operation</description> <comment xml:space="preserve">If this is NOT an Add event for a User, then Break from further processing.</comment> <conditions> <or> <if-operation mode="nocase" op="not-equal">add</if-operation> <if-class-name mode="nocase" op="not-equal">user</if-class-name> </or> </conditions> <actions> <do-break/> </actions> </rule> <rule> <description>Break If Associated</description> <comment xml:space="preserve">If Associated then Break out of this Policy and let the operation proceed</comment> <conditions> <and> <if-association op="associated"/> </and> </conditions> <actions> <do-break/> </actions> </rule> <rule> <description>Check for existing Match with NULL Association</description> <comment xml:space="preserve">Due to prior issues the existing Vault object may already have a NULL Association value. Query for an existing object that matches the desired ID number (target Association) and verify the existing Association value is not NULL. If the existing Association value is NULL, remove that entry and set the Association with the correct value. Note: If the input Association value (source Association) is NULL, then this policy will exit with no changes made as the provided Association will also be NULL.</comment> <conditions> <and> <if-association mode="regex" op="equal">. </if-association> </and> </conditions> <actions> <do-set-local-variable name="match" scope="policy"> <arg-node-set> <token-query class-name="User"> <arg-match-attr name="UserID"/> <arg-string> <token-text xml:space="preserve">DirXML-Associations</token-text> </arg-string> </token-query> </arg-node-set> </do-set-local-variable> <do-for-each> <arg-node-set> <token-xpath expression='$match/attr[@attr-name="DirXML-Associations"]/value'/> </arg-node-set> <arg-actions> <do-set-local-variable name="assoc-state" scope="policy"> <arg-string> <token-xpath expression='$current-node/component[@name="nameSpace"]'/> </arg-string> </do-set-local-variable> <do-set-local-variable name="assoc-driver" scope="policy"> <arg-string> <token-xpath expression='$current-node/component[@name="volume"]'/> </arg-string> </do-set-local-variable> <do-set-local-variable name="assoc-value" scope="policy"> <arg-string> <token-xpath expression='$current-node/component[@name="path"]'/> </arg-string> </do-set-local-variable> <do-if> <arg-conditions> <and> <if-local-variable mode="nocase" name="assoc-driver" op="equal"></if-local-variable> <if-local-variable mode="regex" name="assoc-value" op="not-equal">. </if-local-variable> </and> </arg-conditions> <arg-actions> <do-trace-message disabled="true"> <arg-string> <token-text xml:space="preserve">FIX ME!!</token-text> </arg-string> </do-trace-message> <do-remove-dest-attr-value direct="true" name="DirXML-Associations"> <arg-dn> <token-xpath expression="$match/@src-dn"/> </arg-dn> <arg-value type="structured"> <arg-component name="nameSpace"> <token-local-variable name="assoc-state"/> </arg-component> <arg-component name="volume"> <token-local-variable name="assoc-driver"/> </arg-component> <arg-component name="path"> <token-local-variable name="assoc-value"/> </arg-component> </arg-value> </do-remove-dest-attr-value> <do-add-association direct="true"> <arg-dn> <token-xpath expression="$match/@src-dn"/> </arg-dn> <arg-association> <token-op-attr name="UserID"/> </arg-association> </do-add-association> </arg-actions> <arg-actions/> </do-if> </arg-actions> </do-for-each> </actions> </rule> </policy>


Although this method works well enough, I disliked the additional overhead of reading the DirXML-Associations attribute and iterating through the values to get to the one for this driver in need of correction. I also could clean up the use of local variables in each loop as well. I felt I had all of the information in front of my, just not enough time to figure it out.

While digging into another problem I stumbled across a hint in the Roles and Resources driver that cleans up any association generated by trying to migrate objects in that driver. In that rule any association is removed and if the association state is "migrate" (or nameSpace='4') remove it using a null value for the path component. (Interestingly this code sets the null value using an Xpath token, though my testing shows simply using "" also works.)


<rule> <description>Get rid of any association that might be there and veto the original event</description> <comment xml:space="preserve">Clean up any association attribute associated with the source object because this driver doesn't need associations and in large numbers they cause a lot of overhead.</comment> <conditions/> <actions> <do-if> <arg-conditions> <and> <if-association op="available"/> </and> </arg-conditions> <arg-actions> <do-remove-association> <arg-association> <token-association/> </arg-association> </do-remove-association> </arg-actions> <arg-actions> <do-if> <arg-conditions> <and> <if-xpath op="true">association/@state='migrate'</if-xpath> </and> </arg-conditions> <arg-actions> <do-remove-src-attr-value name="DirXML-Associations"> <arg-value type="structured"> <arg-component name="volume"> <token-text xml:space="preserve"></token-text> </arg-component> <arg-component name="nameSpace"> <token-text xml:space="preserve">4</token-text> </arg-component> <arg-component name="path"> <token-xpath expression="''"/> </arg-component> </arg-value> </do-remove-src-attr-value> </arg-actions> </do-if> </arg-actions> </do-if> <do-veto/> </actions> </rule>


Using that example, I cleaned up my previous solution's final rule with a nice simple and compact test to check for any null association values and replace it with the desired attribute value.


<rule> <description>Check for existing Match with NULL Association</description> <conditions> <and> <if-association mode="regex" op="equal">. </if-association> </and> </conditions> <actions> <do-set-local-variable name="match" scope="policy"> <arg-node-set> <token-query class-name="User"> <arg-match-attr name="UserID"/> </token-query> </arg-node-set> </do-set-local-variable> <do-if> <arg-conditions> <and> <if-xpath op="true">$match/association/@state='associated'</if-xpath> <if-xpath op="true">$match/association=''</if-xpath> </and> </arg-conditions> <arg-actions> <do-remove-dest-attr-value direct="true" name="DirXML-Associations"> <arg-dn> <token-xpath expression="$match/@src-dn"/> </arg-dn> <arg-value type="structured"> <arg-component name="volume"> <token-text xml:space="preserve"></token-text> </arg-component> <arg-component name="nameSpace"> <token-text xml:space="preserve">1</token-text> </arg-component> <arg-component name="path"> <token-text xml:space="preserve"/> </arg-component> </arg-value> </do-remove-dest-attr-value> <do-add-dest-attr-value direct="true" name="DirXML-Associations"> <arg-dn> <token-xpath expression="$match/@src-dn"/> </arg-dn> <arg-value type="structured"> <arg-component name="volume"> <token-global-variable name=""/> </arg-component> <arg-component name="nameSpace"> <token-text xml:space="preserve">1</token-text> </arg-component> <arg-component name="path"> <token-op-attr name="UserID"/> </arg-component> </arg-value> </do-add-dest-attr-value> </arg-actions> <arg-actions/> </do-if> </actions> </rule>


I hope this has been an enlightening journey and provides some helpful code examples for your use as well.



Comment List