Wikis - Page

Really Optimized Modifies

1 Likes

Problem



When an IDM driver such as delimited text or triggerless publication in JDBC doesn't have the information of the prior value of data after a change, your logic ends up having to remove all values then add them all back. This can cause a lot of extra unnecessary events. This is a solution to truly optimize those modify operations.

This approach could be applied in many cases where a multi-valued attribute is coming from a source where the old values are not needed. Whether this would be necessary, or if a simple remove-all-values approach will do, has to do with the downstream drivers, including whether they support loopback and whether they could be affected by all those additional changes.

Scenario



One client I work with has a commercial database application to support non-profit membership registration. The vendor will not support their application if they modify the schema in any way, including adding any triggers to application tables. This made even an indirect triggered driver a nonstarter. The triggerless publisher JDBC implementation was the only way to interface to this application.

The application stores its group memberships in a single text field as a comma-delimited string. In order to synchronize this into the Identity Vault, it needed to be translated to a list of DNs. After eDirectory, the next downstream targets are two separate instances of Active Directory.

AD can subscribe to get just the changes from eDirectory. Because that implementation has no loopback protection (which is impossible for Novell to implement, because AD doesn’t actually record who makes each change), the change comes back on the next polling cycle on the publisher channel. Prior to IDM 3.5.1 and Active Directory 2003 Native, this came back as the entire group list with the one addition or deletion. (This solution is run on mixed mode AD 2003, Native Mode AD 2000 and IDM 3.01 on eDirectory 8.8.1.)

To make matters worse, since eDirectory was removing the user from all groups and then adding it back again, this might be 15 changes from eDirectory to AD, and all 15 groups would then be re-published and have to be re-optimized by both AD publishers. This was taking so long that a race condition was set up between the two AD drivers, and as a result some group memberships were getting lost.

My first attempt at this was to strip down the change in the event transformation to only the adds that are new. This worked to simmer down the add to a group behavior, but the "remove from a group" stopped working. My faulty assumption was that optimize modify would actually check the destination and compare it to the source to calculate the deltas. It does not do that. What it did was to take separate adds for each value and combine them into a single add with several add-values.

Solution



I wrote the following rule that queries for the destination value and calculates the delta. It sits in the command transformation policy set of the publisher channel, after the optimize modify has completed. This particular solution has been written for IDM 3.01. It could also run on 3.5x or later and would have been easier to write that way, but that upgrade isn't scheduled until later this year.

In this example, the user was removed from the group “Sharepoint Readers” and added to the groups “Whats New” and “All ISOT Users”. When a change to group membership comes in via the triggerless publisher implementation within the JDBC driver, we get the entire record as follows:

 
<nds dtdversion="2.0" ndsversion="8.x" xmlns:jdbc="urn:dirxml:jdbc">
<source>
<product build="20061207_1003" instance="IMIS-IDV"version="2.1.5">DirXML Driver for JDBC</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<modify class-name="dbo.Members" event-id="PK_ID=267385,table=MEMBERS,schema=DBO"jdbc:database-local-time="2008-03-26 19:34:02.0" src-dn="PK_ID=267385,table=MEMBERS,schema=DBO">
<association>PK_ID=267385,table=MEMBERS,schema=DBO</association>
<modify-attr attr-name="LAST_FIRST">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LAST_FIRST">
<add-value>
<value type="string">SCANNON, LOU</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ADDRESS_1">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ADDRESS_1">
<add-value>
<value type="string">1234 Any Place</value>
</add-value>
</modify-attr>
<modify-attr attr-name="FULL_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FULL_NAME">
<add-value>
<value type="string">Lou Scannon</value>
</add-value>
</modify-attr>
<modify-attr attr-name="JOIN_DATE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="WEBSITE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FAX">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FAX">
<add-value>
<value type="string">(617) 555-1111</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ADDRESS_2">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ADDRESS_2">
<add-value>
<value type="string"></value>
</add-value>
</modify-attr>
<modify-attr attr-name="EMAIL">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="EMAIL">
<add-value>
<value type="string">lscannon@hri.com</value>
</add-value>
</modify-attr>
<modify-attr attr-name="COMPANY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="COMPANY">
<add-value>
<value type="string">Horseshoe Road Inn</value>
</add-value>
</modify-attr>
<modify-attr attr-name="STATUS">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="STATUS">
<add-value>
<value type="string">A</value>
</add-value>
</modify-attr>
<modify-attr attr-name="WORK_PHONE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="WORK_PHONE">
<add-value>
<value type="string">(617) 555-2112</value>
</add-value>
</modify-attr>
<modify-attr attr-name="EXCLUDE_DIRECTORY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="EXCLUDE_DIRECTORY">
<add-value>
<value type="integer">0</value>
</add-value>
</modify-attr>
<modify-attr attr-name="PAID_THRU">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="MEMBER_STATUS">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="MEMBER_STATUS">
<add-value>
<value type="string">S</value>
</add-value>
</modify-attr>
<modify-attr attr-name="TOLL_FREE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="TOLL_FREE">
<add-value>
<value type="string">TEST COMPANY</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ZIP">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ZIP">
<add-value>
<value type="string">06543</value>
</add-value>
</modify-attr>
<modify-attr attr-name="CATEGORY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="CATEGORY">
<add-value>
<value type="string">S</value>
</add-value>
</modify-attr>
<modify-attr attr-name="MEMBER_STATUS_DATE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="MEMBER_STATUS_DATE">
<add-value>
<value type="time">20060918000000000000000</value>
</add-value>
</modify-attr>
<modify-attr attr-name="CO_ID">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="CO_ID">
<add-value>
<value type="string">C267384</value>
</add-value>
</modify-attr>
<modify-attr attr-name="OLD_LOGIN">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="OLD_LOGIN">
<add-value>
<value type="string">Lou Scannon</value>
</add-value>
</modify-attr>
<modify-attr attr-name="PREFIX">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LOGIN_DISABLED">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LOGIN_DISABLED">
<add-value>
<value type="integer">0</value>
</add-value>
</modify-attr>
<modify-attr attr-name="DATE_ADDED">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="DATE_ADDED">
<add-value>
<value type="time">20060918000000000000000</value>
</add-value>
</modify-attr>
<modify-attr attr-name="COUNTRY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="COUNTRY">
<add-value>
<value type="string">UNITED STATES</value>
</add-value>
</modify-attr>
<modify-attr attr-name="MEMBER_TYPE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="MEMBER_TYPE">
<add-value>
<value type="string">CMC</value>
</add-value>
</modify-attr>
<modify-attr attr-name="IDM">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="IDM">
<add-value>
<value type="integer">1</value>
</add-value>
</modify-attr>
<modify-attr attr-name="EXCLUDE_MAIL">
<remove-all-values/>
</modify-attr>
<modify-attr
attr-name="EXCLUDE_MAIL">
<add-value>
<value type="integer">0</value>
</add-value>
</modify-attr>
<modify-attr attr-name="HOME_PHONE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="WEB_LOGIN">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="WEB_LOGIN">
<add-value>
<value type="string">LScannon</value>
</add-value>
</modify-attr>
<modify-attr attr-name="SEC_GRP">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="SEC_GRP">
<add-value>
<value type="string">All ISOT Users,Whats New</value>
</add-value>
</modify-attr>
<modify-attr attr-name="COUNTY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LAST_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LAST_NAME">
<add-value>
<value type="string">Scannon</value>
</add-value>
</modify-attr>
<modify-attr attr-name="MIDDLE_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FIRST_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FIRST_NAME">
<add-value>
<value type="string">Lou</value>
</add-value>
</modify-attr>
<modify-attr attr-name="CITY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="CITY">
<add-value>
<value type="string">Cambridge</value>
</add-value>
</modify-attr>
<modify-attr attr-name="CO_MEMBER_TYPE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="CO_MEMBER_TYPE">
<add-value>
<value type="string">FMCM</value>
</add-value>
</modify-attr>
<modify-attr attr-name="INITIAL_PASSWORD"><!-- content
suppressed -->
</modify-attr>
<modify-attr attr-name="STATE_PROVINCE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="STATE_PROVINCE">
<add-value>
<value type="string">MA</value>
</add-value>
</modify-attr>
<modify-attr attr-name="FULL_ADDRESS">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FULL_ADDRESS">
<add-value>
<value type="string">1234 Any Place
Cambridge, MA 06543
UNITED
STATES</value>
</add-value>
</modify-attr>
<modify-attr attr-name="GENDER">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="BIRTH_DATE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="D_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="D_NAME">
<add-value>
<value type="string">ldap://ny-idm.ansi.org/CN=LouScannon,O=Horseshoe Road Inn,dc=membership,O=ansi,dc=org</value>
</add-value>
</modify-attr>
<modify-attr attr-name="DESIGNATION">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="PK_ID">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="PK_ID">
<add-value>
<value type="string">267385</value>
</add-value>
</modify-attr>
</modify>
</input>
</nds>



Notice that in this trace, there is no reference to group Sharepoint Readers, just a <remove-all-values>. For reference purposes, this is the stylesheet which converts the comma delimited list to a list of values. The name in the application name space of the field which contained the list of group memberships is “SEC_GRP”. (Had this been IDM 3.5x or later, this would have been done in XML Script.)

 
<?xml
version="1.0" encoding="UTF-8"?><xsl:stylesheet exclude-result-prefixes="query cmd dncv" version="1.0" xmlns:cmd="http://www.novell.com/nxsl/java/com.novell.nds.dirxml.driver.XdsCommandProcessor"
xmlns:dncv="http://www.novell.com/nxsl/java/com.novell.nds.dirxml.driver.DNConverter"
xmlns:query="http://www.novell.com/nxsl/java/com.novell.nds.dirxml.driver.XdsQueryProcessor"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<!-- parameters passed in from the DirXML engine -->
<xsl:param name="srcQueryProcessor"/>
<xsl:param name="destQueryProcessor"/>
<xsl:param name="srcCommandProcessor"/>
<xsl:param name="destCommandProcessor"/>
<xsl:param name="dnConverter"/>
<xsl:param name="fromNds"/>
<!-- identity transformation template -->
<!-- in the absence of any other templates this will cause -->
<!-- the stylesheet to copy the input through unchanged to the output -->
<xsl:template match="node()|@*">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:template>
<!-- add your custom templates here -->
<xsl:template match="//attr[@attr-name='SEC_GRP']">
<xsl:copy>
<xsl:attribute name="attr-name">SEC_GRP</xsl:attribute>
<xsl:variable name="StringToTransform" select="./value/text()"/>
<xsl:call-template name="fieldstoxml">
<xsl:with-param name="StringToTransform" select="$StringToTransform"/>
<xsl:with-param name="FieldNum" select="0"/>
</xsl:call-template>
</xsl:copy>
</xsl:template>
<xsl:template match="//*[@attr-name='SEC_GRP']/add-value">
<xsl:copy>
<xsl:attribute name="attr-name">SEC_GRP</xsl:attribute>
<xsl:variable name="StringToTransform" select="./value/text()"/>
<xsl:call-template name="fieldstoxml">
<xsl:with-param name="StringToTransform" select="$StringToTransform"/>
<xsl:with-param name="FieldNum" select="0"/>
</xsl:call-template>
</xsl:copy>
</xsl:template>
<!--================================================-->
<!--
Template to convert fields to XML nodes -->
<!--================================================-->
<xsl:template name="fieldstoxml">
<xsl:param name="StringToTransform" select="''"/>
<xsl:param name="FieldNum" select="1"/>
<xsl:choose>
<!--==========================-->
<!--
string contains comma -->
<!--==========================-->
<xsl:when
test="contains($StringToTransform,',')">
<!--======================================-->
<!--
Get everything up to the first comma -->
<!--======================================-->
<value type="string">
<xsl:value-of select="substring-before($StringToTransform,',')"/>
</value>
<!--================================================-->
<!--
repeat for the remainder of the original string-->
<!--================================================-->
<xsl:call-template name="fieldstoxml">
<xsl:with-param name="StringToTransform">
<xsl:value-of select="substring-after($StringToTransform,',')"/>
</xsl:with-param>
<xsl:with-param name="FieldNum" select="$FieldNum 1"/>
</xsl:call-template>
</xsl:when>
<xsl:otherwise>
<value type="string">
<xsl:value-of select="$StringToTransform"/>
</value>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
</xsl:stylesheet>



Once it passes through this rule, the XML looks as follows:

 
<nds
dtdversion="2.0" ndsversion="8.x" xmlns:jdbc="urn:dirxml:jdbc">
<source>
<product build="20061207_1003" instance="IMIS-IDV" version="2.1.5">DirXML Driver for JDBC</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<modify class-name="dbo.Members" eventid="PK_ID=267385,table=MEMBERS,schema=DBO" jdbc:database-local-time="2008-03-26 19:34:02.0"
src-dn="PK_ID=267385,table=MEMBERS,schema=DBO">
<association>PK_ID=267385,table=MEMBERS,schema=DBO</association>
<modify-attr attr-name="LAST_FIRST">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LAST_FIRST">
<add-value>
<value type="string">SCANNON, LOU</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ADDRESS_1">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ADDRESS_1">
<add-value>
<value type="string">1234 Any Place </value>
</add-value>
</modify-attr>
<modify-attr attr-name="FULL_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FULL_NAME">
<add-value>
<value type="string">Lou Scannon </value>
</add-value>
</modify-attr>
<modify-attr attr-name="JOIN_DATE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="WEBSITE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FAX">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FAX">
<add-value>
<value type="string">(617) 555-1111 </value>
</add-value>
</modify-attr>
<modify-attr attr-name="EMAIL">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="EMAIL">
<add-value>
<value type="string">lscannon@hri.com </value>
</add-value>
</modify-attr>
<modify-attr attr-name="COMPANY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="COMPANY">
<add-value>
<value type="string">Horseshoe Road Inn </value>
</add-value>
</modify-attr>
<modify-attr attr-name="STATUS">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="STATUS">
<add-value>
<value type="string">A </value>
</add-value>
</modify-attr>
<modify-attr attr-name="WORK_PHONE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="WORK_PHONE">
<add-value>
<value type="string">(617) 555-2112 </value>
</add-value>
</modify-attr>
<modify-attr attr-name="EXCLUDE_DIRECTORY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="EXCLUDE_DIRECTORY">
<add-value>
<value type="integer">0 </value>
</add-value>
</modify-attr>
<modify-attr attr-name="PAID_THRU">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="MEMBER_STATUS">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="MEMBER_STATUS">
<add-value>
<value type="string">S </value>
</add-value>
</modify-attr>
<modify-attr attr-name="TOLL_FREE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="TOLL_FREE">
<add-value>
<value type="string">TEST COMPANY </value>
</add-value>
</modify-attr>
<modify-attr attr-name="ZIP">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ZIP">
<add-value>
<value type="string">06543 </value>
</add-value>
</modify-attr>
<modify-attr attr-name="CATEGORY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="CATEGORY">
<add-value>
<value type="string">S </value>
</add-value>
</modify-attr>
<modify-attr attr-name="MEMBER_STATUS_DATE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="MEMBER_STATUS_DATE">
<add-value>
<value type="time">20060918000000000000000 </value>
</add-value>
</modify-attr>
<modify-attr attr-name="CO_ID">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="CO_ID">
<add-value>
<value type="string">C267384 </value>
</add-value>
</modify-attr>
<modify-attr attr-name="OLD_LOGIN">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="OLD_LOGIN">
<add-value>
<value type="string">Lou Scannon </value>
</add-value>
</modify-attr>
<modify-attr attr-name="PREFIX">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LOGIN_DISABLED">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LOGIN_DISABLED">
<add-value>
<value type="state">false </value>
</add-value>
</modify-attr>
<modify-attr attr-name="DATE_ADDED">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="DATE_ADDED">
<add-value>
<value type="time">20060918000000000000000 </value>
</add-value>
</modify-attr>
<modify-attr attr-name="COUNTRY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="COUNTRY">
<add-value>
<value type="string">UNITED STATES </value>
</add-value>
</modify-attr>
<modify-attr attr-name="MEMBER_TYPE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="MEMBER_TYPE">
<add-value>
<value type="string">CMC </value>
</add-value>
</modify-attr>
<modify-attr attr-name="IDM">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="IDM">
<add-value>
<value type="integer">1 </value>
</add-value>
</modify-attr>
<modify-attr attr-name="EXCLUDE_MAIL">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="EXCLUDE_MAIL">
<add-value>
<value type="integer">0 </value>
</add-value>
</modify-attr>
<modify-attr attr-name="HOME_PHONE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="WEB_LOGIN">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="WEB_LOGIN">
<add-value>
<value type="string">LScannon </value>
</add-value>
</modify-attr>
<modify-attr attr-name="SEC_GRP">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="SEC_GRP">
<add-value attr-name="SEC_GRP">
<value type="string">All ISOT Users </value>
<value type="string">Whats New </value>
</add-value>
</modify-attr>
<modify-attr attr-name="COUNTY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LAST_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="LAST_NAME">
<add-value>
<value type="string">Scannon </value>
</add-value>
</modify-attr>
<modify-attr attr-name="MIDDLE_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FIRST_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FIRST_NAME">
<add-value>
<value type="string">Lou </value>
</add-value>
</modify-attr>
<modify-attr attr-name="CITY">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="CITY">
<add-value>
<value type="string">Cambridge </value>
</add-value>
</modify-attr>
<modify-attr attr-name="CO_MEMBER_TYPE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="CO_MEMBER_TYPE">
<add-value>
<value type="string">FMCM </value>
</add-value>
</modify-attr>
<modify-attr attr-name="INITIAL_PASSWORD"><!—content suppressed -->
</modify-attr>
<modify-attr attr-name="STATE_PROVINCE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="STATE_PROVINCE">
<add-value>
<value type="string">MA </value>
</add-value>
</modify-attr>
<modify-attr attr-name="FULL_ADDRESS">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="FULL_ADDRESS">
<add-value>
<value type="string">1234 Any Place Cambridge, MA 06543 UNITED STATES </value>
</add-value>
</modify-attr>
<modify-attr attr-name="GENDER">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="BIRTH_DATE">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="D_NAME">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="D_NAME">
<add-value>
<value type="string">ldap://ny-idm.ansi.org/CN=Lou Scannon,O=Horseshoe Road Inn,dc=membership,O=ansi,dc=org </value>
</add-value>
</modify-attr>
<modify-attr attr-name="DESIGNATION">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="PK_ID">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="PK_ID">
<add-value>
<value type="string">267385 </value>
</add-value>
</modify-attr>
<operation-data COMPANY-ID="267384" INPUT-ASSOCIATION="PK_ID=267385,table=MEMBERS,schema=DBO"
INPUT-OPERATION="modify" MEMBER-ID="267385"/>
<modify-attr attr-name="ADDRESS_2">
<remove-all-values/>
</modify-attr>
</modify>
</input>
</nds>



The next rule would also have been written differently had IDM 3.5 been available at the time. I would have used a do-query rather than invoking the search method of the query object. It effectively does the same thing but I always prefer to use something built for purpose rather than rolling my own.

This query is necessary because the groups are in a container structure, which is a necessity because of the number of groups involved here and how they need to be organized within Active Directory. This is again provided for reference, it is not really the subject of this particular article.

 
<?xml
version="1.0" encoding="UTF-8"?><policy
xmlns:ANSI="http://www.novell.com/nxsl/java/com.rareprogeny.ansiIDM"
xmlns:date="http://www.novell.com/nxsl/java/java.util.date"
xmlns:query="http://www.novell.com/nxsl/java/com.novell.nds.dirxml.driver.XdsQueryProcessor">
<rule>
<description>[ANSI]
Query DNs</description>
<conditions>
<and>
<if-class-name op="equal">User</if-class-name>
<if-op-attr name="Group Membership" op="available"/>
</and>
</conditions>
<actions>
<do-set-local-variable name="GROUP-LIST">
<arg-node-set>
<token-op-attr name="Group Membership"/>
</arg-node-set>
</do-set-local-variable>
<do-strip-op-attr name="Group Membership"/>
<do-clear-dest-attr-value disabled="true" name="Group Membership"/>
<do-for-each>
<arg-node-set>
<token-local-variable name="GROUP-LIST"/>
</arg-node-set>
<arg-actions>
<do-set-local-variable name="SEARCH-VALUE">
<arg-string>
<token-local-variable name="current-node"/>
</arg-string>
</do-set-local-variable>
<do-set-local-variable name="DEST-DN">
<arg-string>
<token-text xml:space="preserve">org\ansi\Groups\Security</token-text>
</arg-string>
</do-set-local-variable>
<do-set-local-variable name="QUERY-RESULT">
<arg-node-set>
<token-xpath expression="query:search($destQueryProcessor,'subtree','',$DEST-DN,'','CN',$SEARCH-VALUE,'')"/>
</arg-node-set>
</do-set-local-variable>
<do-add-dest-attr-value class-name="User" name="Group Membership">
<arg-value type="dn">
<token-parse-dn src-dn-format="dest-dn" start="1">
<token-xpath expression="$QUERY-RESULT/@src-dn"/>
</token-parse-dn>
</arg-value>
</do-add-dest-attr-value>
</arg-actions>
</do-for-each>
<do-set-op-property name="GROUP-MEMBERSHIP-FORMAT">
<arg-string>
<token-text xml:space="preserve" xmlns:xml="http://www.w3.org/XML/1998/namespace">DN</token-text>
</arg-string>
</do-set-op-property>
</actions>
</rule>
</policy>



After this rule, the XML looks as follows:

 
<nds dtdversion="2.0" ndsversion="8.x" xmlns:jdbc="urn:dirxml:jdbc">
<source>
<product build="20061207_1003" instance="IMIS-IDV" version="2.1.5">DirXML Driver for JDBC</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<modify class-name="User" event-id="PK_ID=267385,table=MEMBERS,schema=DBO" jdbc:database-local-time="2008-03-26 19:34:02.0" src-dn="PK_ID=267385,table=MEMBERS,schema=DBO">
<association>PK_ID=267385,table=MEMBERS,schema=DBO</association>
<modify-attr attr-name="displayName">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="displayName">
<add-value>
<value type="string">SCANNON, LOU</value>
</add-value>
</modify-attr>
<modify-attr attr-name="SA">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="SA">
<add-value>
<value type="string">1234 Any Place</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Full Name">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Full Name">
<add-value>
<value type="string">Lou Scannon</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiJoinDate">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiURL">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Facsimile Telephone Number">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Facsimile Telephone Number">
<add-value>
<value type="string">(617) 555-1111</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Internet EMail Address">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Internet EMail Address">
<add-value>
<value type="string">lscannon@hri.com</value>
</add-value>
</modify-attr>
<modify-attr attr-name="company">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="company">
<add-value>
<value type="string">Horseshoe Road Inn</value>
</add-value>
</modify-attr>
<modify-attr attr-name="employeeStatus">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="employeeStatus">
<add-value>
<value type="string">A</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Telephone Number">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Telephone Number">
<add-value>
<value type="string">(617) 555-2112</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiExcludeDirectory">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiExcludeDirectory">
<add-value>
<value type="integer">0</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiPaidThru">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiMemberStatus">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiMemberStatus">
<add-value>
<value type="string">S</value>
</add-value>
</modify-attr>
<modify-attr attr-name="tollFreePhoneNumber">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="tollFreePhoneNumber">
<add-value>
<value type="string">TEST COMPANY</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Postal Code">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Postal Code">
<add-value>
<value type="string">06543</value>
</add-value>
</modify-attr>
<modify-attr attr-name="businessCategory">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="businessCategory">
<add-value>
<value type="string">S</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiMemberStatusEffectiveDate">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiMemberStatusEffectiveDate">
<add-value>
<value type="time">20060918000000000000000</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiCompanyID">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiCompanyID">
<add-value>
<value type="string">C267384</value>
</add-value>
</modify-attr>
<modify-attr attr-name="DirXML-ADAliasName">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="DirXML-ADAliasName">
<add-value>
<value type="string">Lou Scannon</value>
</add-value>
</modify-attr>
<modify-attr attr-name="personalTitle">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Login Disabled">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Login Disabled">
<add-value>
<value type="state">false</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiDateAdded">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiDateAdded">
<add-value>
<value type="time">20060918000000000000000</value>
</add-value>
</modify-attr>
<modify-attr attr-name="co">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="co">
<add-value>
<value type="string">UNITED STATES</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiMemberType">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiMemberType">
<add-value>
<value type="string">CMC</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiIDMFlag">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiIDMFlag">
<add-value>
<value type="integer">1</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiExcludeMail">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiExcludeMail">
<add-value>
<value type="integer">0</value>
</add-value>
</modify-attr>
<modify-attr attr-name="homePhone">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiUserPrincipalName">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiUserPrincipalName">
<add-value>
<value type="string">LScannon</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiCounty">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Surname">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Surname">
<add-value>
<value type="string">Scannon</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Initials">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Given Name">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Given Name">
<add-value>
<value type="string">Lou</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Physical Delivery Office Name">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Physical Delivery Office Name">
<add-value>
<value type="string">Cambridge</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiCompanyType">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiCompanyType">
<add-value>
<value type="string">FMCM</value>
</add-value>
</modify-attr>
<modify-attr attr-name="nspmDistributionPassword"><!--
content suppressed -->
</modify-attr>
<modify-attr attr-name="S">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="S">
<add-value>
<value type="string">MA</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Postal Address">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Postal Address">
<add-value>
<value type="string">1234 Any Place
Cambridge, MA 06543
UNITED
STATES</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiGender">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiBirthDate">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiLDAPURL">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiLDAPURL">
<add-value>
<value type="string">ldap://ny-idm.ansi.org/CN=Lou
Scannon,O=Horseshoe Road Inn,dc=membership,O=ansi,dc=org</value>
</add-value>
</modify-attr>
<modify-attr attr-name="ansiDesignation">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiMemberID">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiMemberID">
<add-value>
<value type="string">267385</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Postal Office Box">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="Group Membership">
<add-value>
<value type="dn">org\ansi\Groups\Security\ISOT Users\All ISOTUsers</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Group Membership">
<add-value>
<value type="dn">org\ansi\Groups\Security\WhatsNew</value>
</add-value>
</modify-attr>
<operation-data COMPANY-ID="267384" GROUP-MEMBERSHIP-FORMAT="DN"
INPUT-ASSOCIATION="PK_ID=267385,table=MEMBERS,schema=DBO"
INPUT-OPERATION="modify" MEMBER-ID="267385"/>
</modify>
</input>
</nds>



The next transformation of the data happens in the optimize modify. The output of that process looks as follows:

 
[03/26/08
18:35:11.054]: IMIS-IDV PT: Optimize Modify returned:
[03/26/08
18:35:11.054]: IMIS-IDV PT:
<nds
dtdversion="2.0" ndsversion="8.x"
xmlns:jdbc="urn:dirxml:jdbc">
<source>
<product build="20061207_1003" instance="IMIS-IDV"
version="2.1.5">DirXML Driver for JDBC</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<modify class-name="User"
dest-dn="org\ansi\membership\Horseshoe Road Inn\Lou Scannon"
dest-entry-id="81687" event-id="PK_ID=267385,table=MEMBERS,schema=DBO"
jdbc:database-local-time="2008-03-26 19:34:02.0"
src-dn="PK_ID=267385,table=MEMBERS,schema=DBO">
<association>PK_ID=267385,table=MEMBERS,schema=DBO</association>
<modify-attr attr-name="businessCategory">
<remove-value>
<value timestamp="1161813520#14"
type="string">C</value>
</remove-value>
</modify-attr>
<modify-attr attr-name="Postal Address">
<add-value>
<value type="string">1234 Any Place
Cambridge, MA 06543
UNITED
STATES</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Group Membership">
<add-value>
<value type="dn">org\ansi\Groups\Security\ISOT Users\All ISOT
Users</value>
<value type="dn">org\ansi\Groups\Security\Whats
New</value>
</add-value>
</modify-attr>
<modify-attr attr-name="businessCategory">
<add-value>
<value type="string">S</value>
</add-value>
</modify-attr>
<operation-data COMPANY-ID="267384"
GROUP-MEMBERSHIP-FORMAT="DN"
INPUT-ASSOCIATION="PK_ID=267385,table=MEMBERS,schema=DBO"
INPUT-OPERATION="modify" MEMBER-ID="267385"/>
<modify-attr attr-name="nspmDistributionPassword"><!--
content suppressed -->
</modify-attr>
<modify-attr attr-name="ansiLDAPURL">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiLDAPURL">
<add-value>
<value type="string">ldap://ny-idm.ansi.org/CN=Lou
Scannon,O=Horseshoe Road Inn,dc=membership,O=ansi,dc=org</value>
</add-value>
</modify-attr>
</modify>
</input>
</nds>



As you can see, we have consolidated the separate adds into one add, but we still don’t know what data to remove. We could insert the remove-all-values which would work but would be less than optimal. Because the AD driver is so inefficient, a user that is a member of many groups could (and will) get into problems if the two AD drivers are both attempting to update eDirectory simultaneously, removing all group memberships and re-adding them.

Optimizing the Modify

The real topic of this Cool Solution is embodied in the following rule. It lives in the command transform policy set of the publisher channel. Its job is to query the destination (the identity vault) and to optimize the modify to include only the changes required.

The first action sets a node-set variable NEW-GROUP-LIST to the list of groups coming from the source system. This is how the group list should look when we are done. Previously, it would have been preceded by a <remove-all-values> but this is what we are trying to avoid here. Instead, we insert a <remove-value>, with the intent that we will later add the values to be removed.

The next action is a for-each loop that loops on the contents of the destination attribute Group Membership. This loops through all the current values of the Group Membership list on the user.

Because the list of DNs in the operation attribute contains relative DNs (they do not include the root of the tree), but the list of DN’s from the query contains full DNs (they do include the tree name), there is a parse DN verb used to strip off the tree name.

There are two actions in the loop. First, each DN read from the destination, and the value is added under a <value> under the <remove-value>. This may seem counter-intuitive, but the explanation will be clear later. Second, for each attribute it finds in the destination, it strips out the <add-value>. This removes any adds for values that already exist in the destination.

Although I don’t have a real trace of this moment in time, the XML should look something like this:

 
<modify-attr attr-name="Group Membership">
<remove-value>
<value type="dn">org\ansi\Groups\Security\ISOT Users\All ISOTUsers</value>
<value type="dn">org\ansi\Groups\Security\WhatsNew</value>
<value type="dn">org\ansi\Groups\Security\Sharepoint Readers</value>
</remove-value>
<add-value>
<value type="dn">org\ansi\Groups\Security\ISOT Users\All ISOTUsers</value>
<value type="dn">org\ansi\Groups\Security\WhatsNew</value>
</add-value>



The next part of the rule cleans this up. This is where the counter-intuitive part should become clearer. A second for-each loop loops through the original operation attribute value, which contains what the final value should end up being. This loop strips off the remove-value/value tags for the group members that should remain. The result of this is that the values that are in the destination but not in the source end up being left in the remove-value section. Also, the add-value only has the values that are in the source but not in the destination. The final XML looks as follows:

 
<nds
dtdversion="2.0" ndsversion="8.x"
xmlns:jdbc="urn:dirxml:jdbc">
<source>
<product build="20061207_1003" instance="IMIS-IDV"
version="2.1.5">DirXML Driver for JDBC</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<modify class-name="User" dest-dn="org\ansi\membership\Horseshoe Road Inn\Lou Scannon" dest-entry-id="81687" event-id="PK_ID=267385,table=MEMBERS,schema=DBO"
jdbc:database-local-time="2008-03-26 19:34:02.0" src-dn="PK_ID=267385,table=MEMBERS,schema=DBO">
<association>PK_ID=267385,table=MEMBERS,schema=DBO</association>
<modify-attr attr-name="businessCategory">
<remove-value>
<value timestamp="1161813520#14" type="string">C</value>
</remove-value>
</modify-attr>
<modify-attr attr-name="Postal Address">
<add-value>
<value type="structured">
<component name="string">1234 Any Place</component>
<component name="string">Cambridge, MA 06543</component>
<component name="string">UNITED STATES</component>
<component name="string"/>
<component name="string"/>
<component name="string"/>
</value>
</add-value>
</modify-attr>
<modify-attr attr-name="Group Membership">
<add-value>
<value type="dn">org\ansi\Groups\Security\ISOT Users\All ISOTUsers</value>
<value type="dn">org\ansi\Groups\Security\WhatsNew</value>
</add-value>
<remove-value>
<value>org\ansi\Groups\Security\SharePointReaders</value>
</remove-value>
</modify-attr>
<modify-attr attr-name="businessCategory">
<add-value>
<value type="string">S</value>
</add-value>
</modify-attr>
<modify-attr attr-name="nspmDistributionPassword"><!-- content suppressed -->
</modify-attr>
<modify-attr attr-name="ansiLDAPURL">
<remove-all-values/>
</modify-attr>
<modify-attr attr-name="ansiLDAPURL">
<add-value>
<value type="string">ldap://ny-idm.ansi.org/CN=Lou Scannon,O=Horseshoe Road Inn,dc=membership,O=ansi,dc=org</value>
</add-value>
</modify-attr>
<operation-data COMPANY-ID="267384"
GROUP-MEMBERSHIP-FORMAT="DN"
INPUT-ASSOCIATION="PK_ID=267385,table=MEMBERS,schema=DBO"
INPUT-OPERATION="modify" MEMBER-ID="267385"/>
<modify-attr attr-name="Postal Address">
<remove-all-values/>
</modify-attr>
</modify>
</input>
</nds>



The code that makes this all happen is in this rule:

 
<?xml version="1.0"
encoding="UTF-8"?><policy>
<rule>
<description>[ANSI] Add removes to
optimized modifies</description>
<conditions>
<and>
<if-operation op="equal">modify</if-operation>
<if-op-attr name="Group Membership" op="available"/>
<if-dest-attr name="Group Membership" op="available"/>
</and>
</conditions>
<actions>
<do-set-local-variable name="NEW-GROUP-LIST">
<arg-node-set>
<token-op-attr name="Group Membership"/>
</arg-node-set>
</do-set-local-variable>
<do-append-xml-element expression="modify-attr[@attr-name='Group Membership']" name="remove-value"/>
<do-for-each>
<arg-node-set>
<token-dest-attr name="Group Membership"/>
</arg-node-set>
<arg-actions>
<do-set-local-variable name="DN">
<arg-string>
<token-parse-dn src-dn-format="dest-dn" start="1">
<token-local-variable name="current-node"/>
</token-parse-dn>
</arg-string>
</do-set-local-variable>
<do-trace-message level="3">
<arg-string>
<token-text xml:space="preserve">Adding a remove of value </token-text>
<token-local-variable name="current-node"/>
</arg-string>
</do-trace-message>
<do-append-xml-element expression="modify-attr[@attr-name='Group Membership']/remove-value" name="value"/>
<do-append-xml-text expression="modify-attr[@attr-name='Group Membership']/remove-value/value[last()]">
<arg-string>
<token-local-variable name="DN"/>
</arg-string>
</do-append-xml-text>
<do-strip-xpath expression="modify-attr[@attr-name='Group Membership']/add-value/value[text()=$DN]"/>
</arg-actions>
</do-for-each>
<do-for-each>
<arg-node-set>
<token-local-variable name="NEW-GROUP-LIST"/>
</arg-node-set>
<arg-actions>
<do-trace-message level="3">
<arg-string>
<token-text xml:space="preserve">Stripping the remove of </token-text>
<token-local-variable name="current-node"/>
</arg-string>
</do-trace-message>
<do-strip-xpath expression="modify-attr[@attr-name='Group
Membership']/remove-value/value[text()=$current-node]"/>
</arg-actions>
</do-for-each>
<do-strip-xpath expression="modify-attr[@attr-name='Group
Membership']/remove-value/value[not(text())]"/>
<do-strip-xpath expression="modify-attr[@attr-name='Group Membership']/add-value/value[not(text())]"/>
</actions>
</rule>
</policy>



This approach could be applied in many cases where a multi-valued attribute is coming from a source where the old values are not needed. Whether this would be necessary, or if a simple remove-all-values approach will do, has to do with the downstream drivers – whether they support loopback, and whether they could be affected by all those additional changes.
Comment List
Related
Recommended