Problem

I ran into the following problem at a client site. They used a sample C# (C-Sharp) .NET application to change users passwords from a web page. All looks like it worked well, but while they could log in to the eDirectory tree via iManager or via LDAP, it did not synchronize over IDM to Active Directory.

For a variety of reasons they are running IDM 3.01 with SS204 on the two IDM servers, but the LDAP server they were pointing the web application at was running SS2.01.

As far as we could tell, with SS2.01 installed, the password change via an LDAP modify of the userPassword attribute (Replace the value, after binding as an administrative user) only changed the NDS password.

Thus in the Dstrace logs, we never saw a modify of the nspmDistributionPassword. Thus no passwords were sycned from that change event.

Solution

This was happening because Password Sync 1.0, which came with the DirXML 1.1a era, "cheated" and fudged password sync into something that worked. For the eDir to eDir driver, it literally synced the RSA public/private key pairs. For AD, it somehow under the covers caught password changes before they made it to the RSA key pair and sent them on.

With Identity Manager and higher (2.x and up), we got access to Password Sync 2.0, which uses Universal Password to store the password in a retrievable fashion (3DES, aka AES encrypted) and that
attribute can cause an event when changed.

The attribute we see changing is nspmDistributionPassword. The rule sets in the Subscriber and Publisher Channel Command Transforms for pretty much all drivers that touch passwords convert the modify of this attribute to a modify-password event or convert the modify-password event coming in, into a modify of this attribute. Read the article at: http://www.novell.com/coolsolutions/appnote/19304.html
for an example of how the rule set works, step by step walking through a trace.

Although we did not test it, something interesting might have happened that would have made troubleshooting even more difficult! If we had tested a Client32 login for the user whose password we changed, after changing just the NDS password, odds are pretty good that the login event itself would have caused a modify of the Universal Password for the user.

When a user has a UP policy applied and the NDS and UP password are out of sync, Client32 with NMAS is smart enough to try the NDS password first. If the supplied password works, it checks the
timestamps of the UP and RSA passwords and updates the one that is out of sync. If the NDS password fails to authenticate, then it tries the same password against the UP password.

To troubleshoot this problem, we used Dstrace on the server that was being contacted, and enabled the LDAP and NMAS flags. (This could be in Dstrace.nlm, dstrace.dlm (on Windows), ndstrace (on Unixen), or iMonitor's web based DSTrace.)

The tricky part was it looked the same, regardless of whether we changed RSA or UP passwords. But we were able to see LDAP do the modify, NMAS recognize the password change, and then with the DirXML
trace on, see (or not see) an event for that user on nspmDistributionPassword.

Now we are off to the traces! (Hey, puns can be funny!)

First we have a LDAP password change on an SS2.01 server.

11:18:28 7D8921E0 LDAP: DoBind on connection 0x7cf27460
11:18:28 7D8921E0 LDAP: Bind name:cn=admin,ou=applications,o=acme,
version:3, authentication:simple
11:18:28 7D8921E0 LDAP: Sending operation result 0:"":"" to connection
0x7cf27460
11:18:28 7D8921E0 LDAP: Operation 0x12:0x60 on connection 0x7cf27460
completed in 0 seconds
11:18:28 7D8921E0 LDAP: DoModify on connection 0x7cf27460
11:18:28 7D8921E0 LDAP: modify: dn (cn=adtest2,ou=PEOPLE,o=ACME)
11:18:28 7D8921E0 LDAP:  replace: userPassword
11:18:28 7D8921E0 LDAP: Sending operation result 0:"":"" to connection
0x7cf27460
11:18:28 7D8921E0 LDAP: Operation 0x13:0x66 on connection 0x7cf27460
completed in 0 seconds
11:18:28 7D8921E0 LDAP: DoUnbind on connection 0x7cf27460

We have an LDAP bind as admin.applications.acme user, who is an admin who can reset passwords.

11:18:28 7D8921E0 LDAP: DoBind on connection 0x7cf27460
11:18:28 7D8921E0 LDAP: Bind name:cn=admin,ou=applications,o=acme,
version:3, authentication:simple

Then there is a DoModify for the password. To do an adminstrative password change, you REPLACE the value. As a user changing your own via LDAP, you need to delete the value and add the new value in one operation. This is different from our case.

11:18:28 7D8921E0 LDAP: DoModify on connection 0x7cf27460
11:18:28 7D8921E0 LDAP: modify: dn (cn=adtest2,ou=PEOPLE,o=ACME)
11:18:28 7D8921E0 LDAP:  replace: userPassword
11:18:28 7D8921E0 LDAP: Sending operation result 0:"":"" to connection
0x7cf27460

This ended up just changing the NDS/RSA password, even though the user has a UP policy assigned. Not good!

Next, we have the same exact operation on the server with SS2.04.

11:04:04 95ACA1C0 LDAP: DoBind on connection 0x99e70700
11:04:04 95ACA1C0 LDAP: Bind name:cn=admin,o=acme, version:3,
authentication:simple
11:04:04 95ACA1C0 LDAP: Sending operation result 0:"":"" to connection
0x99e70700
11:04:04 95ACA1C0 LDAP: Operation 0x26:0x60 on connection 0x99e70700
completed in 0 seconds
11:04:04 95ACA1C0 LDAP: DoModify on connection 0x99e70700
11:04:04 95ACA1C0 LDAP: modify: dn (cn=adtest2,ou=PEOPLE,o=ACME)
11:04:04 95ACA1C0 NMAS: Successful set password for CN=adtest2.ou=PEOPLE.O=ACME
11:04:04 95ACA1C0 LDAP:  replace: userPassword
11:04:04 95ACA1C0 LDAP: Sending operation result 0:"":"" to connection
0x99e70700
11:04:04 95ACA1C0 LDAP: Operation 0x27:0x66 on connection 0x99e70700
completed in 0 seconds
11:04:04 95ACA1C0 LDAP: DoUnbind on connection 0x99e70700
11:04:04 99AF3160 NMAS: Successful get distribution password for
adtest2.PEOPLE.ACME

Same bind as an admin user, same DoModify. What is key is the following line:

11:04:04 95ACA1C0 NMAS: Successful set password for CN=adtest2.ou=PEOPLE.O=ACME

Note that the tag came from NMAS, not LDAP. That was the hint that it really set UP.

The last line:

11:04:04 99AF3160 NMAS: Successful get distribution password for adtest2.PEOPLE.ACME

hints to us that IDM just synchronized the password. Everytime IDM queries the nspmDistributionPassword for an event, an NMAS trace event would show up (if the trace was enabled).

Here is the event from the DXML DVRS trace.

<nds dtdversion="3.0" ndsversion="8.x">
<source>
  <product version="3.0.10.20060630 ">DirXML</product>
  <contact>Novell, Inc.</contact>
</source>
<input>
  <modify class-name="User" event-id="driver#Publisher#126309"
qualified-src-dn="O=ACME\OU=PEOPLE\CN=adtest2"
src-dn="\LDAP-TREE\ACME\PEOPLE\adtest2" src-entry-id="106498"
timestamp="1196971303#118">
    <modify-attr attr-name="nspmDistributionPassword">
    </modify-attr>
  </modify>
</input>
</nds>

The lesson to take out of this is that you need to run Security Services at a reasonably up-to-date revision. The word of mouth I have heard is that SS2.03 is the bare minimum you can survive on, and that 2.05 is the best. But be at least at 2.03 across the board.

In this case there are some internal reasons not to upgrade, which would have fixed the problem as well.