In Identity Manager 4.5 one of the hidden, unsung new bits is a tool called the Password Sync Troubleshooter. The guys in NTS (Novell Technical Support) tell me that they have been using this tool for years when they have to support an issue with Password Sync. They very much wanted to get it out to the rest of the community and finally in IDM 4.5 it became part of the product.

As you can imagine, if NTS has been using and developing this tool, it covers a large number of common use cases where Password Sync has an issue, and can identify them. Hopefully it also has a large number of edge cases, but alas there is no change log I can find to see what has been added to this tool over the years. Alas, and alack I say! I am a bit selfish about wanting change logs, which not everyone agrees with.

I am not entirely sure where it installs too, since I usually just run the copy I grabbed from the install DVD. Thankfully, the Linux and Windows DVD include it. (Some of the components for Windows are not included on the Linux DVD, since the theory is you would install the Windows'y bits via the Windows DVD and the Linux'y bits via the Linux DVD. I do wish a simple Remote Loader DVD was available, so I did not have to download and mover around a 4GB file, just to install the Remote Loader on my Windows boxes.)

The path on the install DVD, from the Linux DVD is:
\products\IDM\linux\setup\utilities\PassSyncTroubleshootingTool

And yes, I know, the path delimiters are backwards, and should all be forward slashes on Linux, but you guys are big boys and girls and probably would have figured that out on your own. I am being lazy and generated that path on my Windows desktop from extracting the DVD.

And on Windows it is:
\products\IDM\windows\setup\utilities\PassSyncTroubleshootingTool

There is a x86 and x64 directory with different bit depth versions of the tool. Choose wisely.

The user interface is simple. What happens next is not.



There are really only 5 options.

Trace File: This is a smidgen misleading. In the context of IDM, Trace file usually means the engine or remote loader trace file. However in this case they mean where to write the output from this tool. It took me a while to figure this out, since I expected it to want to parse a Remote Loader or engine trace file. Thus I waste the time so you don't have too.

Domain name is meant to be the Active Directory domain name that you are working with. Remember that in the current Active Directory driver each driver can only talk to a single domain.

At BrainShare they talked about a new Multi Domain Active Directory (MDAD) driver (I personally liked the name at BrainShare of AD^2 (Active Directory Advanced Driver better, but I am not the marketing guy) and have been running a beta of it since then. This new driver will be released out of band of IDM, so no need to wait for 4.6 or the like to get it. They have agreed to do an IDM User Group session on the topic. I run an IDM User Group on the side (wearing yet another hat. Not work, not CS, not forums, but now IDM UG hat) that tries to get people to present on interesting IDM topics. Mostly because I want to learn new stuff and where do I go to learn new things? I use you people to encourage them to talk about new stuff. It is for you guys, really, ok, I am just being selfish, but you are all welcome to tag along). Send me an email if you are interested in joining the IDM User Group.

The new MDAD driver will talk to an entire forest, or multiple forests and do password sync from them all. At which point of course this tool will need a serious re-write. But that is a good thing, I think.

There is an Advanced option on the Check Driver Machine option, and if you are like me, you always tick Advanced options. (Remember when in iMonitor, if you click the Novell iMonitor logo in the upper left an About screen, and there is an option to "Enable Advanced option" but it has a warning "Possibly dangerous". The first time I saw that, I of course immediately clicked it. I mean it was like waving a red flag in front of a bull. How could I not?

Finally, there is a choice of Check Driver Machine and Check Domain Controllers. This is important since the password synchronization functionality really has two major components. There is the Remote Loader server running the driver and collecting passwords. The second is the set of Domain Controllers (all of which needs the Password Filter) that need to be able to communicate back to the Remote Loader server.

If you have followed the various troubleshooting TID's on password sync, you will know that each DC collects all the passwords in a local registry cache, and when it is communicating, sends them on to the Remote Loader. The Remote Loader collects the passwords in its (different path) registry cache and that is the list it generates Publisher channel events to send to the engine. If the Remote Loader happens to be running on a Domain Controller it then has two registry caches, the first for the PWfilter collected passwords, and then the Remote Loader cache.

This division of options allows you to focus on the two major components, one at a time.

I am going to leave the entire trace history at the bottom of this article intact, and then cut and paste the interesting bits for commentary and discussion. This way you can see the important things to note, but also see the complete flow since it is instructive and helps understand what is happening.

Let's begin shall we, with a Check Driver Machine option first.

	Tue Jun 16 11:22:38 2015 : Starting Checks on Driver Machine ..... 

	Tue Jun 16 11:23:10 2015: Logging as admin user.

You will get asked to login, if your current account does not have sufficient permissions, which is a nice touch. It logs the user used so you can decide if maybe an account with more permissions is needed.

	Tue Jun 16 11:23:10 2015 : 
	The List of all Domain Controllers -
	1. ADDEV2.dev.acme.edu
	2. ADDEV1.dev.acme.edu

Although we are not checking Domain Controllers in this test, it is important to be sure the driver machine can resolve the domain controllers. If you had 5 DC's and this only returned 2 as above, right off the bat you would know you have a problem in DNS.

	Tue Jun 16 11:23:10 2015 : RPC Service is running 

This is very important, and a nice edge case. You could potentially have disabled the RPC Service, in which case, none of this is going to work. The Remote Loader and PWfilter service all communicate over Windows RPC. There is a secondary problem that you might have fire walled the Domain Controllers to only allow RPC between DC's (as happened at a client) in which case you would need to add an exception for the Remote Loader machine as well.

	Tue Jun 16 11:23:10 2015 : Full DNS name of the driver machine is IDVAULTDEV2.dev.acme.edu 

This seems silly, but Active Directory is very dependent on DNS and if there is a DNS issue, there are going to be problems.

	Tue Jun 16 11:23:10 2015 : The version of the Operating System is : Microsoft  Service Pack 1 (build 7601)

I have no idea what version that corresponds to, but I think it was 2008 R2.

	Tue Jun 16 11:23:10 2015 : An AD driver instance is found configured on Remote Loader
	Tue Jun 16 11:23:10 2015 : AD Driver which is configured with Connection port 8092 and Command port 8003 is running

This identifies that we are running in a Remote Loader, not the engine, and the ports in use. Now as it happens, on this Windows server, we were running 5 instances of the Remote Loader (4 for Active Directory, and a fifth for Delimited Text.

	Tue Jun 16 11:23:10 2015 : List of local files related to Driver are :
		C:\Novell\remoteloader\64bit\ADDriver.dll
		C:\Novell\remoteloader\64bit\ActiveDirectoryEnterprise-Config.txt
		C:\Novell\drivertrace\ActiveDirectoryEnterprise-Trace.log

This nicely identifies files you would care about. The DLL itself, the Config file, and the Trace file. You could look at the config file and figure this out yourself, but in a case of multiple Remote Loader instances this is much more helpful. Now I can associate the specific ports with a file, without having to go looking.

	Tue Jun 16 11:23:14 2015 : Driver version is "4.0.1.0">AD</pr"20150311_120000"AAi8P757zh70a80
	Tue Jun 16 11:23:14 2015 : Driver version is "4.0.1.0">AD</pr"20150311_120000"AAi8P and Build ID is "20150311_120000"AAi8P757zh70a80

The versioning is nice. No clue what the "AAi8P757zh70a80" is supposed to mean. Bet it decodes to something funny.

	Tue Jun 16 11:23:21 2015 : The 'Driver Machine' value in the registry key[SOFTWARE\NOVELL\PASSSYNC] is : 1.

This is an important test. The key in question determines that when you run the Password Sync Control Panel applet whether it displays the full Domain level interface, or the just a DC interface. The good news is, you can just switch the value if it is wrong and run it again.

	Tue Jun 16 11:23:21 2015 : The 'Domains' value in registry key[SOFTWARE\NOVELL\PASSSYNC\DATA] is dev.acme.edu

Only the domain the local machine is a member of can collect passwords, so in theory you might have this set wrong, so good to confirm the domain is correct. Especially in a case like mine, with 4 AD Drivers and three domains on one box. In this configuration only the first AD Driver instance that starts can successfully sync passwords, and it has to be the domain this is member server (or DC) belongs to. (Oops, dangling participle that I cannot see a way to avoid).

	Tue Jun 16 11:23:21 2015 : Number of subkeys(passwords cached) under the key[SOFTWARE\NOVELL\PASSSYNC\DATA\dev.acme.edu]is 1 

	Tue Jun 16 11:23:21 2015 : The information of the user passwords cached:
		User: BSMITH
		Timestamp of password change:  14:03:32.57 March 13 2015 

This is very helpful, since in my specific case, password sync was not working, and this showed that the Driver Machine only had a single password cached. There should have been dozens. So now I knew that something was up in communication between the DC's and the Remote Loader server.

	Tue Jun 16 11:23:21 2015 : Tests on this driver machine are done 

	Press any key to close this trace ... 

And that was about it for Driver Machine. I learned that most things were working ok, but that clearly the passwords were not properly being sent from the DC's to the Remote Loader machine.

Seems like the next question to look into is, what do the Domain Controllers think is going on.

Here is the Domain Controller check log.

	Tue Jun 16 11:23:31 2015 : Starting Checks on All DCs ..... 

	Tue Jun 16 11:23:31 2015: Logging as admin user.

Again, tracks who we logged in as. Turns out I started out doing this as the wrong user the first few times, so this let me pick the right example from the various runs I took.

	Tue Jun 16 11:23:31 2015 : 
	The List of all Domain Controllers -
	1. ADDEV2.dev.acme.edu
	2. ADDEV1.dev.acme.edu

List of DC's is key as above. Make sure they are all really there, with proper DNS names.

	Tue Jun 16 11:23:31 2015 : Checking the Domain Controller ADDEV2.dev.acme.edu ....

Now it will loop through all the DC's in the list, and it calls them out by name, so you know which one it is testing each loop. I will skip the second loop for the second DC in my example.

	Running Basic Diagnostic Checks.

	Password filter files installed on this DC are C:\Windows\System32\PWFILTER.DLL and C:\Windows\System32\PSEVENT.DLL

So it finds the files properly. Good.

	The value of 'Host Names' '[IDVAULTDEV2.dev.acme.edu]' in DC[ADDEV2.dev.acme.edu] is same as the name of driver machine[IDVAULTDEV2.dev.acme.edu]

The Host Names in the registry on the DC match the host name of the Driver Machine. This is important. If not, it sure as heck ain't gonna communicate the passwords properly.

	Error occurred while opening the registry key[SOFTWARE\NOVELL\PWFILTER\DATA]. Access is denied.

Now here we start seeing some issues. My user account cannot read the password cache for the filter. As I went through troubleshooting this issue, I did manage to resolve the permission issue with the Registry, and when I did the log added content that looks like this:

	Opened key [SOFTWARE\NOVELL\PWFILTER\DATA].

	 The password was last updated for user adminbob on 4/6/2015 at 16 hrs and 57 mins
	 The password was last updated for user CJSMITH on 15/6/2015 at 13 hrs and 35 mins
	 The password was last updated for user CJSMITH.lds on 20/5/2015 at 12 hrs and 28 mins
	 The password was last updated for user GCARMAN on 24/4/2015 at 5 hrs and 58 mins
	 The password was last updated for user ServiceLDS on 20/5/2015 at 15 hrs and 28 mins
	 The password was last updated for user BSMITH on 7/4/2015 at 15 hrs and 15 mins
	 The password was last updated for user testbob on 4/5/2015 at 11 hrs and 29 mins
	 The password was last updated for user testbob2 on 4/5/2015 at 11 hrs and 29 mins
	 The password was last updated for user testbob3 on 16/6/2015 at 11 hrs and 56 mins
	 The password was last updated for user testbob4 on 2/6/2015 at 10 hrs and 49 mins
	 No more items to process Currently 
	.
	 Number of Entries Processed is 12

This shows that the passwords were stuck on the DC with the filter, and not getting sent back to the Remote Loader. So I suspected RPC as the next possibility and guess what they test next?

	Running RPC Checks.

	Checking whether this tool can reach the filter through RPC
	This tool can reach the filter through RPC

This test is possibly the most critical test of them. Can the driver machine (Where running the tool) talk via RPC to the Filter on the DC. This is the one that is hardest to otherwise detect and where this tool really shines in it's value.

	Checking if the filter can connect to the driver
	pwFilter can connect to PassSync RPC server on driver machine  - 0 

	Tue Jun 16 11:23:31 2015 : Checking the Domain Controller ADDEV1.dev.acme.edu ....

Skipping the loop through the second DC.
	
	Tue Jun 16 11:23:31 2015 : Tests on all DCs are done 

	Press any key to close this trace ... 

As you can see this identified an issue, I resolved it, but was still left with a problem. Now as it turns out, it did not find my specific problem as an issue to report, but it hinted at it strongly. I will continue troubleshooting my specific issue in part 2 of this series because I think it is an interesting issue worth discussing and it is not clear that NetIQ has actually documented the solution to my specific issue. I initially got told I was in an unsupported mode, but they could not provide documentation to support that. Instead it turns out that there is a timing issue when running more than one ADDriver.dll on a Windows server, and only the first one loaded will succeed at making RPC connections. If it loads second, it will fail.





Complete logs for the record:

Tue Jun 16 11:22:38 2015 : Starting Checks on Driver Machine ..... 

Tue Jun 16 11:23:10 2015: Logging as admin user.

Tue Jun 16 11:23:10 2015 : 
The List of all Domain Controllers -
1. ADDEV2.dev.acme.edu
2. ADDEV1.dev.acme.edu

Tue Jun 16 11:23:10 2015 : RPC Service is running 
Tue Jun 16 11:23:10 2015 : Full DNS name of the driver machine is IDVAULTDEV2.dev.acme.edu 

Tue Jun 16 11:23:10 2015 : The version of the Operating System is : Microsoft  Service Pack 1 (build 7601)
Tue Jun 16 11:23:10 2015 : An AD driver instance is found configured on Remote Loader
Tue Jun 16 11:23:10 2015 : AD Driver which is configured with Connection port 8092 and Command port 8003 is running 

Tue Jun 16 11:23:10 2015 : List of local files related to Driver are :
	C:\Novell\remoteloader\64bit\ADDriver.dll
	C:\Novell\remoteloader\64bit\ActiveDirectoryEnterprise-Config.txt
	C:\Novell\drivertrace\ActiveDirectoryEnterprise-Trace.log
Tue Jun 16 11:23:14 2015 : Driver version is "4.0.1.0">AD</pr"20150311_120000"AAi8P757zh70a80
Tue Jun 16 11:23:14 2015 : Driver version is "4.0.1.0">AD</pr"20150311_120000"AAi8P and Build ID is "20150311_120000"AAi8P757zh70a80
Tue Jun 16 11:23:14 2015 : 
Tue Jun 16 11:23:14 2015 : An AD driver instance is found configured on Remote Loader
Tue Jun 16 11:23:14 2015 : AD Driver which is configured with Connection port 8090 and Command port 8000 is running 

Tue Jun 16 11:23:14 2015 : List of local files related to Driver are :
	C:\Novell\remoteloader\64bit\ADDriver.dll
	C:\novell\remoteloader\64bit\ActiveDirectoryHealthcare-Config.txt
	C:\novell\drivertrace\ActiveDirectoryHealthcare-Trace.log
Tue Jun 16 11:23:16 2015 : An AD driver instance is found configured on Remote Loader
Tue Jun 16 11:23:16 2015 : AD Driver which is configured with Connection port 8091 and Command port 8001 is running 

Tue Jun 16 11:23:16 2015 : List of local files related to Driver are :
	C:\Novell\remoteloader\64bit\ADDriver.dll
	C:\Novell\remoteloader\64bit\ActiveDirectoryUniversity-Config.txt
	C:\Novell\remoteloader\64bit\ActiveDirectoryUniversity-Trace.log
Tue Jun 16 11:23:16 2015 : Driver version is "4.0.1.0">AD</pr"20150311_120000"blisher8/0/sLE 
Tue Jun 16 11:23:16 2015 : Driver version is "4.0.1.0">AD</pr"20150311_120000"blisher8/0/sLE 
Tue Jun 16 11:23:16 2015 : Driver version is "4.0.1.0">AD</pr"2015
Tue Jun 16 11:23:16 2015 : An AD driver instance is found configured on Remote Loader
Tue Jun 16 11:23:16 2015 : AD Driver which is configured with Connection port 8094 and Command port 8004 is running 

Tue Jun 16 11:23:16 2015 : List of local files related to Driver are :
	C:\Novell\remoteloader\64bit\ADDriver.dll
	C:\novell\remoteloader\64bit\ADPSInteractive-Config.txt
	C:\novell\drivertrace\ADPSInteractive-Trace.log
Tue Jun 16 11:23:21 2015 : Driver version is "4.0.1.0">AD</pr"20150311_120000" D7dgVtEAQN 1QJ
Tue Jun 16 11:23:21 2015 : Driver version is "4.0.1.0">AD</pr"20150311_120000" D7dgVtEAQN 1QJ
Tue Jun 16 11:23:21 2015 : Driver version is "4.0.1.0">AD</pr"2015
Tue Jun 16 11:23:21 2015 : An AD driver instance is found configured on Remote Loader
Tue Jun 16 11:23:21 2015 : AD Driver which is configured with Connection port 8096 and Command port 8006 is not running 

Tue Jun 16 11:23:21 2015 : List of local files related to Driver are :
	C:\Novell\remoteloader\64bit\ADDriver.dll
	C:\Novell\remoteloader\64bit\ADPSNoninteractive-Config.txt
	C:\Novell\drivertrace\ADPSNoninteractive-Trace.log
Tue Jun 16 11:23:21 2015 : The 'Driver Machine' value in the registry key[SOFTWARE\NOVELL\PASSSYNC] is : 1. 

Tue Jun 16 11:23:21 2015 : The 'Domains' value in registry key[SOFTWARE\NOVELL\PASSSYNC\DATA] is dev.acme.edu 

Tue Jun 16 11:23:21 2015 : Number of subkeys(passwords cached) under the key[SOFTWARE\NOVELL\PASSSYNC\DATA\dev.acme.edu]is 1 

Tue Jun 16 11:23:21 2015 : The information of the user passwords cached:
	User: ABALLY
	Timestamp of password change:  14:03:32.57 March 13 2015 

Tue Jun 16 11:23:21 2015 : Tests on this driver machine are done 

Press any key to close this trace ... 

Tue Jun 16 11:23:31 2015 : Starting Checks on All DCs ..... 



Domain Controller Check:

Tue Jun 16 11:23:31 2015 : Starting Checks on All DCs ..... 

Tue Jun 16 11:23:31 2015: Logging as admin user.

Tue Jun 16 11:23:31 2015 : 
The List of all Domain Controllers -
1. ADDEV2.dev.acme.edu
2. ADDEV1.dev.acme.edu

Tue Jun 16 11:23:31 2015 : Checking the Domain Controller ADDEV2.dev.acme.edu ....

Running Basic Diagnostic Checks.

Password filter files installed on this DC are C:\Windows\System32\PWFILTER.DLL and C:\Windows\System32\PSEVENT.DLL

The value of 'Host Names' '[IDVAULTDEV2.dev.acme.edu]' in DC[ADDEV2.dev.acme.edu] is same as the name of driver machine[IDVAULTDEV2.dev.acme.edu]

Error occurred while opening the registry key[SOFTWARE\NOVELL\PWFILTER\DATA]. Access is denied.

Running RPC Checks.

Checking whether this tool can reach the filter through RPC
This tool can reach the filter through RPC

Checking if the filter can connect to the driver
pwFilter can connect to PassSync RPC server on driver machine  - 0 

Tue Jun 16 11:23:31 2015 : Checking the Domain Controller ADDEV1.dev.acme.edu ....

Running Basic Diagnostic Checks.

Password filter files installed on this DC are C:\Windows\System32\PWFILTER.DLL and C:\Windows\System32\PSEVENT.DLL

The value of 'Host Names' '[IDVAULTDEV2.dev.acme.edu]' in DC[ADDEV1.dev.acme.edu] is same as the name of driver machine[IDVAULTDEV2.dev.acme.edu]

Error occurred while opening the registry key[SOFTWARE\NOVELL\PWFILTER\DATA]. Access is denied.

Running RPC Checks.

Checking whether this tool can reach the filter through RPC
This tool can reach the filter through RPC

Checking if the filter can connect to the driver
pwFilter can connect to PassSync RPC server on driver machine  - 0 

Tue Jun 16 11:23:31 2015 : Tests on all DCs are done 

Press any key to close this trace ...