Basic Troubleshooting Tips for eDirectory on Linux


Novell eDirectory is a powerful directory service, and one of its nicest features is that it is very cross platform. It pretty much runs on most major server operating systems. (Basically everything but Mac or BSD). One of the upsides is that the same basic code base is used on all these platforms which helps expose bugs that otherwise you would never figure out. The downside is that they have to support the same code on all those platforms, and deal with platform specific bugs on all of them.

The major downside to this cross platform'ness from my perspective is that the troubleshooting techniques are slightly different for each platform.

For example, running on Netware, eDirectory is automatically started and you can only have a single instance running on a server.

Running on Windows you can have multiple instances, but eDirectory is a service that runs and can fail to start. There is a GUI to show loaded modules, and load or unload them from it.

On Linux you can have multiple instances, and unloading modules is a little trickier, being command line based, but conversly more powerful since it is inherently scriptable.

You can see some more about the many face of Dstrace in the article, The Many Faces of DSTRACE.

Of course with supporting eDirectory on so many platforms, locations of files, ports used by application, can be different.

For example, on Netware, there was a truly excellent tool called DSbrowse. Its predecesor, DSView was very powerful, but the interface was beyond cryptic. With DSBrowse, it got a new C-Worthy interface (the blue background with yellow text (Or is it green? I am a bit colour blind and cannot tell) menuing system common on Netware) and was very easy to use to examine the current state of objects in the directory.

For other platforms, there was a short lived port to Windows called XBrowse, but that went away. The replacement is to use iMonitor, which is cross platform, to get a web view of the object state in the local replicas. The idea is that all the functionality of DSBrowse should be exposed via iMonitor in a cross platform way.

iMonitor runs as part of the HTTPSTK web server, which on Open Enterprise Server (Netware AND Linux kernels) runs the Novell Remote Manager interface. If you have just eDirectory installed, you get an instance of HTTPSTK running that has a secure and unsecure port, for DHost and iMonitor. DHost is a Web view of the loaded modules to try and let you load and unload them in a cross platform way.

On Netware, HTTPSTK listens by default on port 8008 (http), and 8009 (https).

On Windows, HTTPSTK listens by default on port 8010 (http), and 8011 (https).

On Linux, AIX, HP-UX, and Solaris (the Unix platforms), HTTPSTK listens by default on port 8028 (http), and 8030. (https).

If there is a port conflict at install time, the ports usually default to increment up by 2 until they find a free port.

The web based DSTrace, is part of iMonitor and is probably the most common part of iMonitor that is used, but it is only a single part of the very useful tool for examining eDirectory on a local server.

The DIB (the files where the actual eDirectory database is stored) files are located differently on each platform by default, and on Unix builds differs between eDirectory 8.7.3.x and 8.8.x versions.

Once you go to multiple instances, you need a seperate path for each instance, so the defaults become pretty meaningless quickly.

The old standby Dsrepair is very different on the various platforms. On Netware it DSREPAIR.NLM with a C-Worthy interface. On Windows it is dsrepair.dlm that is loaded via the DHost GUI, or called via iMonitor. On Linux/Unix it is ndsrepair, with a large series of command line switches to make it dance to your tune. Again, this makes it inherently more scriptable, but has a steeper learning curve to get up to speed. There is a free 'curses' (Sort of a Linux/Unix equivalent to C-Worthy) interface that wraps ndsrepair and gives you back many of the old menus familair from the Netware days.

ndstrace is a very powerful tool on Linux, and beyond the usual tracing of events that have been done in Dstrace over the years, it also has some interesting functions, like the "modules" command. Here is the output of the modules command from an eDir 8.8.1, server Novell Identity Manager installed but not running. It turns out the core modules for Identity Manager are "vrdim" (from the days when the product was first code named virtual replicas I imagine) and "dxevent".

vrdim           Not Loaded
statuslg Not Loaded
sss Not Loaded
ssncp Not Loaded
ssldp Not Loaded
spmdclnt Running
snmpinst Not Loaded
snmp Not Loaded SNMP Trap Server for Novell eDirectory 8.8.1
snmp-0 Not Loaded
sasl Not Loaded
repair Not Loaded
pkiserver Not Loaded Novell Certificate Server
pkiinst Not Loaded
nmasldap Not Loaded
nmasinst Not Loaded
nmas Running
nldap Not Loaded LDAP Agent for Novell eDirectory 8.8 SP1
niciext Not Loaded SDI/SASDFM Transport Service
ndsinfo Not Loaded
ndsclone Not Loaded Directory Clone Agent For Novell eDirectory 8.8 SP1
merge Not Loaded
lsss Not Loaded
ldapxs Not Loaded
krbpwd Not Loaded
jvmload Not Loaded
imon Not Loaded NDS iMonitor 2.4 for Novell eDirectory v20111.59
hconserv Not Loaded HTTP Console Server For Novell eDirectory 8.8 SP1
gwadj2 Not Loaded
gams Running Graded Authentication Management Service
embox Not Loaded eDirectory Management Tool Box Engine
dxldap Not Loaded
dxevent Running
dump Not Loaded
dsr Not Loaded
dsi Not Loaded
dsbk Not Loaded
backupcr Not Loaded
auditds Not Loaded
dstrace Running Trace For Novell eDirectory 8.8 SP1
httpstk Running HTTP Protocol Stack For Novell eDirectory 8.8 SP1 [ nds ]
nds Running Directory Agent For Novell eDirectory 8.8 SP1
masv Running Mandatory Access Control Service
dsloader Running Loader Services For Novell eDirectory 8.8 SP1 [ httpstk ]
dhlog Running DHost message logging module for Novell eDirectory 8.8 SP1
ncpengine Running NCP Protocol Stack For Novell eDirectory 8.8 SP1
ndsd Running Novell eDirectory 8.8 SP1 Host Environment

[1] Instance at /etc/opt/novell/eDirectory/conf/nds.conf: NYCSUSE2.OU=servers.O=twcny.TWCIDVAULT

When you run ndstrace in a terminal session, at least via SSH it does not seem to recognize that the screen has been resized, if you change its size to make it easier to read. I find I have to stop and restart the ndstrace process to get to use the extra real estate. I imagine a terminal on the server console would catch the resize event.

Above we have the output from a "modules" command. There is also the ability to "load <moduleName>" and "unload <moduleName>". For example, you could load or unload vrdim. It is the short name you see in the left most column that needs to be done.

To make this scriptable, you call ndstrace with the -c switch, followed by the command you want it to execute. Thus you could execute:

ndstrace -c unload vrdim

to stop Identity Manager. (Probably not something you would need to do often, but good to have it available).

The usual DSTRACE commands apply here. Some are throwbacks to the old days of Netware 4.11, like "set dstrace=nodebug" to clear all values, instead of the more modern DSTRACE -ALL that we got used to on Netware.

To trigger processes you would use the same commands as at a Netware console, like "set dstrace=*h" to kick off the sync process (H for heartbeat). "set dstrace=*ss" to start a Schema sync process. "set dstrace=*b" to kick off the backlinker process.

To enable more filters you can use commands like "set dstrace= LDAP" or "set dstrace=-NMAS" to add LDAP events or remove NMAS events from the output to the screen.


A useful file to look at is ndsd.log, which should be wherever your DIB set is stored, in a log subdirectory.

An example is shown below.

May 09 09:49:43  NMAS Server Version: Build:20071127 started
May 09 09:49:43 SPM DClient already started (2)
May 09 09:49:43 Loading SecretStore Server...
May 09 09:49:43 Novell SecretStore Service Version 3.4.0 Loaded Successfully
May 09 09:49:43 Loading SecretStore LDAP Transport Plugin...
May 09 09:49:43 Novell SecretStore LDAP Plugin Version 3.4.0 Loaded Successfully.
May 09 09:49:43 SecretStore LDAP Extension Handler Loaded Successfully
May 09 09:49:43 LDAP Agent for Novell eDirectory 8.8 SP1 (20114.57) started
May 09 09:49:43 NDS iMonitor 2.4 for Novell eDirectory v20111.59 started successfully.
May 09 09:49:44 SASL Version: Build:20070914 started
We are in DHost.
Novell JClient 1.3.1172-1.3.1172. Copyright 1999 Novell Inc. All Rights Reserved.
[ -- DHost Logging STARTED Fri May 9 09:49:38 2008 -- ]
May 09 09:49:47 Novell PKI Services Started Successfully
May 09 09:49:47 Loading SecretStore NCP Transport Plugin...
May 09 09:49:47 Novell SecretStore NCP Plugin Version 3.4.0 Loaded Successfully.
May 09 09:53:30 Couldn't open macaddr
May 29 13:26:27 Repair utility for Novell eDirectory 8.8 - 8.8 SP1 v20112.44 Successfully loaded
May 29 13:26:27 Repair utility for Novell eDirectory 8.8 - 8.8 SP1 v20112.44 Successfully unloade
Jun 03 10:58:08 Got SIGINT/SIGTERM signal, server shutting down
Jun 03 10:58:40 Path of Novell eDirectory configuration file /var/opt/novell/eDirectory/nds.conf
Jun 03 10:58:40 Host process for Novell eDirectory 8.8 SP1 v20114.24 successfully started
Jun 03 10:58:40 DHLog: file size 1048576
Jun 03 10:58:40 MASVInit called
Jun 03 10:58:40 MASV : Initialized
Jun 03 10:58:40 NMAS Server Version: Build:20071127 started
Jun 03 10:58:40 SPM DClient Version: Build:20070914 started

You can see the modules that loaded, and version numbers for some of them. Most importantly, if something did not load, often the error might appear in this log, giving you a starting point to troubleshoot. Once you have an error message you are halfway to solving the problem.


The ndsmodules.conf controls what modules start when ndsd is started. There are a couple of flags to control the start behavior. Auto is pretty obvious, automatically start when ndsd starts.

Modules flagged as system will not unload, which make sense when unloading them would pull the rug out from underneath the rest of the system.

If you flag something as "fail" it is usually foundational, and if it does not properly start, you can imagine that everything else should not load after it.

# ndsmodules.conf: NDS Module Description File
# This file describes the modules to be loaded at bootup. Note that modules
# that need to be loaded would have auto flags set. Other modules can also
# be present here if a default command line need to be specified. Modules
# will be loaded in the order that's listed here.
# Syntax:
# modulename flags cmdline
# Each line in this file represents a modulename. It should not
# contain prefix(lib) or suffix(.so, .la etc.). We'll look at a
# corresponding .la file to pickup the correct modulefile.
# flags: should be a comma seperated (no whitespace) list of valid options.
# auto -> autoloaded when dhost comes up
# system -> Will not be unloaded.
# fail -> Treat as an error and exit if loading fails.
# noop -> No flags. MUST for specifying command line without any flags

dhlog auto,fail #DHost logger
ncpengine auto,system,fail #Core NCP Services
dsloader auto,system,fail #Loader
masv auto,system,fail #Modular Authentication Services
nds auto,system,fail #Core DS Services
niciext auto
gams auto
snmp auto #snmp
httpstk auto #DHost HTTP Stack
hconserv auto #HConServ
nldap auto #LDAP Server
imon auto #iMon
embox auto #eMBox
pkiserver auto #PKI server
ssncp auto #SecretStore

If for some reason you did not want LDAP loading on your server at all, you could easily edit this file and comment out the line loads it as auto.

Knowing about these few tips and tricks can be very helpful in working on eDirectory/Identity Manager when running on Linux. If you have any other good suggestions, please respond with an article to contribute, so we all can benefit!


How To-Best Practice
Comment List