If you have been working with NDS, or eDirectory as they have been calling it for the last couple of years, then you should be familair with DSTRACE. There have been at least five versions and flavors of DSTRACE made available, depending on the version and platform.

DSTRACE and NetWare

Let's start with NetWare first. Since the earliest days of NDS that I was involved in (NetWare 4.10 and Intra NetWare 4.11, although I used Netware 3.11 and 3.12 - there was no NDS in those versions), there has been a DSTRACE.

The first version was managed by SET DSTRACE commands. That is:

set dstrace = on
set ttf = on (Trace to File, so that you log the output to a file, usually sys:\system\dstrace.log)
set dstrace=NODEBUG to clear all flags.

Then you would use "set dstrace=" followed by a command, of which there were two types: filter flags that were activated or deactivated with a plus or minus sign before the filter flag.

Some examples are:

S to show synchronization traffic
limber to show limber process traffic
blink to show backlinker activity

Then there were the command flags, starting with an asterisk (*) such as:

*H to kick off the heartbeat that forced synchronization to begin.
*L to kick off the limber process
*B to kick off the Backlinker process

There were many more filters and actions, but those above are some of the most commonly used. Search the Novell Knowledge base for examples of more filters and flags.

This model used was not really extensible, so to add trace support for new features like NMAS, PKI, and Identity Manager (still flagged by the old DirXML name), Novell needed something that was more flexible going forward.

DSTRACE with eDirectory

The release of eDirectory really forked at the point at which the underlying database for storing everything in moved from RECMAN (Record Manager) used in Netware 4.x, and 5.x, to FLAIM (the same database that Groupwise uses, which is much more scalable). With this, we got a new DSTRACE console tool. Interestingly, the old set DSTRACE interface is still there on NetWare only.

This new DSTRACE was activated with similar commands, except that DSTRACE was actually an NLM instead of SET commands. For example, you could run LOAD DSTRACE, then you could execute DSTRACE commands like DSTRACE SCREEN ON, or
DSTRACE -ALL to clear all the flags that are on by default. This used a nice color-coded screen display. It shows by color what will log to screen, what will log to the file, and what will log to both.

The next trick is to realize that you can combine commands, often on a single line. So after DSTRACE is loaded, you could enable a bunch of flags at once with a command like DSTRACE DXML DVRS to turn on the two filter flags and show Identity Manager-related events. This is the model Novell seems to be using going forward with DSTRACE.

DSTRACE on Windows

With the release of the cross-platform versions of eDirectory, a non-NLM-based trace tool was needed. On Windows, Novell wrote the modules as DLMs that could run in the eDirectory memory space.

Thus from the eDirectory console on a Windows server running eDirectory, you can start the Dstrace.dlm tool. A screen pops up to show you the DSTRACE output. (Note: This happens if you connected to screen 0 on the Windows server - that is, if you are at the console, or use the Terminal Services client, and run with the command "mstsc /console" on Windows or "rdesktop hostname -0" on UNIX-like operating systems. Otherwise, the window for DSTRACE will not pop up. See numerous articles on this topic, such as: http://www.novell.com/communities/node/1460/solving-rdp-issues-remote-loader). If you check the menus, you will find the options to set the filter flags.

The filter flags are named the samed as in the DSTRACE.NLM, and that seems to be the model across the rest of the DSTRACE interfaces. This is a nice GUI interface and is very familiar from the NetWare DSTRACE.NLM perspective.

DSTRACE and *NIX Systems

The fourth interface to DSTRACE is for the UNIX-like operating systems. I have to be careful, since technically Linux (all the various distributions), AIX, Solaris, HPUX, and the like are all sort of UNIX-like. I would call them all UNIX-like and basically similar, but that could start a holy war among those who use those operating systems. However, regardless of what some users may believe, the interface to all the UNIX-like operating systems that eDirectory can run on, is basically the same.

On UNIX-like OS's, the interface is called by the use of the "ndstrace" executable. The path to it sort of depends on the exact operating system you are using, as well as the version of eDirectory, since it changed between the eDirectory 8.7.3.x family and eDirectory 8.8.x family. One of the neat things with all the tools coming from the UNIX-like eDirectory build is that they all support multiple instances of eDirectory on a single server. When you run each tool (ndstrace, ndsrepair, ndsmanage, etc.) you get asked which instance you are referring to.

You should be able to use ndstrace once you know its path. This opens a text console window in your current shell that looks much like the Netware DSTRACE.NLM screen. The commands are pretty much the same as the rest of the version, but interestingly, they revert back to the original format of commands from the NDS version of set dstrace.

At the ndstrace console, you can type things like:

set dstrace=NODEBGUG to clear all the flags (annoyingly, "-ALL" does not work!
set dstrace= LDAP to see LDAP events
set dstrace=-LDAP DXML DVRS to stop seeing LDAP, but enable the Identity Manager filter flags

The "set dstrace=NODEBUG" took me a while to find out, and I thought was kind of annoying to not support -ALL. Oh, well - live and learn.

DSTRACE and iMonitor

The fifth and final DSTRACE interface that I can think of is the iMonitor version. This is probably the most powerful version of them all, because all versions of eDirectory support it, across all platforms. The ports in use depend on your operating system, and they are paired with a clear text port and an SSL encrypted port. On NetWare, this defaults to 8008 and 8009; Windows defaults to 8010 and 8012; and UNIX-like operating systems default to 8028 and 8030.

You can jump straight to https://serverIP:80xx/nds/trace, which will force you to log in. Then you'll see the DSTRACE configuration screen and a web-based version of the options in the Windows version of the DSTRACE.DLM interface.

What is nice about this interface is that error codes that show up are usually hyperlinked to the error code index included with Novell iMonitor. Also, you get color-coded events, which makes picking things out of the trace easy. There is a trace history option on the left frame that enables you to choose between the last 10 log files.

The new extensible DSTRACE model that is being used across the platforms (whether it is via iMonitor, ndstrace, dstrace.nlm, or dstrace.dlm) is great, because all sorts of interesting things have been added to the trace filter list.

Use Cases

I found a very powerful example when I was troubleshooting a DSREPAIR problem for a client. We would run DSTRACE, and it would error with a text box, with a very generic error, but we had no error code or message to pursue. This was a real stumper. For fun, since I was out of options, I did a:

DSTRACE
DSTRACE SCREEN ON
DSTRACE FILE ON
DSTRACE ALL

Then I repeated the DSREPAIR that caused the error and looked at the log file for any errors. It turns out the REPR flag (DSREPAIR tracing) showed that we were out of disk space. But the DSREPAIR interface was not surfacing the error code.

Another example is Universal Password troubleshooting. Often you will need to see what is going on when a UP bind fails or has issues. The best way is to enable the NMAS trace option and see what happens. This is how you find the error codes to look up in the Knowledge Base, such as -1697 and -16049. These errors basically tell you that either your SDI keys are bad or you have not really set up a Universal Password policy for the user in question.

Probably the most common use I have for DSTRACE these days is for Identity Manager and LDAP tracing. When working with Novell Identity Manager with trace enabled, these will show up in Dstrace when you enable the DXML and DVRS flags.

There is another entire article topic on reading DSTRACE for Identity Manager events: see http://www.novell.com/communities/node/4428/edit. Suffice it to say, you cannot debug an Identity Manager driver without using DSTRACE.

(This, of course, brings up a minor sixth style of DSTRACE that is specific to Identity Manager, where the general DSTRACE shows the co-mingled trace from all drivers in a single view. This can make it VERY hard to read and pick out the events you care about. To make it easier, Identity Manager drivers can trace into indvidual files, one per driver, so that you can see a consistent set of events from a single driver. This still co-mingles the two channels into one file, along with the Query and responses, making it require skill to read. Still, this is much better than having all the drivers in a single view. However, since this final version of DSTRACE is only for Identity Manager drivers, I do not really count it as a sixth type.)

My favorite use for DSTRACE is for when you run into a problem where you have no idea how to begin troubleshooting. You can always turn on all the flags, repeat the task that causes the problem, and then search for any errors in the trace. It will be way too much stuff to read, but you might get a hint to point you in the correct direction. That's sort of the "blast it with all the tracing you can enable, and see if anything shows up" approach. Be aware that you will see a LOT of errors that are meaningless. DSAgent shows a large number of -603 and -601 errors. Many applications that use NDAP (Novell Directory Access Protocol, analogous to LDAP, which is the Lightweight DAP, both of which are versions of the X.500 DAP concept) will be looking for things, and those failures are usually quite benign and should be ignored. DSTRACE can trace iMonitor, PKI, NMAS, DS stuff, Identity Manager and more, so you never know where the error is coming from. Still, the odds are good you will find it with this approach.

Conclusion

Remember to turn OFF the trace once you have found the error, as you will be adding load to the server and adding lots of file writes that are not needed .

In summary, DSTRACE is one of the more powerful troubleshooting tools for eDirectory and Novell-related products. If you are not using it now, you will probably have to on the next issue you need to troubleshoot.