Original Publish Date: 2010-03-09
This article will guide the reader in a graphical way though IDM Traces. It assumes that the reader is already familiar with IDM and basic trace reading. This article will detail regular channel flow. A second article will cover Queries, Direct Commands from Policy, Status messages and Add-Association events.
This article assumes you are already familiar with the IDM product. If that is not the case, please read this excellent series by David Gersic that explains what you need to know before:
This article expands and complements the information provide in article Capturing and Reading Novell Identity Manager Traces, so it would be a good idea to read that article as well.
Before going into the traces themselves, it is important to break certain misconceptions about the IDM engine. There is a part of the engine that is responsible for converting information from the eDirectory domain (data format and schema structure/name) to the application domain, and vice versa. Those comprise the Schema Mapping policy set, the Input Transformation Policy set, the Output transformation policy set and the Association Reference processor.
This portion of the engine is actually independent of the Publisher and Subscriber channels, and it is instantiated by both channels. All events coming in a channel will pass that portion, and the result of the command (status message any other result) will pass through this portion in the opposite direction. So as we can see below, each channel has its instance of those policy sets. This is important to keep in mind, and will be crucial to understand the Out-of-Band events.
The diagrams shown in iManager and Designer (fishbone view) are simplified diagrams. The Publisher diagram below is also simplified, but covers a bit more than what we see in iManager or Designer. The extra blocks are the Association Processors (pre and post-filter), the Optimize Modify block and the Merge Processor block.
The Optimize Modify block exists because writing to eDirectory is expensive (slow) compared to querying eDirectory. Due to that, if we compare the attribute value from the current transaction with what is already in eDirectory, and discard attributes that contain the same value, we may gain performance by dropping unnecessary writes before they happen. This behavior is controlled per attribute and set in the driver's filter. This is triggered only on <modify> events going into eDirectory, and it happens before the Command Transformation Policy set as can be seen in the diagram below. Since each connected system has a different behavior regarding queries and writes, there is no equivalent block in the Subscriber channel.
The merge process exists to reconcile objects in eDirectory with the ones in the connected application when equivalent objects already exist on both sides. Its behavior can be controlled per attribute in the driver filter. A merge is triggered by:
The Merge Processor, when invoked, query destination data store for all attributes listed in the driver filter for the current object's class, then build <modify> events for both source and destination data stores based on the ¨Merge Authority¨ on each attribute in the Driver's filter
Each event comes as XML under the <input> document. Right after the Event Transformation Policy set, if there are 2 or more events coming through, only one will be sent at a time by the Event Sequencer. It will hold the other events in memory until a <status> message comes back as a result of the event currently being processed.
The Pre-Filter Association Processor receives the events from the Event Sequencer that exists right after the Event Transformation Policy set. Its function is to lookup eDirectory to check if the object referenced by the event exists, and if so, add the object class (if missing) and handle cases where multiple associated objects are found. It performs the lookup by destination entry id, destination dn or association value in this order of priority.
The post-Filter Association Processor receives events from the Sync&Ignore Filter and performs certain actions. Depending on certain criteria it will do one of the following actions:
The Synthetic Add process on the Publisher channel happens in the Post-Filter association processor block. The Synthetic Add is triggered by:
The Synthetic Add process query back the source data store for all attributes listed in the driver filter for the current object's class. It then build an <add> document with those attribute values and discards the original <modify> or <sync> event.
The diagrams shown in iManager and Designer (fishbone view) are simplified diagrams. The Subscriber diagram below is also simplified, but covers a bit more than what we see in iManager or Designer. The extra blocks are the Association Processor, Event Sequencer and the Merge Processor block.
The Association Processor receives events from the Event Sequencer and performs certain actions. Depending on certain criteria it will do one of the following actions:
Each event comes as XML under the <input> document. Right after the Event Transformation Policy set, if there are 2 or more events coming through, only one will be sent at a time by the Event Sequencer. It will hold the other events in memory until a <status> message comes back as a result of the event currently being processed.
It is important to remember also that the Sync&Ignore filter and the event cache (TAO file) for the Subscriber channel are outside of the driver's channel. This means that even if a driver is stopped, events will be evaluated against the filter and sent to that driver's cache file. Only if the driver "Startup Option" is set to disabled the caching of events will stop. This also means that the caching of events by the driver in the Subscriber channel will not show in IDM trace to file.
Now that we are familiar with some extra details in each channel, lets delve deeper on how Policy sets, Policies and Rules process events.
A Policy Set is a grouping of policies. Each policy set has its own meaning and purpose, as seen in previous in David's guided tour of IDM. On trace level 3 and above, the processing of each policy inside a policy set will be shown in trace, as well as the XDS resulting for that policy's processing.
Policies inside a policy set are executed in order, top to bottom.
A Policy is a set of Rules that accomplish a certain goal.
A Policy processes events one at a time, by evaluating and executing its rules. When it finishes processing all events within an <input> or <output> tag it returns the resulting XML. The resulting XML will then be passed to the next step in the IDM engine flow.
Rules are the actual workers in an IDM Driver. They are composed of conditions and actions.
A Rule will only execute its actions if its condition set evaluates to be boolean True, otherwise it will be skipped.
Rules can issue special requests (Out of Band requests) from certain conditions and actions. When one of those requests is issued, the IDM Engine will stop processing the current event until the request if fulfilled.
One example of those special requests is a Query. When a Query is issued by a condition/action in a Rule, the IDM engine will:
Out of Band Events will be the subject of another article.
IDM events are represented as XML, and can be under either an <input> or an <output> tag
There can be multiple events under the same top level tag. Those events are processed in order, top to bottom, by each policy in a driver
To make things easier to understand, lets go over an example. In this example we have the result of an coming back, and that result has actually 2 events: an and a message.
An IDM engine trace level 3 or above will show as we enter a policy the text "Applying policy" followed by the Policy's name. The extra characters (% C and %-C)around a policy's name are for dstrace usage, they tell dstrace to write the text between them in yellow color.
Note that a Policy name is defined in the Policy's own XML, and doesn't have to match the eDirectory object that holds that Policy. The policy's own name appears in Designer and also on iManager plug ins that come with IDM 3.6.1 or later. Earlier iManager plug ins won't show that name.
Since we process one event at a time, the IDM trace show us which event is being processed and what is that event's position within the <input> or <output> tag. So in this case the rules inside this policy will be applied to the <add-association> event now, and that event is the 1st inside this <output> tag.
Now the rules inside this policy will be applied to the <status> event now, and that event is the 2nd inside this <output> tag. Note that the #2 means second within the <output> tag. It does not mean we have 2 <status> messages.
After showing us the Policy name and which event is being processed, the trace then display information about each rule it evaluates, starting with the Rule's name.
Since a Policy processes one event at a time, we will see in the trace the same Policy's Rules being evaluated once per event. That is normal and expected.
In the image below, we have a Policy with two rules, and we happen to have 2 events as well. As seen in the image each rule was executed twice, one time for each event. If we had 10 events, each rule would have been executed 10 times, once per event present in the <input> or <output> tag.
Now that we saw more about how policy sets, policies and events appear in the IDM trace, lets see how rules behave. Rules are the actual workers on the IDM engine, so understanding how they behave will allow us to understand the traces better.
The condition set for a Rule will always be evaluated by the IDM engine. When that is happening, the text "Evaluating selection criteria for rule" will appear, followed by the rule's name.
If that whole condition set evaluates to be boolean True, then the Rule will execute its actions (Rule selected). If the condition set evaluates to boolean False, the rule's actions will be skipped (Rule rejected).
Keep in mind that "Rule rejected" does not mean that you have an issue in your logic. Since each rule reacts to certain events & conditions, it is normal and expected that several of your driver's rules will be skipped. For example, lets say you synchronize both users and groups. All rules that handle only users should be skipped (rejected) when a group event is coming through.
Before a rule executes its actions, the trace will show "Applying Rule" followed by the rule's name.
Hope that the example above helped the reader understand how the engine processes events. There are a few more details that are relevant, so lets get to them now:
Since conditions are evaluated using boolean logic, where each individual condition (and ultimately, condition group) will evaluate to True or False, the driver does not necessarily pass through all conditions in the Rule.
In boolean logic, an OR only results in False if all its individual components are False. An AND only results in True if all its individual components are True. Given that information, True OR anything = True, so we can skip the other components. In the same token, False AND anything = False, so we can also skip those components. That is why in the IDM trace we see only part of the conditions of a Rule being evaluated. What we see is the engine evaluating the boolean logic in an optimal way.
Lastly, each whole condition appears in the IDM trace between parenthesis, and the result of evaluating that logic appears outside of the parenthesis. Keeping this in mind will help avoid confusing when the condition itself is comparing something to True or False.
Each Action in the Rule's logic will be shown in the IDM trace after the text "Action:", and any processing needed to evaluate its arguments will also be shown.
Here in the first line we have the actual action that will be executed, as well as its arguments.
And below it we see each individual portion of the Argument being resolved until we finally have the Argument's value that will be used in the action.
In the example the Argument for the command "Set Operation Destination DN" was composed from 3 pieces (Tokens): Unmatched Source DN, some text (a comma character) and the value of a Global Variable. The final value of the argument, that will be used in the command, appears after the text "Arg Value:".
So that's how you match an IDM trace to the logic in iManager/Designer. In the next article of this series we will cover Out-of-Band events.
8 Comments:
Anonymous_User:
Absent Member.
2010-03-09 20:11
I did not think you would top your first article in this series in terms of quality and content, but well done sir!
Thanks for sharing this! It is going on my recommended reading list for people starting in IDM right now! (as is your decoding Dstrace article, and David Gersics three articles you quoted).
Anonymous_User:
Absent Member.
2010-03-09 23:26
Puts any of mine to shame.
But thank you for taking the time to do this on your own. I will use this for some time to come.
Anonymous_User:
Absent Member.
2010-03-10 19:56
Thank you very much for the great article!
Anonymous_User:
Absent Member.
2010-03-10 00:29
Thanks for the nice article.
Anonymous_User:
Absent Member.
2010-03-15 17:22
The pictures making it much easier to understand what is going on.
Looking forward to the next in the series.
Anonymous_User:
Absent Member.
2010-04-05 22:44
Nice job on this.
Anonymous_User:
Absent Member.
2010-04-07 14:07
Everyone else says "Great Article", same thing.
Love the process flow graphics, they are going up on the cube wall.
Anonymous_User:
Absent Member.
2010-08-17 10:03
Great Stuff, Fernando! But you made me curious to see the non-simplified channel diagrams...
Support
Documentation
Learning Services
Partner Programs