Anonymous_User Absent Member.
Absent Member.
327 views

Delimited Text driver re-processing after association


Hi,

I'm working on fixing a Delimted text (Aleph) driver which seems to be
re-processing an event on the subscriber channel and i'm not sure why.

Currently the driver is used to create an XML document with user
attributes depending on what Vault attributes are present.

There is a rule in the command Transformation which creates an
association using the WorkforceID and this works correctly, however once
the association is made the Command Transform then re-runs through all
the rules, so the resultant file it submits has the user's details as an
<Add> and appended below it has the details as a <Modify>. The expected
behavior is to have it process it as an <add>, then submit the document
and stop there.

The rules are all working correctly, I'm just not sure why the Command
Transform re-runs when an association is made.

I've tried moving the Add association rule to the Output Transform but
it then no longer works and all events that come through as an Add even
when there is an existing association.

I can try to post a level 3 trace if needed, but I was hoping to get a
higher level understanding of why the Command Transformation
re-processes all its rules if an association is made there.

Any help would be greatly appreciated.

Thanks


--
Jevans78
------------------------------------------------------------------------
Jevans78's Profile: https://forums.netiq.com/member.php?userid=7684
View this thread: https://forums.netiq.com/showthread.php?t=51549

Labels (1)
0 Likes
4 Replies
Anonymous_User Absent Member.
Absent Member.

Re: Delimited Text driver re-processing after association

On 08/18/2014 04:37 AM, Jevans78 wrote:
>
> I'm working on fixing a Delimted text (Aleph) driver which seems to be
> re-processing an event on the subscriber channel and i'm not sure why.


What do you mean by reprocessing? Is there just one event at this point,
or are there actually two events in the current operation document and so
the rules are just running for both of them? This would be clear in a
trace as the policies calls out to which event the rules are being applied.

> Currently the driver is used to create an XML document with user
> attributes depending on what Vault attributes are present.
>
> There is a rule in the command Transformation which creates an
> association using the WorkforceID and this works correctly, however once


Why would you do this in the Command Transformation policyset? This may
be part of the issue; normally this should happen in the Matching
policyset, though normally setting up an association with the Delimited
Text shim does not happen at all since querying the application for
anything doesn't work unless you are using something like the Generic File
shim (which is a CoolSolution that is really great).

> the association is made the Command Transform then re-runs through all
> the rules, so the resultant file it submits has the user's details as an
> <Add> and appended below it has the details as a <Modify>. The expected


This may be normal; after all, you have an association on the event for
some reason, and an association on the Subscriber channel implies that the
event is NOT an add (otherwise you would never have an association).

> I've tried moving the Add association rule to the Output Transform but
> it then no longer works and all events that come through as an Add even
> when there is an existing association.


Just to find out more about the business case of this driver, why add an
association? Is it just to prevent subsequent events from coming through
as an add, or to somehow let deletes eventually go through?

> I can try to post a level 3 trace if needed, but I was hoping to get a
> higher level understanding of why the Command Transformation
> re-processes all its rules if an association is made there.


A trace would be helpful. Understanding exactly what you're seeing, even
with a good description like this, will be much easier if the trace is
available.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Delimited Text driver re-processing after association


Many thanks for your fast response.

A full trace for the event is on pastebin:
http://pastebin.com/8LjGWe5n

I'm not sure on the business logic for associating in the command
transform, only that if a user has already been processed by the driver
once that only certain attributes should be entered into the Doc at the
end.

If the association event is removed, then the event comes through as an
add and the Doc is created correctly, however it always does this.
I believe the association rule was put in to ensure that future changes
coming through the driver come through as a modify event. I agree the
positioning of this rule is wrong, but if it is in the matching policy
then it would become associated before all the Command Transform rules
run and be always seen as a modify event in respect of creating the XML
document.

Thanks for your help on this


--
Jevans78
------------------------------------------------------------------------
Jevans78's Profile: https://forums.netiq.com/member.php?userid=7684
View this thread: https://forums.netiq.com/showthread.php?t=51549

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Delimited Text driver re-processing after association

Okay, a quick look shows the following:

This trace is apparently where the problem exists only, which is what was
requested. If you could post a trace of the case where you are not adding
an association that would be interesting; presumably in that case you ONLY
see one application of the Command Transformation policyset's policies, at
least per your description of the issue.

The problem in your case, I think, is happening much earlier in your
driver config. Adding the association where you are doing so may not be
terrible (though I'd probably try to handle that on the returning success
status) but merely adding that association there is not causing the second
event, and instead you are explicitly doing so early in the channel:


[08/18/14 13:34:20.040]:NEW-IDV-ALEPH Staff ST: Applying rule 'Add
workforceID to the XML stream if not available'.
[08/18/14 13:34:20.041]:NEW-IDV-ALEPH Staff ST: Action:
do-add-dest-attr-value("workforceID",token-src-attr("workforceID",notrace="true")).
[08/18/14 13:34:20.041]:NEW-IDV-ALEPH Staff ST:
arg-string(token-src-attr("workforceID",notrace="true"))
[08/18/14 13:34:20.041]:NEW-IDV-ALEPH Staff ST:
token-src-attr("workforceID",notrace="true")
[08/18/14 13:34:20.044]:NEW-IDV-ALEPH Staff ST: -- trace
suppressed --
[08/18/14 13:34:20.044]:NEW-IDV-ALEPH Staff ST: Arg Value: "217732".
[08/18/14 13:34:20.044]:NEW-IDV-ALEPH Staff ST:Policy returned:
[08/18/14 13:34:20.045]:NEW-IDV-ALEPH Staff ST:
<nds dtdversion="4.0" ndsversion="8.x">
<source>
<product edition="Standard" version="4.0.1.0">DirXML</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<sync cached-time="20140818123419.937Z" class-name="User"
event-id="isls-testidm1#20140818123419#99#1:1daac2fa-a363-4e78-73b8-fac2aa1d63a3"
qualified-src-dn="O=wmin\OU=staff\CN=abdurau"
src-dn="\TEST-WMIN-URS\wmin\staff\abdurau" src-entry-id="56203"
timestamp="0#0">
<association state="migrate"></association>
</sync>
<modify class-name="User"
event-id="isls-testidm1#20140818123419#99#1:1daac2fa-a363-4e78-73b8-fac2aa1d63a3"
qualified-src-dn="O=wmin\OU=staff\CN=abdurau"
src-dn="\TEST-WMIN-URS\wmin\staff\abdurau" src-entry-id="56203">
<modify-attr attr-name="workforceID">
<add-value>
<value>217732</value>
</add-value>
</modify-attr>
</modify>
</input>
</nds>


Notice that before this GateForUsers policy was applied you only had the
migrate event in play. At this point, though, the driver config checked
(incorrectly, I think; do this much later in life after you're officially
an add event) to ensure there was a workforceID and, of course, on a sync
there is not. That will be auto-added later on a sync/migrate, but your
driver config is explicitly adding it here, which is causing a second
event to be added to the operation document as I suggested earlier. In
this case, doing a migrate/sync, this check is redundant. In other cases
(regular modifies) it may not be so, but there are better times to
forcefully add attributes, such as the Command Transform policyset.

A few questions that may help resolve the root of the issue:
1. Do you need workforceID in the final document to be sent to your
application? I presume so since I see it in the z305-id field. If not,
and it is just added so that the association logic works later, then
remove this action adding the workforceID now as it is can be done
elsewhere with better results.

2. Is the normal way to cause events from a Migrate/Sync? Does the
current logic work during a regular change of a user, for example, when
the workforceID is initially added as an attribute, or the user is
initially created? I would guess it does because the GateForUsers rule
would, if necessary, add the workforceID to the current event and not to a
new one right after the current one in the same input document.

3. If nothing else can change, disable the 'Add workforceID to the XML
stream if not available' rule within the GateForUsers policy and then copy
that same rule down to the beginning of the Command Transformation
policyset (into a new policy if it does not make sense in an existing
policy). This same rule will probably work there nicely which means you
will not have a second event on a migrate, and you will also ensure that
workforceID is present in the final add document.

And in case it helps in the future, this was a clue that was needed to
better understand that there were multiple events in the single input
document:

[08/18/14 13:34:20.452]:NEW-IDV-ALEPH Staff ST:Re-reading associations in
case they were changed by previous event processing

Because there were two events and one was completely processed, the engine
tries to ensure that everything is updated in case an association was
added previously (to avoid duplicate queries and really odd states) and
this is the line indicating the update within the engine. After this
point you can see that the modify document has the association which came
from the previous operation, where before the modify did not have an
association.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Delimited Text driver re-processing after association


That's perfect, thanks ab.

I did see the line "[08/18/14 13:34:20.452]:NEW-IDV-ALEPH Staff
ST:Re-reading associations in
case they were changed by previous event processing"
and was very confused why it was doing that. I could see no rule that
was triggering that, but it makes sense that it has two events being
processed.

I think I have enough information now to tie down the behavior of the
driver rules and hopefully get them working as the business wants.

Thanks ever so much for your help. I was completely stuck
John


--
Jevans78
------------------------------------------------------------------------
Jevans78's Profile: https://forums.netiq.com/member.php?userid=7684
View this thread: https://forums.netiq.com/showthread.php?t=51549

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.