Delimited Text driver re-processing after association


Hi,

I'm working on fixing a Delimted text (Aleph) driver which seems to be
re-processing an event on the subscriber channel and i'm not sure why.

Currently the driver is used to create an XML document with user
attributes depending on what Vault attributes are present.

There is a rule in the command Transformation which creates an
association using the WorkforceID and this works correctly, however once
the association is made the Command Transform then re-runs through all
the rules, so the resultant file it submits has the user's details as an
<Add> and appended below it has the details as a <Modify>. The expected
behavior is to have it process it as an <add>, then submit the document
and stop there.

The rules are all working correctly, I'm just not sure why the Command
Transform re-runs when an association is made.

I've tried moving the Add association rule to the Output Transform but
it then no longer works and all events that come through as an Add even
when there is an existing association.

I can try to post a level 3 trace if needed, but I was hoping to get a
higher level understanding of why the Command Transformation
re-processes all its rules if an association is made there.

Any help would be greatly appreciated.

Thanks


--
Jevans78
------------------------------------------------------------------------
Jevans78's Profile: https://forums.netiq.com/member.php?userid=7684
View this thread: https://forums.netiq.com/showthread.php?t=51549

  • On 08/18/2014 04:37 AM, Jevans78 wrote:
    >
    > I'm working on fixing a Delimted text (Aleph) driver which seems to be
    > re-processing an event on the subscriber channel and i'm not sure why.


    What do you mean by reprocessing? Is there just one event at this point,
    or are there actually two events in the current operation document and so
    the rules are just running for both of them? This would be clear in a
    trace as the policies calls out to which event the rules are being applied.

    > Currently the driver is used to create an XML document with user
    > attributes depending on what Vault attributes are present.
    >
    > There is a rule in the command Transformation which creates an
    > association using the WorkforceID and this works correctly, however once


    Why would you do this in the Command Transformation policyset? This may
    be part of the issue; normally this should happen in the Matching
    policyset, though normally setting up an association with the Delimited
    Text shim does not happen at all since querying the application for
    anything doesn't work unless you are using something like the Generic File
    shim (which is a CoolSolution that is really great).

    > the association is made the Command Transform then re-runs through all
    > the rules, so the resultant file it submits has the user's details as an
    > <Add> and appended below it has the details as a <Modify>. The expected


    This may be normal; after all, you have an association on the event for
    some reason, and an association on the Subscriber channel implies that the
    event is NOT an add (otherwise you would never have an association).

    > I've tried moving the Add association rule to the Output Transform but
    > it then no longer works and all events that come through as an Add even
    > when there is an existing association.


    Just to find out more about the business case of this driver, why add an
    association? Is it just to prevent subsequent events from coming through
    as an add, or to somehow let deletes eventually go through?

    > I can try to post a level 3 trace if needed, but I was hoping to get a
    > higher level understanding of why the Command Transformation
    > re-processes all its rules if an association is made there.


    A trace would be helpful. Understanding exactly what you're seeing, even
    with a good description like this, will be much easier if the trace is
    available.

    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...

  • Many thanks for your fast response.

    A full trace for the event is on pastebin:
    http://pastebin.com/8LjGWe5n

    I'm not sure on the business logic for associating in the command
    transform, only that if a user has already been processed by the driver
    once that only certain attributes should be entered into the Doc at the
    end.

    If the association event is removed, then the event comes through as an
    add and the Doc is created correctly, however it always does this.
    I believe the association rule was put in to ensure that future changes
    coming through the driver come through as a modify event. I agree the
    positioning of this rule is wrong, but if it is in the matching policy
    then it would become associated before all the Command Transform rules
    run and be always seen as a modify event in respect of creating the XML
    document.

    Thanks for your help on this


    --
    Jevans78
    ------------------------------------------------------------------------
    Jevans78's Profile: https://forums.netiq.com/member.php?userid=7684
    View this thread: https://forums.netiq.com/showthread.php?t=51549

  • Okay, a quick look shows the following:

    This trace is apparently where the problem exists only, which is what was
    requested. If you could post a trace of the case where you are not adding
    an association that would be interesting; presumably in that case you ONLY
    see one application of the Command Transformation policyset's policies, at
    least per your description of the issue.

    The problem in your case, I think, is happening much earlier in your
    driver config. Adding the association where you are doing so may not be
    terrible (though I'd probably try to handle that on the returning success
    status) but merely adding that association there is not causing the second
    event, and instead you are explicitly doing so early in the channel:


    [08/18/14 13:34:20.040]:NEW-IDV-ALEPH Staff ST: Applying rule 'Add
    workforceID to the XML stream if not available'.
    [08/18/14 13:34:20.041]:NEW-IDV-ALEPH Staff ST: Action:
    do-add-dest-attr-value("workforceID",token-src-attr("workforceID",notrace="true")).
    [08/18/14 13:34:20.041]:NEW-IDV-ALEPH Staff ST:
    arg-string(token-src-attr("workforceID",notrace="true"))
    [08/18/14 13:34:20.041]:NEW-IDV-ALEPH Staff ST:
    token-src-attr("workforceID",notrace="true")
    [08/18/14 13:34:20.044]:NEW-IDV-ALEPH Staff ST: -- trace
    suppressed --
    [08/18/14 13:34:20.044]:NEW-IDV-ALEPH Staff ST: Arg Value: "217732".
    [08/18/14 13:34:20.044]:NEW-IDV-ALEPH Staff ST:Policy returned:
    [08/18/14 13:34:20.045]:NEW-IDV-ALEPH Staff ST:
    <nds dtdversion="4.0" ndsversion="8.x">
    <source>
    <product edition="Standard" version="4.0.1.0">DirXML</product>
    <contact>Novell, Inc.</contact>
    </source>
    <input>
    <sync cached-time="20140818123419.937Z" class-name="User"
    event-id="isls-testidm1#20140818123419#99#1:1daac2fa-a363-4e78-73b8-fac2aa1d63a3"
    qualified-src-dn="O=wmin\OU=staff\CN=abdurau"
    src-dn="\TEST-WMIN-URS\wmin\staff\abdurau" src-entry-id="56203"
    timestamp="0#0">
    <association state="migrate"></association>
    </sync>
    <modify class-name="User"
    event-id="isls-testidm1#20140818123419#99#1:1daac2fa-a363-4e78-73b8-fac2aa1d63a3"
    qualified-src-dn="O=wmin\OU=staff\CN=abdurau"
    src-dn="\TEST-WMIN-URS\wmin\staff\abdurau" src-entry-id="56203">
    <modify-attr attr-name="workforceID">
    <add-value>
    <value>217732</value>
    </add-value>
    </modify-attr>
    </modify>
    </input>
    </nds>


    Notice that before this GateForUsers policy was applied you only had the
    migrate event in play. At this point, though, the driver config checked
    (incorrectly, I think; do this much later in life after you're officially
    an add event) to ensure there was a workforceID and, of course, on a sync
    there is not. That will be auto-added later on a sync/migrate, but your
    driver config is explicitly adding it here, which is causing a second
    event to be added to the operation document as I suggested earlier. In
    this case, doing a migrate/sync, this check is redundant. In other cases
    (regular modifies) it may not be so, but there are better times to
    forcefully add attributes, such as the Command Transform policyset.

    A few questions that may help resolve the root of the issue:
    1. Do you need workforceID in the final document to be sent to your
    application? I presume so since I see it in the z305-id field. If not,
    and it is just added so that the association logic works later, then
    remove this action adding the workforceID now as it is can be done
    elsewhere with better results.

    2. Is the normal way to cause events from a Migrate/Sync? Does the
    current logic work during a regular change of a user, for example, when
    the workforceID is initially added as an attribute, or the user is
    initially created? I would guess it does because the GateForUsers rule
    would, if necessary, add the workforceID to the current event and not to a
    new one right after the current one in the same input document.

    3. If nothing else can change, disable the 'Add workforceID to the XML
    stream if not available' rule within the GateForUsers policy and then copy
    that same rule down to the beginning of the Command Transformation
    policyset (into a new policy if it does not make sense in an existing
    policy). This same rule will probably work there nicely which means you
    will not have a second event on a migrate, and you will also ensure that
    workforceID is present in the final add document.

    And in case it helps in the future, this was a clue that was needed to
    better understand that there were multiple events in the single input
    document:

    [08/18/14 13:34:20.452]:NEW-IDV-ALEPH Staff ST:Re-reading associations in
    case they were changed by previous event processing

    Because there were two events and one was completely processed, the engine
    tries to ensure that everything is updated in case an association was
    added previously (to avoid duplicate queries and really odd states) and
    this is the line indicating the update within the engine. After this
    point you can see that the modify document has the association which came
    from the previous operation, where before the modify did not have an
    association.

    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...

  • That's perfect, thanks ab.

    I did see the line "[08/18/14 13:34:20.452]:NEW-IDV-ALEPH Staff
    ST:Re-reading associations in
    case they were changed by previous event processing"
    and was very confused why it was doing that. I could see no rule that
    was triggering that, but it makes sense that it has two events being
    processed.

    I think I have enough information now to tie down the behavior of the
    driver rules and hopefully get them working as the business wants.

    Thanks ever so much for your help. I was completely stuck
    John


    --
    Jevans78
    ------------------------------------------------------------------------
    Jevans78's Profile: https://forums.netiq.com/member.php?userid=7684
    View this thread: https://forums.netiq.com/showthread.php?t=51549