IDM 4.x - Able to see Driver Cache entries in file system?

We have Oracle driver that has a very very slow latency problem ever since they update that DB view many months ago.
It still functions fine as there are only a dozen events from that driver each day.
** We tried to get them to fix it but nothing. They're now migrating to Postgres so that DB view's performance will hopefully improve**

Until then, the problem is that we need to synchronize that Oracle driver now and this results in thousands of "sync" events that are very slow to process.

I need to selectively delete many of the "sync" events that are not needed.

Do these exist somewhere on the file system such that I can stop the driver and selectively purge some of those events?
Is there a file/folder I can examine to manually modify this events list?

Thanks,
Marc
  • On 5/2/2019 7:14 AM, ohico wrote:
    >
    > We have Oracle driver that has a very very slow latency problem ever
    > since they update that DB view many months ago.
    > It still functions fine as there are only a dozen events from that
    > driver each day.
    > ** We tried to get them to fix it but nothing. They're now migrating to
    > Postgres so that DB view's performance will hopefully improve**
    >
    > Until then, the problem is that we need to synchronize that Oracle
    > driver now and this results in thousands of "sync" events that are very
    > slow to process.
    >
    > I need to selectively delete many of the "sync" events that are not
    > needed.
    >
    > Do these exist somewhere on the file system such that I can stop the
    > driver and selectively purge some of those events?
    > Is there a file/folder I can examine to manually modify this events
    > list?


    Sub Channel sync, is caused by one of two things. Events in the IDM
    Driver cache (Stop the driver, iManager, IDM tools, Cache browser, and
    you can see the events).

    If you submit a Query as a command, (via dxcmd/Console2) the results of
    the query, hit the Pub channel (Though confusingly, flagged as ST: for
    sub thread) and the resulting <instnce> nodes become <sync> events.
    (This is how imanager does Migrate into IDV, you GUI select the class,
    attrs for the Query it submits).

    Events queued up on the Oracle side, they are in the DB.



  • I think it would be prudent to provide a trace of exactly what you are
    seeing, level three (3) or higher, from start to finish of at least one
    whole event. At first I thought this was a Publisher channel thing, but I
    see Geoffrey responded as if it were related to the Subscriber channel.
    Depending on where, and why, there could be a few things we could do to
    prevent unnecessary events. Assuming this is a triggerless setup, you
    should not have unnecessary sync events on the Publisher channel, but
    maybe a bug is interfering and we can possibly help with that if we know
    by seeing.

    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below.

    If you want to send me a private message, please let me know in the
    forum as I do not use the web interface often.
  • I've found the .TAO file in the DIB folder "/var/opt/novell/eDirectory/data/dib/98227.TAO".
    https://support.microfocus.com/kb/doc.php?id=3514553

    As expected, it's also the larger TAO file (2.6MB) as it holds over 17,000 "sync" events but many thousands of these events can be deleted.
    I know the count from the "Driver Cache Inspector" in iManager.

    Is there any way outside of iManager's "Driver Cache Inspector" to modify this TAO file's "sync" events?
    Maybe we can stop and disabled the driver and use some custom TAO file editor? Does that exist?

    Thanks,
    Marc
  • On 5/2/2019 8:34 AM, ohico wrote:
    >
    > I've found the .TAO file in the DIB folder
    > "/var/opt/novell/eDirectory/data/dib/98227.TAO".
    > https://support.microfocus.com/kb/doc.php?id=3514553
    >
    > As expected, it's also the larger TAO file (2.6MB) as it holds over
    > 17,000 "sync" events but many thousands of these events can be deleted.
    > I know the count from the "Driver Cache Inspector" in iManager.
    >
    > Is there any way outside of iManager's "Driver Cache Inspector" to
    > modify this TAO file's "sync" events?
    > Maybe we can stop and disabled the driver and use some custom TAO file
    > editor? Does that exist?


    Why yes, yes we can! Simplistically you can use dxcmd to export to
    contents of the binary TAO file to an XML text file which you can then
    process, and resubmit via dxcmd again.

    Or you can get a GUI tool written by a fine fellow, who posts commonly
    on this forum (Alekz) called Console2 that has a GUI to allow you to
    export and import more easily. You still have to edit the file, butu
    hey, alekz? That would be a cool feature! Throw in an XML editor that
    would allow somem XPATH to remove nodes? That would be cool and useful!

    Go get Console2 at http://sneakycat.biz (side not, I get no kickbacks
    from this, why is that Alekz? I should get something for pushing you
    tool, eh?)



  • On 5/2/2019 8:27 AM, ab wrote:
    > I think it would be prudent to provide a trace of exactly what you are
    > seeing, level three (3) or higher, from start to finish of at least one


    Agreed, trace is great!

    > whole event. At first I thought this was a Publisher channel thing, but I
    > see Geoffrey responded as if it were related to the Subscriber channel.


    I agree I assumed. But the later responses do seem to confirm the events
    are in the TAO on the sub channel.

    > Depending on where, and why, there could be a few things we could do to
    > prevent unnecessary events. Assuming this is a triggerless setup, you
    > should not have unnecessary sync events on the Publisher channel, but
    > maybe a bug is interfering and we can possibly help with that if we know
    > by seeing.
    >


  • Thanks guys,

    I'm going to open a SR for assistance on this latency issue with that Oracle driver.

    Regards,
    Marc
  • Great! I missed your post before deciding to open an SR.
    I'm looking into this tool now.
    Marc
  • On 5/2/2019 10:14 AM, ohico wrote:
    >
    > Great! I missed your post before deciding to open an SR.
    > I'm looking into this tool now.


    The author did a session for my IDM User Group showing off the tool a
    week or three ago. I asked him to present because in a previous IDM
    User Group session on new features in IDM 4.7.2 DirXML Script (Betcha
    you do not know what those are! Everyone who watched does now!
    Extensions to the Query token that are quite interesting) and the
    developer from the Bangalore Dev center was doing it all via dxcmd and
    text files, when he could have used a simple GUI tool to make the demo
    easier and cooler.

    Anyone interested in the IDM UG, please email me at:
    geoffreycarman@gmail.com

    I want to know your industry (Edu, Manu, Consulting, whatever) and ow
    many user licenses you represent. (Consultants don't count). Then I
    will add you to my mailing list for future sessions and give you access
    to the recordings and notes from the past 26 events.


  • geoffc;2499138 wrote:
    On 5/2/2019 8:34 AM, ohico wrote:
    >
    > I've found the .TAO file in the DIB folder
    > "/var/opt/novell/eDirectory/data/dib/98227.TAO".
    > https://support.microfocus.com/kb/doc.php?id=3514553
    >
    > As expected, it's also the larger TAO file (2.6MB) as it holds over
    > 17,000 "sync" events but many thousands of these events can be deleted.
    > I know the count from the "Driver Cache Inspector" in iManager.
    >
    > Is there any way outside of iManager's "Driver Cache Inspector" to
    > modify this TAO file's "sync" events?
    > Maybe we can stop and disabled the driver and use some custom TAO file
    > editor? Does that exist?


    Why yes, yes we can! Simplistically you can use dxcmd to export to
    contents of the binary TAO file to an XML text file which you can then
    process, and resubmit via dxcmd again.

    Or you can get a GUI tool written by a fine fellow, who posts commonly
    on this forum (Alekz) called Console2 that has a GUI to allow you to
    export and import more easily. You still have to edit the file, butu
    hey, alekz? That would be a cool feature! Throw in an XML editor that
    would allow somem XPATH to remove nodes? That would be cool and useful!

    Go get Console2 at http://sneakycat.biz (side not, I get no kickbacks
    from this, why is that Alekz? I should get something for pushing you
    tool, eh?)


    Great minds think alike! (Russian version of the same sentence: "fools seldom differ"). :)
    Alekz C2 was my first answer when I saw this question.

    And I can see a number of possible (and very usable) C2 extensions that can help in similar situations:
    1. Select/Deselect events based on the event type.
    2. Select (search) based on object name (or more complex conditions).
  • On 5/3/2019 9:04 AM, al b wrote:
    >
    > geoffc;2499138 Wrote:
    >> On 5/2/2019 8:34 AM, ohico wrote:
    >>>
    >>> I've found the .TAO file in the DIB folder
    >>> "/var/opt/novell/eDirectory/data/dib/98227.TAO".
    >>> https://support.microfocus.com/kb/doc.php?id=3514553
    >>>
    >>> As expected, it's also the larger TAO file (2.6MB) as it holds over
    >>> 17,000 "sync" events but many thousands of these events can be

    >> deleted.
    >>> I know the count from the "Driver Cache Inspector" in iManager.
    >>>
    >>> Is there any way outside of iManager's "Driver Cache Inspector" to
    >>> modify this TAO file's "sync" events?
    >>> Maybe we can stop and disabled the driver and use some custom TAO

    >> file
    >>> editor? Does that exist?

    >>
    >> Why yes, yes we can! Simplistically you can use dxcmd to export to
    >> contents of the binary TAO file to an XML text file which you can then
    >> process, and resubmit via dxcmd again.
    >>
    >> Or you can get a GUI tool written by a fine fellow, who posts commonly
    >> on this forum (Alekz) called Console2 that has a GUI to allow you to
    >> export and import more easily. You still have to edit the file, butu
    >> hey, alekz? That would be a cool feature! Throw in an XML editor that
    >> would allow somem XPATH to remove nodes? That would be cool and
    >> useful!
    >>
    >> Go get Console2 at http://sneakycat.biz (side not, I get no kickbacks
    >> from this, why is that Alekz? I should get something for pushing you
    >> tool, eh?)

    >
    > Great minds think alike! (Russian version of the same sentence: "fools
    > seldom differ"). :)
    > Alekz C2 was my first answer when I saw this question.
    >
    > And I can see a number of possible (and very usable) C2 extensions that
    > can help in similar situations:
    > 1. Select/Deselect events based on the event type.
    > 2. Select (search) based on object name (or more complex conditions).


    I emailed him just to be sure he sees it. :) He actually has a day
    job, but is really nice about this stuff.

    Anyone interested in seeing C2 in action (go get a copy) or email me to
    get access to my IDM User Group where Alekz did a session on Console2.

    Plus the 26 other sessions on interesting IDM topics.