ohico Frequent Contributor.
Frequent Contributor.
777 views

IDM 4.x - Able to see Driver Cache entries in file system?

We have Oracle driver that has a very very slow latency problem ever since they update that DB view many months ago.
It still functions fine as there are only a dozen events from that driver each day.
** We tried to get them to fix it but nothing. They're now migrating to Postgres so that DB view's performance will hopefully improve**

Until then, the problem is that we need to synchronize that Oracle driver now and this results in thousands of "sync" events that are very slow to process.

I need to selectively delete many of the "sync" events that are not needed.

Do these exist somewhere on the file system such that I can stop the driver and selectively purge some of those events?
Is there a file/folder I can examine to manually modify this events list?

Thanks,
Marc
Labels (1)
0 Likes
15 Replies
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file system?

On 5/2/2019 7:14 AM, ohico wrote:
>
> We have Oracle driver that has a very very slow latency problem ever
> since they update that DB view many months ago.
> It still functions fine as there are only a dozen events from that
> driver each day.
> ** We tried to get them to fix it but nothing. They're now migrating to
> Postgres so that DB view's performance will hopefully improve**
>
> Until then, the problem is that we need to synchronize that Oracle
> driver now and this results in thousands of "sync" events that are very
> slow to process.
>
> I need to selectively delete many of the "sync" events that are not
> needed.
>
> Do these exist somewhere on the file system such that I can stop the
> driver and selectively purge some of those events?
> Is there a file/folder I can examine to manually modify this events
> list?


Sub Channel sync, is caused by one of two things. Events in the IDM
Driver cache (Stop the driver, iManager, IDM tools, Cache browser, and
you can see the events).

If you submit a Query as a command, (via dxcmd/Console2) the results of
the query, hit the Pub channel (Though confusingly, flagged as ST: for
sub thread) and the resulting <instnce> nodes become <sync> events.
(This is how imanager does Migrate into IDV, you GUI select the class,
attrs for the Query it submits).

Events queued up on the Oracle side, they are in the DB.



0 Likes
ohico Frequent Contributor.
Frequent Contributor.

Re: IDM 4.x - Able to see Driver Cache entries in file syste

I've found the .TAO file in the DIB folder "/var/opt/novell/eDirectory/data/dib/98227.TAO".
https://support.microfocus.com/kb/doc.php?id=3514553

As expected, it's also the larger TAO file (2.6MB) as it holds over 17,000 "sync" events but many thousands of these events can be deleted.
I know the count from the "Driver Cache Inspector" in iManager.

Is there any way outside of iManager's "Driver Cache Inspector" to modify this TAO file's "sync" events?
Maybe we can stop and disabled the driver and use some custom TAO file editor? Does that exist?

Thanks,
Marc
0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file system?

On 5/2/2019 8:34 AM, ohico wrote:
>
> I've found the .TAO file in the DIB folder
> "/var/opt/novell/eDirectory/data/dib/98227.TAO".
> https://support.microfocus.com/kb/doc.php?id=3514553
>
> As expected, it's also the larger TAO file (2.6MB) as it holds over
> 17,000 "sync" events but many thousands of these events can be deleted.
> I know the count from the "Driver Cache Inspector" in iManager.
>
> Is there any way outside of iManager's "Driver Cache Inspector" to
> modify this TAO file's "sync" events?
> Maybe we can stop and disabled the driver and use some custom TAO file
> editor? Does that exist?


Why yes, yes we can! Simplistically you can use dxcmd to export to
contents of the binary TAO file to an XML text file which you can then
process, and resubmit via dxcmd again.

Or you can get a GUI tool written by a fine fellow, who posts commonly
on this forum (Alekz) called Console2 that has a GUI to allow you to
export and import more easily. You still have to edit the file, butu
hey, alekz? That would be a cool feature! Throw in an XML editor that
would allow somem XPATH to remove nodes? That would be cool and useful!

Go get Console2 at http://sneakycat.biz (side not, I get no kickbacks
from this, why is that Alekz? I should get something for pushing you
tool, eh?)



0 Likes
ohico Frequent Contributor.
Frequent Contributor.

Re: IDM 4.x - Able to see Driver Cache entries in file syste

Great! I missed your post before deciding to open an SR.
I'm looking into this tool now.
Marc
0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file system?

On 5/2/2019 10:14 AM, ohico wrote:
>
> Great! I missed your post before deciding to open an SR.
> I'm looking into this tool now.


The author did a session for my IDM User Group showing off the tool a
week or three ago. I asked him to present because in a previous IDM
User Group session on new features in IDM 4.7.2 DirXML Script (Betcha
you do not know what those are! Everyone who watched does now!
Extensions to the Query token that are quite interesting) and the
developer from the Bangalore Dev center was doing it all via dxcmd and
text files, when he could have used a simple GUI tool to make the demo
easier and cooler.

Anyone interested in the IDM UG, please email me at:
geoffreycarman@gmail.com

I want to know your industry (Edu, Manu, Consulting, whatever) and ow
many user licenses you represent. (Consultants don't count). Then I
will add you to my mailing list for future sessions and give you access
to the recordings and notes from the past 26 events.


0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file syste

geoffc;2499138 wrote:
On 5/2/2019 8:34 AM, ohico wrote:
>
> I've found the .TAO file in the DIB folder
> "/var/opt/novell/eDirectory/data/dib/98227.TAO".
> https://support.microfocus.com/kb/doc.php?id=3514553
>
> As expected, it's also the larger TAO file (2.6MB) as it holds over
> 17,000 "sync" events but many thousands of these events can be deleted.
> I know the count from the "Driver Cache Inspector" in iManager.
>
> Is there any way outside of iManager's "Driver Cache Inspector" to
> modify this TAO file's "sync" events?
> Maybe we can stop and disabled the driver and use some custom TAO file
> editor? Does that exist?


Why yes, yes we can! Simplistically you can use dxcmd to export to
contents of the binary TAO file to an XML text file which you can then
process, and resubmit via dxcmd again.

Or you can get a GUI tool written by a fine fellow, who posts commonly
on this forum (Alekz) called Console2 that has a GUI to allow you to
export and import more easily. You still have to edit the file, butu
hey, alekz? That would be a cool feature! Throw in an XML editor that
would allow somem XPATH to remove nodes? That would be cool and useful!

Go get Console2 at http://sneakycat.biz (side not, I get no kickbacks
from this, why is that Alekz? I should get something for pushing you
tool, eh?)


Great minds think alike! (Russian version of the same sentence: "fools seldom differ"). 🙂
Alekz C2 was my first answer when I saw this question.

And I can see a number of possible (and very usable) C2 extensions that can help in similar situations:
1. Select/Deselect events based on the event type.
2. Select (search) based on object name (or more complex conditions).
0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file system?

On 5/3/2019 9:04 AM, al b wrote:
>
> geoffc;2499138 Wrote:
>> On 5/2/2019 8:34 AM, ohico wrote:
>>>
>>> I've found the .TAO file in the DIB folder
>>> "/var/opt/novell/eDirectory/data/dib/98227.TAO".
>>> https://support.microfocus.com/kb/doc.php?id=3514553
>>>
>>> As expected, it's also the larger TAO file (2.6MB) as it holds over
>>> 17,000 "sync" events but many thousands of these events can be

>> deleted.
>>> I know the count from the "Driver Cache Inspector" in iManager.
>>>
>>> Is there any way outside of iManager's "Driver Cache Inspector" to
>>> modify this TAO file's "sync" events?
>>> Maybe we can stop and disabled the driver and use some custom TAO

>> file
>>> editor? Does that exist?

>>
>> Why yes, yes we can! Simplistically you can use dxcmd to export to
>> contents of the binary TAO file to an XML text file which you can then
>> process, and resubmit via dxcmd again.
>>
>> Or you can get a GUI tool written by a fine fellow, who posts commonly
>> on this forum (Alekz) called Console2 that has a GUI to allow you to
>> export and import more easily. You still have to edit the file, butu
>> hey, alekz? That would be a cool feature! Throw in an XML editor that
>> would allow somem XPATH to remove nodes? That would be cool and
>> useful!
>>
>> Go get Console2 at http://sneakycat.biz (side not, I get no kickbacks
>> from this, why is that Alekz? I should get something for pushing you
>> tool, eh?)

>
> Great minds think alike! (Russian version of the same sentence: "fools
> seldom differ"). 🙂
> Alekz C2 was my first answer when I saw this question.
>
> And I can see a number of possible (and very usable) C2 extensions that
> can help in similar situations:
> 1. Select/Deselect events based on the event type.
> 2. Select (search) based on object name (or more complex conditions).


I emailed him just to be sure he sees it. 🙂 He actually has a day
job, but is really nice about this stuff.

Anyone interested in seeing C2 in action (go get a copy) or email me to
get access to my IDM User Group where Alekz did a session on Console2.

Plus the 26 other sessions on interesting IDM topics.


0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file syste

Hi,

Thanks for idea but I don't think C2 will implement any cache editor any time soon due to the time required.

It's quite easy to export the cache to XML and then use a XSLT stylesheet with Saxon (http://www.saxonica.com/download/java.xml) to manipulate the cache.

Using Saxon also gives you access to XSLT/XPath v2.0 and v3.0.

If I would build it in C2 you could only use the XML library that comes with Java and it only supports XSLT/XPath 1.0

Also it's much easier just to add a temporary policy on the subscriber event channel and veto the events you don't want 🙂
0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file system?

On 5/21/2019 7:24 AM, alekz wrote:
>
> Hi,
>
> Thanks for idea but I don't think C2 will implement any cache editor any
> time soon due to the time required.


Slacker! Thank you for considering it however. One day, when you might
be bored...

> It's quite easy to export the cache to XML and then use a XSLT
> stylesheet with Saxon (http://www.saxonica.com/download/java.xml) to
> manipulate the cache.
>
> Using Saxon also gives you access to XSLT/XPath v2.0 and v3.0.
>
> If I would build it in C2 you could only use the XML library that comes
> with Java and it only supports XSLT/XPath 1.0
>
> Also it's much easier just to add a temporary policy on the subscriber
> event channel and veto the events you don't want 🙂
>
>


0 Likes
TE Super Contributor.
Super Contributor.

Re: IDM 4.x - Able to see Driver Cache entries in file syste

geoffc;2499138 wrote:
On 5/2/2019 8:34 AM, ohico wrote:
>
> I've found the .TAO file in the DIB folder
> "/var/opt/novell/eDirectory/data/dib/98227.TAO".
> https://support.microfocus.com/kb/doc.php?id=3514553
>
> As expected, it's also the larger TAO file (2.6MB) as it holds over
> 17,000 "sync" events but many thousands of these events can be deleted.
> I know the count from the "Driver Cache Inspector" in iManager.
>
> Is there any way outside of iManager's "Driver Cache Inspector" to
> modify this TAO file's "sync" events?
> Maybe we can stop and disabled the driver and use some custom TAO file
> editor? Does that exist?


Why yes, yes we can! Simplistically you can use dxcmd to export to
contents of the binary TAO file to an XML text file which you can then
process, and resubmit via dxcmd again.

Or you can get a GUI tool written by a fine fellow, who posts commonly
on this forum (Alekz) called Console2 that has a GUI to allow you to
export and import more easily. You still have to edit the file, butu
hey, alekz? That would be a cool feature! Throw in an XML editor that
would allow somem XPATH to remove nodes? That would be cool and useful!

Go get Console2 at http://sneakycat.biz (side not, I get no kickbacks
from this, why is that Alekz? I should get something for pushing you
tool, eh?)


I thought you were getting a full 10% of what he makes off the tool!
0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file system?

On 5/15/2019 4:54 PM, tse7147 wrote:
>
> geoffc;2499138 Wrote:
>> On 5/2/2019 8:34 AM, ohico wrote:
>>>
>>> I've found the .TAO file in the DIB folder
>>> "/var/opt/novell/eDirectory/data/dib/98227.TAO".
>>> https://support.microfocus.com/kb/doc.php?id=3514553
>>>
>>> As expected, it's also the larger TAO file (2.6MB) as it holds over
>>> 17,000 "sync" events but many thousands of these events can be

>> deleted.
>>> I know the count from the "Driver Cache Inspector" in iManager.
>>>
>>> Is there any way outside of iManager's "Driver Cache Inspector" to
>>> modify this TAO file's "sync" events?
>>> Maybe we can stop and disabled the driver and use some custom TAO

>> file
>>> editor? Does that exist?

>>
>> Why yes, yes we can! Simplistically you can use dxcmd to export to
>> contents of the binary TAO file to an XML text file which you can then
>> process, and resubmit via dxcmd again.
>>
>> Or you can get a GUI tool written by a fine fellow, who posts commonly
>> on this forum (Alekz) called Console2 that has a GUI to allow you to
>> export and import more easily. You still have to edit the file, butu
>> hey, alekz? That would be a cool feature! Throw in an XML editor that
>> would allow somem XPATH to remove nodes? That would be cool and
>> useful!
>>
>> Go get Console2 at http://sneakycat.biz (side not, I get no kickbacks
>> from this, why is that Alekz? I should get something for pushing you
>> tool, eh?)

>
> I thought you were getting a full 10% of what he makes off the tool!


I was hoping for 16% personally.



0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file syste

ohico;2499133 wrote:
I've found the .TAO file in the DIB folder "/var/opt/novell/eDirectory/data/dib/98227.TAO".
https://support.microfocus.com/kb/doc.php?id=3514553

As expected, it's also the larger TAO file (2.6MB) as it holds over 17,000 "sync" events but many thousands of these events can be deleted.
I know the count from the "Driver Cache Inspector" in iManager.

Is there any way outside of iManager's "Driver Cache Inspector" to modify this TAO file's "sync" events?
Maybe we can stop and disabled the driver and use some custom TAO file editor? Does that exist?

Thanks,
Marc


If you can identify which <sync> events you want to process, or which ones you do not want to process, a quick and simple policy on the subscriber event transform could be used to quickly process and discard the unwanted events, allowing through only the ones you actually do want.
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file system?

I think it would be prudent to provide a trace of exactly what you are
seeing, level three (3) or higher, from start to finish of at least one
whole event. At first I thought this was a Publisher channel thing, but I
see Geoffrey responded as if it were related to the Subscriber channel.
Depending on where, and why, there could be a few things we could do to
prevent unnecessary events. Assuming this is a triggerless setup, you
should not have unnecessary sync events on the Publisher channel, but
maybe a bug is interfering and we can possibly help with that if we know
by seeing.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Knowledge Partner
Knowledge Partner

Re: IDM 4.x - Able to see Driver Cache entries in file system?

On 5/2/2019 8:27 AM, ab wrote:
> I think it would be prudent to provide a trace of exactly what you are
> seeing, level three (3) or higher, from start to finish of at least one


Agreed, trace is great!

> whole event. At first I thought this was a Publisher channel thing, but I
> see Geoffrey responded as if it were related to the Subscriber channel.


I agree I assumed. But the later responses do seem to confirm the events
are in the TAO on the sub channel.

> Depending on where, and why, there could be a few things we could do to
> prevent unnecessary events. Assuming this is a triggerless setup, you
> should not have unnecessary sync events on the Publisher channel, but
> maybe a bug is interfering and we can possibly help with that if we know
> by seeing.
>


0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.