Anonymous_User Absent Member.
Absent Member.
380 views

AD driver stops for no apparent reason

Hello,

I recently migrated my AD connector. I moved the driver itself from an
IDM 4.0.2.7 engine server to IDM 4.6.3, and the RL from 4.0.2.7 on
Windows Server 2012 to 4.6.3 on Server 2016. Everything seems to work
fine, except that on two occasions, the driver just... stopped. Once it
happened exactly at midnight, the other time at 00:10.

Here's the trace (only level 1, unfortunately):


[04/02/19 00:10:00.115]:AD-OSUMC ST:Injecting User Agent XDS command
document into Subscriber channel.
[04/02/19 00:10:00.116]:AD-OSUMC ST:Applying command transformation
policies.
[04/02/19 00:10:00.116]:AD-OSUMC ST:Applying policy: %+C%14CCommand
Transform%-C.
[04/02/19 00:10:00.116]:AD-OSUMC ST:Applying policy:
%+C%14COSUMC-sub-ctp-ManagePasswordGroup%-C.
[04/02/19 00:10:00.116]:AD-OSUMC ST:Applying policy:
%+C%14CPassword(Sub)-Transform Distribution Password%-C.
[04/02/19 00:10:00.117]:AD-OSUMC ST:Applying policy:
%+C%14CPassword(Sub)-Default Password Policy%-C.
[04/02/19 00:10:00.117]:AD-OSUMC ST:Applying policy:
%+C%14CPassword(Sub)-Check Password GCV%-C.
[04/02/19 00:10:00.117]:AD-OSUMC ST:Applying policy:
%+C%14CPassword(Sub)-Add Password Payload%-C.
[04/02/19 00:10:00.117]:AD-OSUMC ST:Applying policy: %+C%14CPassword
Notify Ver1%-C.
[04/02/19 00:10:00.117]:AD-OSUMC ST:Filtering out notification-only
attributes.
[04/02/19 00:10:00.118]:AD-OSUMC ST:Fixing up association references.
[04/02/19 00:10:00.118]:AD-OSUMC ST:Applying schema mapping policies to
output.
[04/02/19 00:10:00.118]:AD-OSUMC ST:Applying policy:
%+C%14CSchemaMapping%-C.
[04/02/19 00:10:00.118]:AD-OSUMC ST:Applying output transformation policies.
[04/02/19 00:10:00.118]:AD-OSUMC ST:Applying policy: %+C%14COutput
Transform%-C.
[04/02/19 00:10:00.119]:AD-OSUMC ST:Applying policy:
%+C%14CPassword(Sub)-Pub Email Notifications%-C.
[04/02/19 00:10:00.119]:AD-OSUMC ST:Applying policy:
%+C%14COSUMC-otp-AttributeTranslations%-C.
[04/02/19 00:10:00.119]:AD-OSUMC ST:Remote Interface Driver: Sending...
[04/02/19 00:10:00.119]:AD-OSUMC ST:Remote Interface Driver: Document sent.
[04/02/19 00:10:00.121]:AD-OSUMC :Remote Interface Driver: Received.
[04/02/19 00:10:00.121]:AD-OSUMC :Remote Interface Driver: Received
document for subscriber channel
[04/02/19 00:10:00.121]:AD-OSUMC :Remote Interface Driver: Waiting for
receive...
[04/02/19 00:10:00.121]:AD-OSUMC ST:Applying input transformation policies.
[04/02/19 00:10:00.121]:AD-OSUMC ST:Applying policy: %+C%14CInput
Transform%-C.
[04/02/19 00:10:00.121]:AD-OSUMC ST:Applying XSLT policy:
%+C%14CInput+Transform+SS%-C.
[04/02/19 00:10:00.122]:AD-OSUMC ST:Applying policy:
%+C%14CPassword(Pub)-Sub Email Notifications%-C.
[04/02/19 00:10:00.122]:AD-OSUMC ST:Applying policy: %+C%14CVeto Delete
User%-C.
[04/02/19 00:10:00.122]:AD-OSUMC ST:Applying policy:
%+C%14COSUMC-itp-AttributeTranslations%-C.
[04/02/19 00:10:00.122]:AD-OSUMC ST:Applying schema mapping policies to
input.
[04/02/19 00:10:00.123]:AD-OSUMC ST:Applying policy:
%+C%14CSchemaMapping%-C.
[04/02/19 00:10:00.123]:AD-OSUMC ST:Resolving association references.
[04/02/19 00:10:00.223]:AD-OSUMC ST:Received state change event.
[04/02/19 00:10:00.224]:AD-OSUMC ST:Transitioned from state
'%+C%14CRunning%-C' to state '%+C%14CShutdown Pending%-C'.
[04/02/19 00:10:00.224]:AD-OSUMC ST:Successfully processed state change
event.
[04/02/19 00:10:00.224]:AD-OSUMC ST:Leaving event loop.
[04/02/19 00:10:00.224]:AD-OSUMC ST:Shutting down DirXML driver
\ID1\OSUMC\Drivers\edirIDv1\AD-OSUMC.
[04/02/19 00:10:00.225]:AD-OSUMC ST:No shutdown policies.
[04/02/19 00:10:00.225]:AD-OSUMC ST:Remote Interface Driver: Sending...
[04/02/19 00:10:00.225]:AD-OSUMC ST:Remote Interface Driver: Document sent.
[04/02/19 00:10:00.248]:AD-OSUMC :Remote Interface Driver: Received.
[04/02/19 00:10:00.248]:AD-OSUMC :Remote Interface Driver: Received
document for subscriber channel
[04/02/19 00:10:00.248]:AD-OSUMC :Remote Interface Driver: Waiting for
receive...
[04/02/19 00:10:00.248]:AD-OSUMC ST:Remote Interface Publisher: Received
shutdown.
[04/02/19 00:10:00.249]:AD-OSUMC ST:Remote Interface Driver: Closing
connection...
[04/02/19 00:10:00.250]:AD-OSUMC :Remote Interface Driver: Closing
connection...
[04/02/19 00:10:00.250]:AD-OSUMC :Remote Interface Driver: Connection closed
[04/02/19 00:10:00.250]:AD-OSUMC PT:Remote Interface Driver: Closing
connection...
[04/02/19 00:10:00.250]:AD-OSUMC ST:Remote Interface Driver: Connection
closed
[04/02/19 00:10:00.250]:AD-OSUMC PT:Remote Interface Driver: Connection
closed
[04/02/19 00:10:00.250]:AD-OSUMC ST:Remote Interface Subscriber:
Received shutdown.
[04/02/19 00:10:00.251]:AD-OSUMC PT:Applying input transformation policies.
[04/02/19 00:10:00.251]:AD-OSUMC ST:Reading XML attribute
vnd.nds.stream://ID1/OSUMC/Drivers/edirIDv1/AD-OSUMC#DirXML-DriverStorage.
[04/02/19 00:10:00.251]:AD-OSUMC PT:Applying policy: %+C%14CInput
Transform%-C.
[04/02/19 00:10:00.252]:AD-OSUMC ST:Writing driver state.
[04/02/19 00:10:00.252]:AD-OSUMC PT:Applying XSLT policy:
%+C%14CInput+Transform+SS%-C.
[04/02/19 00:10:00.252]:AD-OSUMC ST:Writing XML attribute
vnd.nds.stream://ID1/OSUMC/Drivers/edirIDv1/AD-OSUMC#DirXML-DriverStorage.
[04/02/19 00:10:00.252]:AD-OSUMC PT:Applying policy:
%+C%14CPassword(Pub)-Sub Email Notifications%-C.
[04/02/19 00:10:00.253]:AD-OSUMC PT:Applying policy: %+C%14CVeto Delete
User%-C.
[04/02/19 00:10:00.253]:AD-OSUMC PT:Applying policy:
%+C%14COSUMC-itp-AttributeTranslations%-C.
[04/02/19 00:10:00.253]:AD-OSUMC PT:Applying schema mapping policies to
input.
[04/02/19 00:10:00.253]:AD-OSUMC PT:Applying policy:
%+C%14CSchemaMapping%-C.
[04/02/19 00:10:00.254]:AD-OSUMC PT:Resolving association references.
[04/02/19 00:10:00.254]:AD-OSUMC PT:Ending publisher thread.
[04/02/19 00:10:00.264]:AD-OSUMC ST:Waiting for Publisher thread to
terminate. Maximum wait time is 60 seconds.
[04/02/19 00:10:00.264]:AD-OSUMC ST:Publisher thread terminated.
[04/02/19 00:10:00.265]:AD-OSUMC ST:Driver terminated.
[04/02/19 00:10:00.265]:AD-OSUMC ST:Writing XML attribute
vnd.nds.stream://ID1/OSUMC/Drivers/edirIDv1/AD-OSUMC#DirXML-PersistentData.

Short of cranking up the trace level to 3 and waiting for it to happen
again, is there any way to find out why the driver stopped unexpectedly?
Could it have anything to do with that "User agent XDS command document"
statement?

Thanks

Labels (1)
0 Likes
15 Replies
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: AD driver stops for no apparent reason

If I did not know better, I'd guess you turned off the driver.

On 04/02/2019 02:15 PM, 6423241 wrote:
> [04/02/19 00:10:00.123]:AD-OSUMC ST:Resolving association references.
> [04/02/19 00:10:00.223]:AD-OSUMC ST:Received state change event.
> [04/02/19 00:10:00.224]:AD-OSUMC ST:Transitioned from state
> '%+C%14CRunning%-C' to state '%+C%14CShutdown Pending%-C'.


Considering this came after some injection of a user doc, do you have
health configuration on this driver, object, and do you have health jobs
running? Perhaps you reached a health state with an action that shuts
down the driver config object, in which case it is working as designed.
That's not normal, of course, but it's the only thing that seems likely
without outside jobs doing the same thing.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: AD driver stops for no apparent reason

ab,

> If I did not know better, I'd guess you turned off the driver.
>
> On 04/02/2019 02:15 PM, 6423241 wrote:
>> [04/02/19 00:10:00.123]:AD-OSUMC ST:Resolving association references.
>> [04/02/19 00:10:00.223]:AD-OSUMC ST:Received state change event.
>> [04/02/19 00:10:00.224]:AD-OSUMC ST:Transitioned from state
>> '%+C%14CRunning%-C' to state '%+C%14CShutdown Pending%-C'.

>
> Considering this came after some injection of a user doc, do you have
> health configuration on this driver, object, and do you have health jobs
> running? Perhaps you reached a health state with an action that shuts
> down the driver config object, in which case it is working as designed.
> That's not normal, of course, but it's the only thing that seems likely
> without outside jobs doing the same thing.
>


I do have a driver health job running. The health configuration for this
driver is as follows:

Green condition group 1: driver is running and
unprocessed transactions <= 30
OR
condition group 2: available history is <= 10 minutes

Yellow condition group 1: unprocessed transactions > 30 or oldest
unprocessed transaction is older than 10 minutes
AND
condition group 2: unprocessed transactions < 50 or oldest
transaction is newer than 30 minutes
Action: Restart the driver and write a trace message

Red
Action: Page on-call person, send email alert, write trace
message

This configuration worked fine on the old server. Now it seems to be
stopping the driver but not restarting it when the condition is yellow.
Yesterday I added a line that restarts the driver when the condition is
red, and that one seems to behave correctly.

I'll try some more tweaks to the health configuration.

Thanks
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: AD driver stops for no apparent reason

On 04/03/2019 06:33 AM, 6423241 wrote:
>
> Yellow condition group 1: unprocessed transactions > 30 or oldest
> unprocessed transaction is older than 10 minutes
> AND
> condition group 2: unprocessed transactions < 50 or oldest
> transaction is newer than 30 minutes
> Action: Restart the driver and write a trace message


I have seen this option before, but I've never thought of a great reason
to ALWAYS do it (vs. having health stuff send an e-mail and let somebody
review things before restarting the driver config object). Restarting
does not fix anything, it just stops and starts, and that seems like a
waste of time or a way to incorrectly lead to a false sense of security in
your case since, after starting, you now have less than ten (10) minutes
of history, meaning your driver config's status will be green for the next
ten (10) minutes.

Perhaps you can explain how this works in your environment, as I have not
seen them all for sure.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: AD driver stops for no apparent reason

On 4/3/2019 08:39, ab wrote:
> On 04/03/2019 06:33 AM, 6423241 wrote:
>>
>> Yellow condition group 1: unprocessed transactions > 30 or oldest
>> unprocessed transaction is older than 10 minutes
>> AND
>> condition group 2: unprocessed transactions < 50 or oldest
>> transaction is newer than 30 minutes
>> Action: Restart the driver and write a trace message

>
> I have seen this option before, but I've never thought of a great reason
> to ALWAYS do it (vs. having health stuff send an e-mail and let somebody
> review things before restarting the driver config object). Restarting
> does not fix anything, it just stops and starts, and that seems like a
> waste of time or a way to incorrectly lead to a false sense of security in
> your case since, after starting, you now have less than ten (10) minutes
> of history, meaning your driver config's status will be green for the next
> ten (10) minutes.
>
> Perhaps you can explain how this works in your environment, as I have not
> seen them all for sure.
>


I've had situations in the past where the driver just stopped
communicating, and restarting it cleared the issue. That was my reason
for configuring this action. I used to get an email about once a week
telling me that the connector had been restarted the night before,
although I have not seen one for several months. Now that the driver has
been migrated, I don't get the 'driver restarted' email, I just get the
"DRIVER IS DOWN -- PANIC!!!" page instead.










0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: AD driver stops for no apparent reason

Well that's weird; restarting the driver config object shouldn't fix
anything that won't fix itself. The driver config automatically retries
connections when there are problems that cause a disconnect (e.g. the
remote application side goes down, or the Remote Loader (RL) restarts, or
something). Anyway, restarting should probably restart if specified, so
maybe there's a new bug there (since I've never seen anybody use Restart
like you do), but the use of restart itself is also invalid, in my
opinion. Take it or leave it, I guess. 🙂

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
6423241 Respected Contributor.
Respected Contributor.

Re: AD driver stops for no apparent reason

Leaving aside the failure to restart, I just don't understand why or how the driver would shut down -- almost always at midnight -- without any indication as to a reason in either the driver trace or the remote loader trace.

I'll keep stumbling on, I guess.

Thanks
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: AD driver stops for no apparent reason

On 04/03/2019 12:04 PM, 6423241 wrote:
>
> Leaving aside the failure to restart, I just don't understand why or how
> the driver would shut down -- almost always at midnight -- without any
> indication as to a reason in either the driver trace or the remote
> loader trace.
>
> I'll keep stumbling on, I guess.


The health job can have a trace file too, so perhaps see if that has a
corresponding near-midnight entry for running. That trace level can be
set to three (3) even if for some reason your real driver's trace level
cannot (though I think you should, as it might tell you the source of th
shutdown).

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: AD driver stops for no apparent reason

ab,
>
> The health job can have a trace file too, so perhaps see if that has a
> corresponding near-midnight entry for running. That trace level can be
> set to three (3) even if for some reason your real driver's trace level
> cannot (though I think you should, as it might tell you the source of th
> shutdown).
>


I configured a trace for the driver health job, but it doesn't seem to
be capturing much even at level 3. I'll check again tomorrow.

Last night I logged on at midnight to see what it was doing. The
connector was up and looked fine ... until 00:15, which was when the
driver health job launched. At that precise moment, the driver unloaded
itself. So it's the driver health job that's stopping the driver?!
That's like getting a disease from the thermometer you used to check to
see if you're sick.

I removed the command to restart the driver from the 'condition yellow'
action list, and set it to page me instead. Fingers crossed.

Thanks

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: AD driver stops for no apparent reason

On 04/04/2019 12:11 PM, 6423241 wrote:
>
> I configured a trace for the driver health job, but it doesn't seem to be
> capturing much even at level 3. I'll check again tomorrow.


Mostly I was looking for a way to verify that the timestamps were very
close between the health job running and the trace of something stopping.
I can never remember if the actual health logic is shown in the driver
config trace or the health job trace, but either way you need level three
(3) traces.

> Last night I logged on at midnight to see what it was doing. The connector
> was up and looked fine ... until 00:15, which was when the driver health
> job launched. At that precise moment, the driver unloaded itself. So
> it's the driver health job that's stopping the driver?! That's like


Seems like it.

> getting a disease from the thermometer you used to check to see if you're
> sick.


Folks can still get diseases from used needles, or probably thermometers
if they are not cleaned.

Analogies aside, maybe the deployed driver indicates the driver config
should be stopped, not restarted, and you are looking in Designer without
seeing the deployed version is different; admittedly that is unlikely.
Maybe there is a bug in the code that does not properly start the driver
again, or maybe it's a timing issue, and since your driver stops slowly
(compared to what the code expects) the start command is lost in the
middle of the shutdown, leaving you with only a stopped driver object.
I'm just speculating, obviously.

> I removed the command to restart the driver from the 'condition yellow'
> action list, and set it to page me instead. Fingers crossed.


Thanks for working through this here; the results will be interesting
either way.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Knowledge Partner
Knowledge Partner

Re: AD driver stops for no apparent reason

6423241;2497839 wrote:
ab,
>
> The health job can have a trace file too, so perhaps see if that has a
> corresponding near-midnight entry for running. That trace level can be
> set to three (3) even if for some reason your real driver's trace level
> cannot (though I think you should, as it might tell you the source of th
> shutdown).
>


I configured a trace for the driver health job, but it doesn't seem to
be capturing much even at level 3. I'll check again tomorrow.

Last night I logged on at midnight to see what it was doing. The
connector was up and looked fine ... until 00:15, which was when the
driver health job launched. At that precise moment, the driver unloaded
itself. So it's the driver health job that's stopping the driver?!
That's like getting a disease from the thermometer you used to check to
see if you're sick.

I removed the command to restart the driver from the 'condition yellow'
action list, and set it to page me instead. Fingers crossed.

Thanks


Guessing here, but doing time calculations around midnight can be interesting to get right when the day underflows. Maybe it's a bug in how they're calculating the recent activity.

Personally, I agree with Aaron. Auto-restart isn't something I would do.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: AD driver stops for no apparent reason

Mystery solved. The driver health job squawked again last night. When I
reviewed the health job trace, I found that it's actually triggered by a
yellow or red condition, but by a custom state:


Checking driver health for '<driver name>' on server '<server name>'
[04/08/19 00:15:00.171]:AD Driver Health JT:: evaluating state: green
[04/08/19 00:15:00.172]:AD Driver Health JT:: current health state:
green
[04/08/19 00:15:00.172]:AD Driver Health JT:: evaluating state:
custom-state
[04/08/19 00:15:00.173]:AD Driver Health JT:: Trying to execute
stop driver action
[04/08/19 00:15:00.381]:AD Driver Health JT:: Trying to execute
send-email
[04/08/19 00:15:00.580]:AD Driver Health JT:


The health configuration for this driver has a Custom State defined. The
trigger and action for it is "if number of deletes (as subscriber
reported events) is greater than 20 in the last 10 minutes, issue an
alert and stop the driver." I guess it's there as a fail-safe in case
rogue code starts deleting user accounts willy-nilly. Sounds like a
good thing to have, though it's apparently a bit too restrictive.
I'll try tweaking the trigger values for it.

Thanks as always for the helpful tips and good advice.

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: AD driver stops for no apparent reason

That can make sense, sure.

You may want to add a trace message before that shutdown, so next time you
look at it you know why it stopped or else also send an e-mail with the
event (stopped driver config) and why (lots of deletes).

Thank-you for sharing your results here.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: AD driver stops for no apparent reason

ab,

> That can make sense, sure.
>
> You may want to add a trace message before that shutdown, so next time you
> look at it you know why it stopped or else also send an e-mail with the
> event (stopped driver config) and why (lots of deletes).
>



I'm still seeing this issue sporadically, even after changing the
trigger to 40 deletions in a five minute period. I have checked, and
there weren't 40 user objects deleted from AD in the entire hour
preceding the triggered driver shutdown.

Someone did tell me that an AD object move is seen as a deletion, so
maybe that's what's happening here. What if I change the triggering
criteria to "<n> number of deletions *as publisher commands*"(instead of
subscriber reported events). Might that be a more trustworthy way of
capturing actual AD user account deletions?

I'm curious: does anyone else have a 'fail safe' rule like this in their
IDM environment?



0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: AD driver stops for no apparent reason

On 04/19/2019 12:09 PM, 6423241 wrote:
> ab,
>
> I'm still seeing this issue sporadically, even after changing the trigger
> to 40 deletions in a five minute period. I have checked, and there weren't
> 40 user objects deleted from AD in the entire hour preceding the triggered
> driver shutdown.


You said these were Subscriber events earlier, which makes more sense to
me. Subscriber events are not those from an application (including
microsoft active directory (MAD)) so I do not think deletes FROM there
should be considered, at least based on how you described it.

> Someone did tell me that an AD object move is seen as a deletion, so maybe
> that's what's happening here. What if I change the triggering criteria to
> "<n> number of deletions *as publisher commands*"(instead of subscriber
> reported events). Might that be a more trustworthy way of capturing actual
> AD user account deletions?


This is still talking about the Publisher, not Subscriber, channel.

> I'm curious: does anyone else have a 'fail safe' rule like this in their
> IDM environment?


I've seen similar rules, sure, but the threshold was typically higher,
like 100, or 1000, depending on what is normal for an environment. For
example if you normally delete only ten (10) users per day, having a
threshold number up at forty (40) might make sense, but you should
definitely get a notification about the stop in addition to stopping things.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.