Highlighted
Super Contributor.. Super Contributor..
Super Contributor..
148 views

Scheduler Misfires

I am currently working on a project that reboots a number of Windows Servers.  I wouldnt say the flows contain a lot of steps. They are literally issue restart and check it came back up.

The flows are set to run against a worker group containing four RAS servers.

When the flows trigger only a third complete while the others dont and just remain in the scheduler with no previous run time.

On investigation of the logs the flows that didnt run and all show "There was a misfire in the scheduler"

I am trying to understand why this is occurring. Has anybody ever had this problem? How was it resolved?

I also read that default on misfire is "Do nothing". I am wondering if there is a bottle neck. Is there a setting for keep trying? Im thinking if it keeps trying once resource has freed it will run

Any help or advice anyone can offer 

 

0 Likes
7 Replies
Highlighted
Honored Contributor.
Honored Contributor.

Scheduler Misfires

Hi,

Personally, the only case where i have issue with scheduler was when i deployed a new flow.

The problem was that  when i scheduled it for eg at 03:00 so at 03:01 or 03:02 no execution launched.

When i tried to find the source, i found that i have an error in my flow.

Regards;

0 Likes
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Re: Scheduler Misfires

Thank you for your response.

I dont believe its an issue with the flow itself. Around a third of the scheduled flows trigger.

As I mentioned the flows that misfire stay in the scheduler. Then if I kick it off manually it completes.

Im leaning towards some sort of bottleneck.

I see on the schedule settings you can set what to do on misfire.

These flows are set to run once and on completion delete from the scheduler.

I see on the misfire options there is a "repeat indefinitely" option which I hope means what it says on the tin. If there is a misfire retry until it can run.

The problem I have is how do I set it using the schedule flow step. There is no input for this parameter

0 Likes
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Re: Scheduler Misfires

All the standard - you should engage support and understand fully the implications of making changes to your system from the internet etc...

This fix is likely unsupported by MicroFocus...

I made this and other changes to my environment for a similar issue...

You'll need to stop tomcat, unzip the OO Central jar in your tomcat/webapps directory. Remove the jar from the directory (otherwise it'll get unpacked on restart) 

Navigate to tomcat/webapps/oo/WEB-INF/lib
backup engine-webapp*.jar in a safe space outside of tomcat/  

using a tool such as 7zip right click and open 
navigate to META-INF/spring/standalone
edit engingeStandaloneQuartzContext.xml -> make the below changes-> save -- it should update the jar file -> restart central. If it doesnt work either restore this jar to your backup or restore the oo jar to the tomcat/webapps directory and delete the oo directory. On startup tomcat will expand the file automatically. If it does work make a note - every upgrade you'll need to make these updates again. 

In <props> (toward the bottom)

set my threadcount to 30
misfireThreshold 30000
clustercheckininterval 15000

and added the following in the <props> 

<prop key="org.quartz.jobStore.lockHandler.class">org.quartz.impl.jdbcjobstore.UpdateLockRowSemaphore</prop>

<prop key="org.quartz.jobStore.acquireTriggersWithinLock">true</prop>

<prop key="org.quartz.jobStore.txIsolationLevelSerializable">true</prop>

0 Likes
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Re: Scheduler Misfires

Following what is in the OO documentation

"By default, each Operations Orchestration node has 20 worker threads. If your flows have a large number of parallel or multi-instance lanes, or if you trigger a large number of flows simultaneously, we recommend increasing this number. For example, you might increase this number to 200 threads per worker or Central."

So we increased the threads but this time seen different behavior

More flows did complete, however some are not showing in Run Explorer. Whereas previously if there was a misfire the flow would remain in the scheduler. This time they didn't

The flows are designed to delete the schedule on successful completion which would suggest they completed but they aren't showing at all in the Run Explorer

Which is really bad as I have no idea if they ran, worked or failed

0 Likes
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Re: Scheduler Misfires

Your central wrapper server and execution logs should be able to assist on determining where your problem exists.
0 Likes
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Re: Scheduler Misfires

Yeah, had a look. I can see a whole bunch of misfires which also occurred last time before we increased the threads.

Didn't delete from the schedule previously though before.

That's as much as I can see with my knowledge and experience so have passed the logs to MF.

I wonder if the flows did complete and thats why they are deleted from the scheduler but if that is the case the run explorer is not showing them which is worrying. I was already concerned that when there is a misfire there is no notification unless you check the logs. This is just a guess though. Not good if you have a Business critical flow scheduled.

0 Likes
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Re: Scheduler Misfires

Agree. You might consider reworking the flows to trigger from a more manageable/ operationally friendly source... I.e. db tables with Params and status tracking - then launching individual run instances from a “Master” flow. That’s at least a very abstract version of what I do for my critical executions
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.