Xavier-G Trusted Contributor.
Trusted Contributor.
1105 views

(UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

Why scheduling call home and inventory by scanner jobs does not have the expected effects, and why it hurts the discovery performance ?

First, let's explain the difference between the inventory discovery by scanner and regular jobs :

Normal jobs, like host connection by shell:

• are executed by the probe, from the start until the end. There is no pauses into the processing, and it keep the probe thread dealing with it busy.
• It definitely make sense to schedule them.

As opposite, the inventory discovery by scanner :

• Is split into steps (around 13 steps)
• The probe is executing all steps ones first, then all step 2, etc, until the end.
• The execution cycle of this job, is usually 14 days. It means that between 2 steps executions of a single job for a given node, we could observe 14 days at max
• The job is in a parking state, doing nothing, until the probe decide to wake it up (see below)
• This job should not be scheduled, it could create far too many problems, and would miss too many cis.

One good reason for not scheduling it, is that this job, as it will split steps, will need to contact at least 13 times the node (plus what we call parking steps, could make 200 connection attempts at the end).
Due to this, as most machines may get another ip in the meantime, we need a way to ensure that :

• Nodes can signal to UD that the ip was changed. This is the callhome mechanism.

As sometimes it could be a laptop, connected for a limited time, the probe need to react in a very fast way.

If the job is stopped due to scheduling, it will fail to react to call home event in a timely maneer, and may reconnected when the node is not there any longer, slowing down the whole discovery cycle.

The exact mechanism is the following :

• When a node receive a new ip, it signal itself to the probe using the call home mechanism
• This call home is received by the probe, which create a call home event for it.
• The job call home, process the event, to update the ip and move the inventory discovery by scanner from a “PARKING” status to “RUNNING”

This lead to the following points :

 The callhome job, MUST be active all the time, to receive events and process them in real time, to ensure there won’t be too much backlog for the next execution scheduled of inventory discovery by scanner.
 The callhome job does not contact any machine by itself, does not create any load, there is no reason to worry that it’s running, it’s designed this way.
 It’s strongly advised to NOT schedule inventory by scanner, which can’t be paused like other jobs, the only thing which will happen, is that it will further delay the discovery cycle, going from 14 day to 30+ very easily when paused.

Please hit the Kudo button, if you find this post useful.

9 Replies
Super Contributor.. RRose Super Contributor..
Super Contributor..

Re: (UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

Xavier,

Thanks for addressing this in a support tip as I have been seeing some performance issues where I've been getting a significant number of devices with delinquent scans.  I do have a couple questions as it pertains to "not scheduling the call home and inventory by scanner job".  I have the following settings.  How would you recommend modifying?

CallHome.PNGCall Home SettingInventorybyScanner.PNGInventory by Scanner Setting

0 Likes
John Goldstein Outstanding Contributor.
Outstanding Contributor.

Re: (UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

RRose, your images show that you are using an interval, which is good for these jobs. A schedule would be where the job was set to run only at predefined times. Since these jobs are long running asynchronous tasks it is best to not have periods where the job is paused waiting for the next activation window.

In my environment I use a Cron schedule for these jobs which works as well.

Call_Home_Processing.pngCall Home ProcessingInventory_Discovery_by_Scanner.pngInventory Discovery by Scanner

I some times have to stop and reschedule the Call Home Processing job because there are too many old Call Home Event CITs. Restarting the job and setting it to run at the next interval is usually enough to clear out the backlog.

My schedule is:
Call Home Processing - Once an hour on the hour
Inventory Discovery by Scanner - Once a week starting second day (Monday) of week at 1 minute past midnight

Once a month I will stop the Inventory by Scanner job on Friday and reschedule it to start on Sunday at 11:00 PM. That helps to reset the Discovery Progress statistics.

Xavier-G Trusted Contributor.
Trusted Contributor.

Re: (UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

Hi RRose,

Those settings are perfectly fine.

What's important, is to never change :

Allow discovery to run at: <<always>>

which would prevent the discovery to always run. It would prevent an efficient callhome handling.

About the "Invoke on new triggered cis immediately", that's also a key option, ie, call home won't wait 3 hours, they will start right away. the 3 hours setting, is just to ensure that there isn't any left over (probe clustering redispatch is a good example)

hope this clarify a little bit

0 Likes
Xavier-G Trusted Contributor.
Trusted Contributor.

Re: (UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

Hi John,

there is a few interesting points in your answer :

I some times have to stop and reschedule the Call Home Processing job because there are too many old Call Home Event CITs. Restarting the job and setting it to run at the next interval is usually enough to clear out the backlog

This does not look as a normal situation. The server should be able to keep up with the load, and process those events in a very fast way.

Could you confirm which version is currently used ? if it's a 10.22, please check that you are using at least the 10.22CUP6, as it improve strongly this kind of situation.

Once a month I will stop the Inventory by Scanner job on Friday and reschedule it to start on Sunday at 11:00 PM. That helps to reset the Discovery Progress statistics.

This is a kind of a problem too, it is reflecting more details about the progress, but stop cis which were progressing. It is suggested to not stop the job, and to wait until it finish, even if it looks stuck, over the year, it would improve the troughtput even if you can't see an immediate feedback.

hope this helps 😉

0 Likes
Xavier-G Trusted Contributor.
Trusted Contributor.

Re: (UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

Received an interesting question :

If the execution cycle for inventory by scanner is 14 days, on average, does that mean we shouldn’t expect to see new results from the same endpoint more than every other week? One of our customers have been expecting to have new scan files weekly to send to our asset management team.

Can we configure the Scanner to have a weekly interval? If so, it’s something you would recommend.

Absolutely, we only expect to see a node completed in 2 weeks, and nothing more.

All the timers of the workflow are designed with this 2 weeks duration in minds.

About getting all results over one week :

  1. this is not supporteded as is
  2. it would require extensive tunning over the workflow
  3. it would be extremely ressource intensive on the probe side (db/disk/cpu)

It is strongly suggested to not change the workflow timers, but to contact support/r&d with questions about why it may be interesting to change them for your use case.

Kind Regards.

Xavier

0 Likes
Trusted Contributor.. jkey Trusted Contributor..
Trusted Contributor..

Re: (UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

Thank you all for your input on this. It was a good read and helped me understand why our Inventory by Scanner job will be humming along and then seemingly come to a halt. 

We just recently switched from running this job through management zones on schedules, to the discovery module without scheduling restrictions. We're still running infrastructure templates in the management zones which includes the Call Home Processing job by default. 

My question is two part: We have not been using the Call Home Processing in discovery modules because it's been running in the MZ infrastructure jobs. If we turn this on as well, will there be a conflict between the discovery job (at 3 hour intervals) and the MZ jobs (at 3 day intervals)? If there will be a conflict, how can I turn off the MZ jobs?

Thanks,

Jeremiah

0 Likes
John Goldstein Outstanding Contributor.
Outstanding Contributor.

Re: (UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

Jeremiah

I would not run both Management Zone and Advanced jobs for the same job. The MZ will create a copy of the job with a unique name for each of the zones you have created. My advice would be to only run one Advanced job of Call Home Processing for all probes and edit the MZ activity to remove call home processing from your zones.

John

0 Likes
Super Contributor.. d_shavrin Super Contributor..
Super Contributor..

Re: (UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

Awesome and a very useful doc! Added to my bookmarks!

 

0 Likes
Highlighted
Trusted Contributor.. mlukezic Trusted Contributor..
Trusted Contributor..

Re: (UCMDB Support Tip) - callhome/inventory by scanner job scheduling slow down the discovery cycle

Without getting into the long history and reasons why... our Asset Management program requires very frequent scanning of desktop devices (less than a week) and the callhome feature has not worked out particularily well in our environment, so we have customized the parking thresholds to move the inventory scanning process along quicker.  Because of the short frequency, we pay close attention to the "scheduling" of discovery jobs (using interval and <<always>>) and also have 3 well-resourced probes dedicated to scanning our desktop environment.

Before modifying any parking thresholds, it is vital to have a good understanding of how they work and are configured.  Not sure what, if any, documentation exists around this.  Probably best to engage with support if you are considering it.

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.