Many CallHomeEvents

Hello:

Can someone help us?

We have found a few days ago many (~8000) CallHomeEvents which are not processed/deleted by the CallHome Processing job. The CallHomeEvents have "Created Time" a few days ago.

If we manually deactivate and activate the job, the CallHomeEvents are processed, but after a few days we have again many CallhomeEvents.

We have only one CallHome Processing job for all probes and the scheduler is every hour (0 0 0/1 * * ?), and we have check the "Invoke on New Triggered CIs Immediately", and discovery to run at "always"

chuchi_1-1580199941472.png

 

chuchi_0-1580199908782.png

 

Why the CallhomeEvents are not processing in time by the job?

 

Thanks.

 

  • What version of UCMDB, Content Pack and Patch?

    How many call home URL do you have configured? One, two or three servers?

  • Universal CMDB 2019.02

    UCMDB Server 11.4.125

    CP 2019.11.102

    We have configure only one callhome URL (CallHomeURL0= x.x.x.x:80) in our clients.

    chuchi_0-1580306123826.png

     

  • I've seen such behavior in many installations of UCMDB. CallHomeEvents after some point stop being properly processed and deleted.

    Petko

  •   I was having the same issue with UCMDB 10.33 CUP 3 CP29 and it was related to broken gates on the TQL calculation. I received hot fix HF_QCIM8D98775.

    This hot fix should be included in your current version but you may want to ask support. Or check your error log for entries with breaking the gate

    John

  • So it has been several weeks since I've had the issue but it just occurred a few hours ago. Here are the relevant error.log entries:

    2020-01-30 00:28:05,945 ERROR [RequestProcessorAsyncPool-99179] (ScenarioLog.java:292) - #### Fail notification from pattern Inventory_Discovery_Trigger to gate Gate14
    2020-01-30 00:28:05,945 ERROR [RequestProcessorAsyncPool-99180] (PatternStatisticsInfoImpl.java:238) - Calculation ended without informing its out of queue, status is : 0
    2020-01-30 00:28:05,945 ERROR [RequestProcessorAsyncPool-99179] (ScenarioLog.java:292) - #### All tqls from gate Gate14 are in TFS2_FAILED state, breaking the gate
    2020-01-30 00:28:05,945 ERROR [RequestProcessorAsyncPool-99180] (ScenarioLog.java:292) - Event CALC_END was not handled for state machine Inventory_Discovery_Trigger! Current state: TSF1_TQL_LEVEL_FAILED
    2020-01-30 00:28:05,945 ERROR [RequestProcessorAsyncPool-99180] (ScenarioLog.java:310) -
    java.lang.RuntimeException: Event CALC_END was not handled for state machine Inventory_Discovery_Trigger! Current state: TSF1_TQL_LEVEL_FAILED

    So it appears that even with the hotfix there is still some condition that causes the issue to reoccur. I fix the issue by doing just as you have done. I stop and restart the Call Home Processing job.

    John