ErrorCode [63014] Reconciliation queue is full slow reconciliation.datain.operation.DataInGracefulAd

[MarcinM]

Hello guys,

we're struggling with this issue for some time now. We're using 2020.11 Containerized version. We are populating one of our Customers with REST API. The problem we're facing is that when we are trying to populate 120 CIs (in this example) we got many errors related to full reconciliation queue. The CIs are populated 4 at a time with curl related script. We didn't had that problems with our old infrastructure based on 10.33 on premises server and SOAP calls. 

And yeah, we also tried SOAP but the problem remained.

In slow.log I can see:

Busy [Reconciliation Data In] (1) :
	LOW com.hp.ucmdb.reconciliation.datain.operation.DataInGracefulAddOrUpdate@7deccb42

What else can I check/do?

Regards,

Marcin

[GabiMedan1]

Hello,

Usually the error message is the following one :

[ErrorCode [63014] Reconciliation queue is full and cannot handle current request]
Manager "Reconciliation Data In" has [30] operations waiting on his queue.

It is a protection limit added since 9.x version to protect reconciliation engine from being spammed. 

You have to check and fix:

• for slowness(mainly DB)
• reduce script load/frequency

Increasing the allowed number of data in threads is not recommended and is more complicated in container deployments.

Hope it helps.

Best Regards,

[MarcinM]

Hi,

thanks for the response. I will contact my company's DB team for the first suggestion. About reduce script load/frequency - could you guide me?

Regards,

Marcin Musiol

[GabiMedan1]

Hello,

You have to discuss with the owner of the "curl related script" that sends the data to UCMDB. Understand its logic and if it sends too often add a delay or something else to avoid spamming ucmdb.

Hope it helps.

Best Regards,

[MarcinM]

Thanks for your response. I believe that's not the case because recently in our current solution we're using the same scripts. We're populating ucmdb's 10.33 db via WebService API but in our new environment (2020.11 Containerized) we had tested both REST API and WebService API and the problem occurred in both solutions.

Regards,

Marcin

[MichaelBaumanns1]

Hello MArcinM,

we had a simlar issue (but we imported more than 100.000 records via Java API).

One of our main outcommings was a bad performance with oracle. In the JMX Dal section we have seen many results from "showMissingIndexes" operation. But even a recreate with the different rebuild operatons did not help.

The only solution we found was changing the DB technology. With postgres it worked immediatly.

When you discovery CIs, is the processing of the results fast? Or do you see an increasing sending queue in the dataflow probe status?

Best regards,

Michael

[MarcinM]

Hi Michael,

the sad thing is that we're already using PostgreSQL, the other thing is our CI amount is about 120-150 CI pushed to UCMDB by REST API call (one CI by one). We will check the db though. Thanks a lot!