Lieutenant Lieutenant
Lieutenant
721 views

Service Manager integration job failure due to NullPointerException

Hello,

We have a periodic issue where Integration jobs from Service Manager are failing with the following error. It seems to happen when multiple jobs are started at the same time, each job importing thousands of data from Service manager. Anybody else faced same/similar problem?

Cheers

Sansar

uCMDB 2018.11 using ServiceManagerEnhancedAdapter9-41

 

Excerpt from probe-error.log
<2019-07-09 16:23:28,082> [ERROR] [JobExecuterWorker-2:DS_IntegrationPointName_IntegrationJobName] (AdapterService.java:226) - Failed running Datastore: IntegrationPointName
com.mercury.topaz.cmdb.shared.fcmdb.dataAccess.exception.AdapterAccessGeneralException: [ErrorCode [802] General Integration Error{IntegrationPointName}]
com.hp.ucmdb.federationspi.exception.DataAccessGeneralException: Cannot get Next Result Chunk, Failed to retrieve or parsing CI message from SM side
	at com.hp.ucmdb.dataAccess.exceptions.ExceptionConverter.createAdapterAccessGeneralException(ExceptionConverter.java:341)
	at com.hp.ucmdb.dataAccess.exceptions.ExceptionConverter.createAdapterAccessException(ExceptionConverter.java:165)
	at com.hp.ucmdb.dataAccess.exceptions.ExceptionConverter.createAdapterAccessException(ExceptionConverter.java:67)
	at com.hp.ucmdb.discovery.probe.services.dynamic.core.AdapterService.getNextTopologyResult(AdapterService.java:1713)
	at com.hp.ucmdb.discovery.probe.services.dynamic.core.AdapterService.runPopulateChangesAdapter(AdapterService.java:1359)
	at com.hp.ucmdb.discovery.probe.services.dynamic.core.AdapterService.runChangesOnPopulateChangesAdapter(AdapterService.java:1498)
	at com.hp.ucmdb.discovery.probe.services.dynamic.core.AdapterService.runQueriesOnPopulateChangesAdapter(AdapterService.java:1227)
	at com.hp.ucmdb.discovery.probe.services.dynamic.core.AdapterService.runQueries(AdapterService.java:382)
	at com.hp.ucmdb.discovery.probe.services.dynamic.core.AdapterService.runDiscovery(AdapterService.java:222)
	at com.hp.ucmdb.discovery.probe.services.dynamic.core.AdapterService.discover(AdapterService.java:166)
	at com.hp.ucmdb.discovery.probe.agents.probemgr.taskexecuter.JobExecuter.launchTask(JobExecuter.java:1270)
	at com.hp.ucmdb.discovery.probe.agents.probemgr.taskexecuter.JobExecuter$JobExecuterWorker.launchAndIsProbeRestarting(JobExecuter.java:995)
	at com.hp.ucmdb.discovery.probe.agents.probemgr.taskexecuter.JobExecuter$JobExecuterWorker.executeTask(JobExecuter.java:908)
	at com.hp.ucmdb.discovery.probe.agents.probemgr.taskexecuter.JobExecuter$JobExecuterWorker.run(JobExecuter.java:754)
Caused by: com.mercury.topaz.cmdb.shared.base.CmdbException: [ErrorCode [-2147483648] undefined error code]
com.hp.ucmdb.federationspi.exception.DataAccessGeneralException: Cannot get Next Result Chunk, Failed to retrieve or parsing CI message from SM side
	at com.hp.ucmdb.spi.data.replication.DiffDataResultImpl.next(DiffDataResultImpl.java:50)
	at com.hp.ucmdb.discovery.probe.services.dynamic.core.AdapterService.getNextTopologyResult(AdapterService.java:1710)
	... 10 more
Caused by: com.hp.ucmdb.federationspi.exception.DataAccessGeneralException: Cannot get Next Result Chunk, Failed to retrieve or parsing CI message from SM side
	at com.mercury.topaz.fcmdb.adapters.serviceDeskAdapter.population.SmGenericPopulator.getNextResultChunk(SmGenericPopulator.java:362)
	at com.hp.ucmdb.adapter.smpush.ServiceManagerGenericAdapter.populate(ServiceManagerGenericAdapter.java:342)
	at com.hp.ucmdb.adapters.GenericAdapter$PopulationAdapterDiffResultChunker.getNextResultChunk(GenericAdapter.java:1526)
	at com.hp.ucmdb.spi.data.replication.DiffDataResultImpl.next(DiffDataResultImpl.java:45)
	... 11 more
Caused by: com.hp.ucmdb.federationspi.exception.DataAccessCommunicationException: Fail to query message path:[/UcmdbCiServiceEnvironmentV001?view=expand&query=type%3D%22serviceenvironment%22+and+istatus%7E%3D%22Retired%22&start=501&count=500]
	at com.mercury.topaz.fcmdb.adapters.serviceDeskAdapter.serviceCenterConnector.ServiceManagerRestConnection.sendQueryMessage(ServiceManagerRestConnection.java:467)
	at com.mercury.topaz.fcmdb.adapters.serviceDeskAdapter.population.SmGenericPopulator.getNextResultChunk(SmGenericPopulator.java:265)
	... 14 more
Caused by: com.hp.ucmdb.federationspi.exception.DataAccessCommunicationException: Unknown failure of Message Constructor for classpath:com.mercury.topaz.fcmdb.adapters.serviceDeskAdapter.serviceCenterConnector.RestProtocolMessage
	at com.mercury.topaz.fcmdb.adapters.serviceDeskAdapter.serviceCenterConnector.ProtocolMessageFactory.createProtocolMessage(ProtocolMessageFactory.java:39)
	at com.mercury.topaz.fcmdb.adapters.serviceDeskAdapter.serviceCenterConnector.ServiceManagerRestConnection.sendQueryMessage(ServiceManagerRestConnection.java:461)
	... 15 more
Caused by: java.lang.NullPointerException

 

 

0 Likes
8 Replies
Fleet Admiral
Fleet Admiral

To exclude concurency as a reason for the error, you can set  the following attribute in sm.properties adapter file:

#concurrent(multi thread) sending: 1 = disabled, 2 or higher = enabled
number.of.concurrent.sending.threads=6

Be sure only one job is running at a time or there may be issues with dependencies.

Cheers,

Petko Popadiyski

Freelance Microfocus CMS UCMDB Consulting

Likes are appreciated!
0 Likes
Lieutenant Lieutenant
Lieutenant

@popadiyskithanks for your suggestions.

I have set the following in sm.properties and the error can still be reproduced.

number.of.concurrent.sending.threads=1

 

Our scenario demands mutiple jobs running at the same time. That is how we are able to reproduce the error. The jobs do have some dependencies on each other in a sense that one job populates CIs of a type with dependencies on other job's population CI type. However that should still be supported by the adapter in my opinion.

0 Likes
Fleet Admiral
Fleet Admiral

1. You have CIs push and relationships push. If you push relationships without the CIs already being there, the submission will fail. The way to support such push is to send first the endpoints and then the relationships. From your message I can conclude you haven't done this. 

2. You have to see what is the error when trying to retrieve "

/UcmdbCiServiceEnvironmentV001?view=expand&query=type%3D%22serviceenvironment%22+and+istatus%7E%3D%22Retired%22&start=501&count=500

 

SM is returning some error, but having multiple threads is just one of the possible reasons to get such a symptom. To answer the question what is wrong with the integration, you have to enable the debug so you can see each REST request and response. 

 

Petko

Likes are appreciated!
0 Likes
Lieutenant Lieutenant
Lieutenant

1. The error is happening during population jobs, not push jobs. Only thing we are pushing back is global id

2. Can you please point out where exactly we should turn on Debug mode?

Fleet Admiral
Fleet Admiral

1. The pushback of the ID is the only thing you have to debug. So you have to catch the request/response mechanism to the SM for pushback.

2. RTFM. https://docs.microfocus.com/UCMDB/11.0/cp-docs/docs/eng/doc_lib/Content/SM_Integrations/CP_HP_Integrations_SM941_Generic.pdf - page 210.

Likes are appreciated!
0 Likes
Cadet 1st Class Cadet 1st Class
Cadet 1st Class

Hi SansarChoinyambuu.

Have you got a chance to resolve this problem?

 

0 Likes
Lieutenant Commander Lieutenant Commander
Lieutenant Commander

Hello everyone,

I can confirm that this issue happenning also in our environment. (SM 9.62.0016 / uCMDB 10.33CUP4)

I enabled debug on both sides. SM does correcty reply to population REST service request from ucmdb.

 

In SM REST debug log I can see at least 3 threads same time responding to GET and no errors. Probe reports communication exception same time.

  6044(  3032) 10/01/2019 12:07:08 JRTE I REST	ucmdb	GET	/ucmdbDevicesPopulation
  6044(  5024) 10/01/2019 12:07:08  RTE D DAOThread JavaScript parameters: {"method":"GET","query":"(ucmdb.id~=\"\" or ucmdb.id~=NULL) and (type=\"Cluster\") and (tc.is.ci=true) and tc.status~=\"Removed\"","SimpleThreadAuth.SecureState":"0","start":"1","count":"1000","SimpleThreadAuth.AuthState":"0","resourceName":"ucmdbDevicesPopulation","uri":"[\"ucmdbDevicesPopulation\"]","view":"expand","daoType":"REST9","X-SMEssUser":"false"} 
 6044(  3032) 10/01/2019 12:07:08 JRTE I REST	ucmdb	GET	/ucmdbDevicesPopulation
  6044(  2560) 10/01/2019 12:07:08  RTE D DAOThread JavaScript parameters: {"method":"GET","query":"(ucmdb.id~=\"\" or ucmdb.id~=NULL) and (type=\"Server\" or type=\"Virtual Server\") and (tc.is.ci=true) and tc.status~=\"Removed\"","SimpleThreadAuth.SecureState":"0","start":"1","count":"1000","SimpleThreadAuth.AuthState":"0","resourceName":"ucmdbDevicesPopulation","uri":"[\"ucmdbDevicesPopulation\"]","view":"expand","daoType":"REST9","X-SMEssUser":"false"} 
6044(  5024) 10/01/2019 12:07:08  RTE D DAOThread generated response: {"ReturnCode":0,"HttpStatus":200,"Messages":[],"ReturnType":"list","ResourceName":"ucmdbDevicePopulation","@totalcount":102,"@start":1,"@count":102,"content":{"value":[{"value":{"ucmdbDevicePopulation":{"type":"object","value":{"UCMDBId":{"
6044(  3032) 10/01/2019 12:07:08 JRTE I REST9 Response: {"ReturnCode":0,"HttpStatus":200,"Messages":[],"ReturnType":"list","ResourceName":"ucmdbDevicePopulation","@totalcount":102,"@start":1,"@count":102,"content":{"value":[{"value":{"ucmdbDevicePopulation":{....
 6044(  2560) 10/01/2019 12:07:18  RTE D DAOThread generated response: {"ReturnCode":0,"HttpStatus":200,"Messages":[],"ReturnType":"list","ResourceName":"ucmdbDevicePopulation","@totalcount":5718,"@start":1,"@count":1000,"content":{"value":[{"value":{"ucmdbDevicePopulation":{"type":"object","value":{....

 

<2019-10-01 12:07:08,994> [ERROR] [JobExecuterWorker-0:DS_HPSM-LIVE-Sync_SM Cluster SM ID Population] (DataAdapterLoggerImpl.java:88) - HPSM-LIVE-Sync >> Fail to query message path:[/ucmdbDevicesPopulation?view=expand&query=%28ucmdb.id%7E%3D%22%22+or+ucmdb.id%7E%3DNULL%29+and+%28type%3D%22Cluster%22%29+and+%28tc.is.ci%3Dtrue%29+and+tc.status%7E%3D%22Removed%22&start=1&count=1000]
com.hp.ucmdb.federationspi.exception.DataAccessCommunicationException: Unknown failure of Message Constructor for classpath:com.mercury.topaz.fcmdb.adapters.serviceDeskAdapter.serviceCenterConnector.RestProtocolMessage

 

0 Likes
Lieutenant Lieutenant
Lieutenant

I want to post the work-around we have found. Hopefully it will help others having same issue.

Work-around is to create seperate integration points for each SM population jobs.

Underlying issue is when chunks of different jobs within single integration point keep coming, uCMDB has difficulties mapping those chunks to correct jobs.

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.