Highlighted
Frequent Contributor.. Frequent Contributor..
Frequent Contributor..
979 views

"another thread has interrupted the request thread"

Error with models dropping CI's every day.

 

Mbean: UCMDB:service=Server Services. Method: viewSystemInformation

*** System information ***
Server version: 10.11.CUP2 build: 328
OS: Linux version 2.6.32-431.29.2.el6.x86_64
DB: [UD_Schema] at SQLServer on [obscured:1433] with userName [obscured] with password [obscured]
HA: Is_ha_enabled = true
        Cluster_name = SQLServer_UD_Schema
        Cluster_nodes_names = [obscured_1421090735693]
        Cluster_node_number = 1
Customers [Total of 2 customer(s)]:
        [Customer ID: 1, Name: Default Client, CP: 14.03.230Description: Default Client, Config ID: 3]
        [Customer ID: 100001, Name: State: Authorized; Customer: 1, CP: -1.0.0Description: null, Config ID: 3]
Probe: obscured, version: 10.11.CUP2.328, build: 328
Probe: IntegrationService, version: 10.11.224, build: 224
Probe: obscured, version: 10.11.CUP2.328, build: 328
Probe: obscured, version: 10.11.CUP2.328, build: 328
Probe: obscured, version: 10.11.CUP2.328, build: 328

 

 

Any thoughts on this error (from cmdb.enrichment.log):

 

2015-01-15 11:35:50,167 [RequestProcessorAsyncPool-445986] INFO  appender.calculateAdHocEnrichmentBusinessView(141) - Deltek Time & Expense -- Calculate ad hoc business view enrichment  
2015-01-15 12:35:50,484 [RequestProcessorAsyncPool-445730] ERROR appender.calculateAdHocEnrichment(204) - Deltek Costpoint App -- Failed to ad hoc calculate enrichment related to pattern: Deltek Costpoint App1421321750125  
appilog.framework.shared.manage.impl.MamResponseException: [ErrorCode [-2147483648] undefined error code]
null : class java.lang.InterruptedException :
CMDB Internal Error: InterruptedException - another thread has interrupted the request thread for request: {request: ID='604c6ff29f390efec1c254f5921dc058' Message='General CMDB request' Operation='com.hp.ucmdb.reconciliation.datain.operation.DataInUpdateDataIfExist :  [ID=1485818137] [Customer ID=1] [Changer=Enrichment: Enrichment's rule: Deltek Costpoint App]         [Datas to update-    1] [Referenced Data-    0] updateIfExist  :  [Objects For Update- business_application(1)] [Links For Update-] [Referenced Objects-] [Referenced Links-]' Customer ID='1, id name: Default Client'
Context='CMDB Context: Customer id = '1, id name: Default Client' , User ID = '911' , Caller Application = 'CMDB''}
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.createCmdbResponseOnInterrupt(RequestProcessor.java:465)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.processRequest(RequestProcessor.java:199)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.handleRequest(RequestProcessor.java:117)
    at com.mercury.topaz.cmdb.server.manage.Framework.handleRequest(Framework.java:571)
    at com.mercury.topaz.cmdb.client.manage.api.impl.AbstractCmdbApi.executeCMDBOperation(AbstractCmdbApi.java:38)
    at com.mercury.topaz.cmdb.client.manage.api.impl.AbstractCmdbApi.executeOperation(AbstractCmdbApi.java:57)
    at com.mercury.topaz.cmdb.client.manage.api.impl.InternalApi.executeOperation(InternalApi.java:32)
    at com.mercury.topaz.cmdb.client.manage.api.impl.InternalApi.executeCMDBOperation(InternalApi.java:24)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeOperation(ServerApiFacade.java:56)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeOperation(ServerApiFacade.java:34)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade$1.executeInContext(ServerApiFacade.java:65)
    at com.mercury.topaz.cmdb.shared.manage.AuthorizationContextUtils.executeInSystemAuthorizationContext(AuthorizationContextUtils.java:24)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeSystemOperation(ServerApiFacade.java:61)
    at com.mercury.topaz.cmdb.server.enrichment.util.EnrichmentBusinessViewUtil.updateBusinessObject(EnrichmentBusinessViewUtil.java:184)
    at com.mercury.topaz.cmdb.server.enrichment.util.EnrichmentBusinessViewUtil.retrieveAndUpdateBusinessObject(EnrichmentBusinessViewUtil.java:113)
    at com.mercury.topaz.cmdb.server.enrichment.util.EnrichmentBusinessViewUtil.retrieveBusinessObject(EnrichmentBusinessViewUtil.java:99)
    at com.mercury.topaz.cmdb.server.enrichment.calculator.impl.EnrichmentAdHocCalculatorManagerImpl.fillListPerOperation(EnrichmentAdHocCalculatorManagerFactory.java:278)
    at com.mercury.topaz.cmdb.server.enrichment.calculator.impl.EnrichmentAdHocCalculatorManagerImpl.calculateAdHocEnrichment(EnrichmentAdHocCalculatorManagerFactory.java:192)
    at com.mercury.topaz.cmdb.server.enrichment.calculator.impl.EnrichmentAdHocCalculatorManagerImpl.calculateAdHocEnrichmentBusinessView(EnrichmentAdHocCalculatorManagerFactory.java:156)
    at com.mercury.topaz.cmdb.shared.enrichment.calculator.operation.command.impl.EnrichmentCommandAdHocBusinessViewCalculator.enrichmentExecute(EnrichmentCommandAdHocBusinessViewCalculator.java:58)
    at com.mercury.topaz.cmdb.shared.enrichment.calculator.operation.impl.AbstractEnrichmentAdHocCalculatorOperation.doExecute(AbstractEnrichmentAdHocCalculatorOperation.java:30)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractFrameworkOperation.commonExecute(AbstractFrameworkOperation.java:17)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation$OperationExecuteFlowTrackingCommand.execute(AbstractCommonOperation.java:87)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation$OperationExecuteFlowTrackingCommand.execute(AbstractCommonOperation.java:60)
    at com.mercury.topaz.cmdb.shared.manage.flowmanagement.api.FlowManager.execute(FlowManager.java:227)
    at com.mercury.topaz.cmdb.shared.manage.operation.flow.OperationInFlowDefaultExecutor.execute(OperationInFlowDefaultExecutor.java:23)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation.execute(AbstractCommonOperation.java:159)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequest(RequestProcessor.java:211)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequestWithQueueLimitation(RequestProcessor.java:220)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.processRequest(RequestProcessor.java:184)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.handleRequest(RequestProcessor.java:117)
    at com.mercury.topaz.cmdb.server.manage.Framework.handleRequest(Framework.java:571)
    at com.mercury.topaz.cmdb.client.manage.api.impl.AbstractCmdbApi.executeCMDBOperation(AbstractCmdbApi.java:38)
    at com.mercury.topaz.cmdb.client.manage.api.impl.AbstractCmdbApi.executeOperation(AbstractCmdbApi.java:57)
    at com.mercury.topaz.cmdb.client.manage.api.impl.InternalApi.executeOperation(InternalApi.java:32)
    at com.mercury.topaz.cmdb.client.manage.api.impl.InternalApi.executeCMDBOperation(InternalApi.java:24)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeOperation(ServerApiFacade.java:56)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeOperation(ServerApiFacade.java:34)
    at com.mercury.topaz.cmdb.server.manage.subsystem.AbstractCommonSubsystemManager.executeOperation(AbstractCommonSubsystemManager.java:82)
    at com.mercury.topaz.cmdb.shared.enrichment.calculator.operation.command.impl.EnrichmentRestrictedToTenantsOperation.enrichmentExecute(EnrichmentRestrictedToTenantsOperation.java:49)
    at com.mercury.topaz.cmdb.shared.enrichment.calculator.operation.impl.AbstractEnrichmentAdHocCalculatorOperation.doExecute(AbstractEnrichmentAdHocCalculatorOperation.java:30)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractFrameworkOperation.commonExecute(AbstractFrameworkOperation.java:17)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation$OperationExecuteFlowTrackingCommand.execute(AbstractCommonOperation.java:87)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation$OperationExecuteFlowTrackingCommand.execute(AbstractCommonOperation.java:60)
    at com.mercury.topaz.cmdb.shared.manage.flowmanagement.api.FlowManager.execute(FlowManager.java:227)
    at com.mercury.topaz.cmdb.shared.manage.operation.flow.OperationInFlowDefaultExecutor.execute(OperationInFlowDefaultExecutor.java:23)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation.execute(AbstractCommonOperation.java:159)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequest(RequestProcessor.java:211)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequestWithQueueLimitation(RequestProcessor.java:220)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.processRequest(RequestProcessor.java:184)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.access$300(RequestProcessor.java:43)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor$4.run(RequestProcessor.java:531)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:996)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at com.mercury.topaz.cmdb.server.manage.semaphore.CmdbReentrantSemaphoreImpl.acquire(CmdbReentrantSemaphoreImpl.java:104)
    at com.mercury.topaz.cmdb.server.manage.semaphore.CmdbReentrantSemaphoreImpl.acquire(CmdbReentrantSemaphoreImpl.java:59)
    at com.mercury.topaz.cmdb.server.manage.rpm.LoadBalancer.acquire(LoadBalancer.java:126)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor$2.lock(RequestProcessor.java:249)
    at com.mercury.topaz.cmdb.server.manage.rpm.LocksActivator.lock(LocksActivator.java:32)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.lock(RequestProcessor.java:267)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequest(RequestProcessor.java:209)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequestWithQueueLimitation(RequestProcessor.java:228)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.processRequest(RequestProcessor.java:184)
    ... 53 more
2015-01-15 12:35:50,485 [RequestProcessorAsyncPool-445986] ERROR appender.calculateAdHocEnrichment(204) - Deltek Time & Expense -- Failed to ad hoc calculate enrichment related to pattern: Deltek Time & Expense1421321750167  
appilog.framework.shared.manage.impl.MamResponseException: [ErrorCode [-2147483648] undefined error code]
null : class java.lang.InterruptedException :
CMDB Internal Error: InterruptedException - another thread has interrupted the request thread for request: {request: ID='f13b6d6b2d223be44d81c5b48c10bc1d' Message='General CMDB request' Operation='com.hp.ucmdb.reconciliation.datain.operation.DataInUpdateDataIfExist :  [ID=1043487241] [Customer ID=1] [Changer=Enrichment: Enrichment's rule: Deltek Time & Expense]        [Datas to update-    1] [Referenced Data-    0] updateIfExist  :  [Objects For Update- business_application(1)] [Links For Update-] [Referenced Objects-] [Referenced Links-]' Customer ID='1, id name: Default Client'
Context='CMDB Context: Customer id = '1, id name: Default Client' , User ID = '911' , Caller Application = 'CMDB''}
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.createCmdbResponseOnInterrupt(RequestProcessor.java:465)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.processRequest(RequestProcessor.java:199)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.handleRequest(RequestProcessor.java:117)
    at com.mercury.topaz.cmdb.server.manage.Framework.handleRequest(Framework.java:571)
    at com.mercury.topaz.cmdb.client.manage.api.impl.AbstractCmdbApi.executeCMDBOperation(AbstractCmdbApi.java:38)
    at com.mercury.topaz.cmdb.client.manage.api.impl.AbstractCmdbApi.executeOperation(AbstractCmdbApi.java:57)
    at com.mercury.topaz.cmdb.client.manage.api.impl.InternalApi.executeOperation(InternalApi.java:32)
    at com.mercury.topaz.cmdb.client.manage.api.impl.InternalApi.executeCMDBOperation(InternalApi.java:24)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeOperation(ServerApiFacade.java:56)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeOperation(ServerApiFacade.java:34)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade$1.executeInContext(ServerApiFacade.java:65)
    at com.mercury.topaz.cmdb.shared.manage.AuthorizationContextUtils.executeInSystemAuthorizationContext(AuthorizationContextUtils.java:24)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeSystemOperation(ServerApiFacade.java:61)
    at com.mercury.topaz.cmdb.server.enrichment.util.EnrichmentBusinessViewUtil.updateBusinessObject(EnrichmentBusinessViewUtil.java:184)
    at com.mercury.topaz.cmdb.server.enrichment.util.EnrichmentBusinessViewUtil.retrieveAndUpdateBusinessObject(EnrichmentBusinessViewUtil.java:113)
    at com.mercury.topaz.cmdb.server.enrichment.util.EnrichmentBusinessViewUtil.retrieveBusinessObject(EnrichmentBusinessViewUtil.java:99)
    at com.mercury.topaz.cmdb.server.enrichment.calculator.impl.EnrichmentAdHocCalculatorManagerImpl.fillListPerOperation(EnrichmentAdHocCalculatorManagerFactory.java:278)
    at com.mercury.topaz.cmdb.server.enrichment.calculator.impl.EnrichmentAdHocCalculatorManagerImpl.calculateAdHocEnrichment(EnrichmentAdHocCalculatorManagerFactory.java:192)
    at com.mercury.topaz.cmdb.server.enrichment.calculator.impl.EnrichmentAdHocCalculatorManagerImpl.calculateAdHocEnrichmentBusinessView(EnrichmentAdHocCalculatorManagerFactory.java:156)
    at com.mercury.topaz.cmdb.shared.enrichment.calculator.operation.command.impl.EnrichmentCommandAdHocBusinessViewCalculator.enrichmentExecute(EnrichmentCommandAdHocBusinessViewCalculator.java:58)
    at com.mercury.topaz.cmdb.shared.enrichment.calculator.operation.impl.AbstractEnrichmentAdHocCalculatorOperation.doExecute(AbstractEnrichmentAdHocCalculatorOperation.java:30)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractFrameworkOperation.commonExecute(AbstractFrameworkOperation.java:17)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation$OperationExecuteFlowTrackingCommand.execute(AbstractCommonOperation.java:87)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation$OperationExecuteFlowTrackingCommand.execute(AbstractCommonOperation.java:60)
    at com.mercury.topaz.cmdb.shared.manage.flowmanagement.api.FlowManager.execute(FlowManager.java:227)
    at com.mercury.topaz.cmdb.shared.manage.operation.flow.OperationInFlowDefaultExecutor.execute(OperationInFlowDefaultExecutor.java:23)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation.execute(AbstractCommonOperation.java:159)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequest(RequestProcessor.java:211)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequestWithQueueLimitation(RequestProcessor.java:220)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.processRequest(RequestProcessor.java:184)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.handleRequest(RequestProcessor.java:117)
    at com.mercury.topaz.cmdb.server.manage.Framework.handleRequest(Framework.java:571)
    at com.mercury.topaz.cmdb.client.manage.api.impl.AbstractCmdbApi.executeCMDBOperation(AbstractCmdbApi.java:38)
    at com.mercury.topaz.cmdb.client.manage.api.impl.AbstractCmdbApi.executeOperation(AbstractCmdbApi.java:57)
    at com.mercury.topaz.cmdb.client.manage.api.impl.InternalApi.executeOperation(InternalApi.java:32)
    at com.mercury.topaz.cmdb.client.manage.api.impl.InternalApi.executeCMDBOperation(InternalApi.java:24)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeOperation(ServerApiFacade.java:56)
    at com.mercury.topaz.cmdb.server.manage.ServerApiFacade.executeOperation(ServerApiFacade.java:34)
    at com.mercury.topaz.cmdb.server.manage.subsystem.AbstractCommonSubsystemManager.executeOperation(AbstractCommonSubsystemManager.java:82)
    at com.mercury.topaz.cmdb.shared.enrichment.calculator.operation.command.impl.EnrichmentRestrictedToTenantsOperation.enrichmentExecute(EnrichmentRestrictedToTenantsOperation.java:49)
    at com.mercury.topaz.cmdb.shared.enrichment.calculator.operation.impl.AbstractEnrichmentAdHocCalculatorOperation.doExecute(AbstractEnrichmentAdHocCalculatorOperation.java:30)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractFrameworkOperation.commonExecute(AbstractFrameworkOperation.java:17)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation$OperationExecuteFlowTrackingCommand.execute(AbstractCommonOperation.java:87)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation$OperationExecuteFlowTrackingCommand.execute(AbstractCommonOperation.java:60)
    at com.mercury.topaz.cmdb.shared.manage.flowmanagement.api.FlowManager.execute(FlowManager.java:227)
    at com.mercury.topaz.cmdb.shared.manage.operation.flow.OperationInFlowDefaultExecutor.execute(OperationInFlowDefaultExecutor.java:23)
    at com.mercury.topaz.cmdb.shared.manage.operation.impl.AbstractCommonOperation.execute(AbstractCommonOperation.java:159)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequest(RequestProcessor.java:211)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequestWithQueueLimitation(RequestProcessor.java:220)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.processRequest(RequestProcessor.java:184)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.access$300(RequestProcessor.java:43)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor$4.run(RequestProcessor.java:531)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:996)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at com.mercury.topaz.cmdb.server.manage.semaphore.CmdbReentrantSemaphoreImpl.acquire(CmdbReentrantSemaphoreImpl.java:104)
    at com.mercury.topaz.cmdb.server.manage.semaphore.CmdbReentrantSemaphoreImpl.acquire(CmdbReentrantSemaphoreImpl.java:59)
    at com.mercury.topaz.cmdb.server.manage.rpm.LoadBalancer.acquire(LoadBalancer.java:126)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor$2.lock(RequestProcessor.java:249)
    at com.mercury.topaz.cmdb.server.manage.rpm.LocksActivator.lock(LocksActivator.java:32)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.lock(RequestProcessor.java:267)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequest(RequestProcessor.java:209)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.doProcessRequestWithQueueLimitation(RequestProcessor.java:228)
    at com.mercury.topaz.cmdb.server.manage.rpm.RequestProcessor.processRequest(RequestProcessor.java:184)
    ... 53 more
2015-01-15 12:35:50,485 [RequestProcessorAsyncPool-445730] INFO  appender.calculateAdHocEnrichmentBusinessView(159) - Deltek Costpoint App -- Finished ad hoc calculate business view enrichment related to pattern: Deltek Costpoint App  
2015-01-15 12:35:50,485 [RequestProcessorAsyncPool-445986] INFO  appender.calculateAdHocEnrichmentBusinessView(159) - Deltek Time & Expense -- Finished ad hoc calculate business view enrichment related to pattern: Deltek Time & Expense

 

 

0 Likes
5 Replies
Highlighted
Frequent Contributor.. Frequent Contributor..
Frequent Contributor..

Re: "another thread has interrupted the request thread"

Right now we have over 400,000 CI's in the sending queues between three probes.  uCMDB's single-threaded reconciliation engine has been churning on this stuff all day, with one of eight CPU's always maxed out.

 

There has GOT to be a way to speed this thing up.  Any suggestions?

 

From the app server command line, running /opt/hp/UCMDB/UCMDBServer/tools/dbscripts/newDbTool.sh consistency --preview reveals:

 

Tue Mar 3 22:06:04 UTC 2015
Running db tool with the following parameters: consistency --preview
2015-03-03 22:06:05,505 [main] INFO  - Running tool: consistency [preview mode = true customerId=1]
2015-03-03 22:06:07,111 [main] INFO  - DB TYPE = SQLServer
2015-03-03 22:06:07,111 [main] INFO  - DB NAME = UD_Schema
2015-03-03 22:06:07,112 [main] INFO  - DB HOST = txdal11sql03.parsons.com
2015-03-03 22:06:07,113 [main] INFO  - DB PORT = 1433
2015-03-03 22:06:07,113 [main] INFO  - DB USER = cmdbadmin
2015-03-03 22:06:10,147 [main] INFO  - Analyzing CIs that exist in the root table and don't exist in any concrete CDM table
2015-03-03 22:06:12,550 [main] INFO  - Finished - found records [ 0 ]
2015-03-03 22:06:12,551 [main] INFO  - Analyzing relations that exist in root table and don't exist in concrete CDM tables
2015-03-03 22:06:13,658 [main] INFO  - Count By Type:
2015-03-03 22:06:13,662 [main] INFO  - composition                                        1050
2015-03-03 22:06:13,662 [main] INFO  - Finished - found records [ 1050 ]
2015-03-03 22:06:13,663 [main] INFO  - Analyzing data that exist in concrete CDM tables but don't exist in ROOT table
2015-03-03 22:06:14,064 [main] INFO  - Analyzing CIs:
2015-03-03 22:06:24,676 [main] INFO  - Analyzing Relations:
2015-03-03 22:06:27,420 [main] INFO  - Finished - found totally [ 0 ] objects
2015-03-03 22:06:27,421 [main] INFO  - Analyzing data that exist in list attributes table and doesn't exist in root
2015-03-03 22:06:29,634 [main] INFO  - Finished. Found ids : [ 0 ]
2015-03-03 22:06:29,635 [main] INFO  - Analyzing broken relations whose end1/end2 doesn't exist in the root table
2015-03-03 22:06:38,505 [main] INFO  - Count By Type:
2015-03-03 22:06:38,505 [main] INFO  - composition                                        6
2015-03-03 22:06:38,506 [main] INFO  - Finished - found records [ 6 ]
2015-03-03 22:06:38,506 [main] INFO  - Analyzing broken relations whose end1/end2 doesn't exist in any of concrete CDM tables
2015-03-03 22:16:38,574 [main] ERROR - Failed to run the db tool : [task: consistency, previewMode: preview]
com.mercury.topaz.cmdb.server.manage.dal.CmdbDalException: [ErrorCode [-2147483648] undefined error code]
SQL error executing statement
                SELECT L.CMDB_ID, L.A_ROOT_CLASS, L.END1_ID, L.END2_ID
                  FROM LINK_1 L
                 WHERE NOT EXISTS (SELECT C.CMDB_ID
                                     FROM OBJECT_1 C
                                    WHERE C.CMDB_ID = L.END1_ID)
                       OR
                       NOT EXISTS (SELECT C.CMDB_ID
                                     FROM OBJECT_1 C
                                    WHERE C.CMDB_ID = L.END2_ID)


        at com.mercury.topaz.cmdb.server.manage.dal.jdbc_template.JDBCTemplate.executeQuery(JDBCTemplate.java:630)
        at com.mercury.topaz.cmdb.server.manage.dal.jdbc_template.JDBCTemplate.executeQuery(JDBCTemplate.java:569)
        at com.mercury.topaz.cmdb.server.manage.dal.jdbc_template.JDBCTemplate.executeQuery(JDBCTemplate.java:525)
        at com.mercury.topaz.cmdb.server.manage.dal.jdbc_template.JDBCTemplate$executeQuery.call(Unknown Source)
        at com.hp.ucmdb.dbtool.data.access.dao.GroovyDalDaoSupport.execSqlQuery(GroovyDalDaoSupport.groovy:50)
        at com.hp.ucmdb.dbtool.data.access.dao.GroovyDalDaoSupport.execSqlQuery(GroovyDalDaoSupport.groovy)
        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrap.invoke(PogoMetaMethodSite.java:252)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
        at com.hp.ucmdb.dbtool.data.access.dao.consistency.preview.DbToolConsistencyPreviewDaoImpl.getLinksWithInvalidEnd1OrEnd2ByCDM(DbToolConsistencyPreviewDaoImpl.groovy:221)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at com.mercury.topaz.cmdb.server.manage.dal.dao.DaoFactory$TransactionProxy$ProxyDalAbstractCommand.perform(DaoFactory.java:155)
        at com.mercury.topaz.cmdb.server.manage.dal.DalAbstractCommand$FlowControlDalCommandExecutor.execute(DalAbstractCommand.java:36)
        at com.mercury.topaz.cmdb.shared.manage.flowmanagement.api.FlowManager.execute(FlowManager.java:227)
        at com.mercury.topaz.cmdb.shared.manage.flowmanagement.api.FlowManager.execute(FlowManager.java:222)
        at com.mercury.topaz.cmdb.server.manage.dal.DalAbstractCommand.execute(DalAbstractCommand.java:54)
        at com.mercury.topaz.cmdb.server.manage.dal.CmdbDalJdbcDAO$CmdbDalDeadlockRetriable.performAction(CmdbDalJdbcDAO.java:120)
        at com.mercury.topaz.cmdb.server.manage.dal.CmdbDalJdbcDAO$CmdbDalDeadlockRetriable.performAction(CmdbDalJdbcDAO.java:105)
        at com.mercury.topaz.cmdb.shared.util.retry.impl.DefaultRetriableExecutorImpl.doExecute(DefaultRetriableExecutorImpl.java:37)
        at com.mercury.topaz.cmdb.shared.util.retry.impl.AbstractRetriableExecutor.execute(AbstractRetriableExecutor.java:37)
        at com.mercury.topaz.cmdb.server.manage.dal.CmdbDalJdbcDAO.executeQuery(CmdbDalJdbcDAO.java:61)
        at com.mercury.topaz.cmdb.server.manage.dal.CmdbDalJdbcDAO.executeQuery(CmdbDalJdbcDAO.java:53)
        at com.mercury.topaz.cmdb.server.manage.dal.dao.DaoFactory$TransactionProxy.invoke(DaoFactory.java:135)
        at com.sun.proxy.$Proxy27.getLinksWithInvalidEnd1OrEnd2ByCDM(Unknown Source)
        at com.hp.ucmdb.dbtool.data.access.dao.consistency.preview.DbToolConsistencyPreviewDao$getLinksWithInvalidEnd1OrEnd2ByCDM.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
        at com.hp.ucmdb.dbtool.processors.consistency.PreviewModeProcessorHelper.examineBrokenLinksByCDMTable(PreviewModeProcessorHelper.groovy:113)
        at com.hp.ucmdb.dbtool.processors.consistency.PreviewModeProcessorHelper$examineBrokenLinksByCDMTable.callCurrent(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
        at com.hp.ucmdb.dbtool.processors.consistency.PreviewModeProcessorHelper.executeConsistencyChecks(PreviewModeProcessorHelper.groovy:33)
        at com.hp.ucmdb.dbtool.processors.consistency.PreviewModeProcessorHelper$executeConsistencyChecks.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at com.hp.ucmdb.dbtool.processors.consistency.ConsistencyDbToolProcessor.runPreviewMode(ConsistencyDbToolProcessor.groovy:30)
        at com.hp.ucmdb.dbtool.processors.GenericDbToolProcessor.execute(GenericDbToolProcessor.groovy:34)
        at com.hp.ucmdb.dbtool.processors.Processor$execute.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:75)
        at com.hp.ucmdb.dbtool.main.DbToolRunner.runDbTool(DbToolRunner.groovy:22)
        at com.hp.ucmdb.dbtool.main.DbTool.main(DbTool.java:12)
Caused by: com.mercury.topaz.cmdb.server.manage.dal.CmdbDalException: [ErrorCode [-2147483648] undefined error code]
Can't execute prepared statement [
                SELECT L.CMDB_ID, L.A_ROOT_CLASS, L.END1_ID, L.END2_ID
                  FROM LINK_1 L
                 WHERE NOT EXISTS (SELECT C.CMDB_ID
                                     FROM OBJECT_1 C
                                    WHERE C.CMDB_ID = L.END1_ID)
                       OR
                       NOT EXISTS (SELECT C.CMDB_ID
                                     FROM OBJECT_1 C
                                    WHERE C.CMDB_ID = L.END2_ID)

             ], due to exception: java.sql.SQLTimeoutException: [mercury][SQLServer JDBC Driver]Execution timeout expired.
        at com.mercury.topaz.cmdb.server.manage.dal.CmdbDalPreparedStatementImpl.executeQuery(CmdbDalPreparedStatementImpl.java:365)
        at com.mercury.topaz.cmdb.server.manage.dal.jdbc_template.JDBCTemplate.executeQuery(JDBCTemplate.java:624)
        ... 54 more
Caused by: java.sql.SQLTimeoutException: [mercury][SQLServer JDBC Driver]Execution timeout expired.
        at com.mercury.jdbc.sqlserverbase.ddcw.b(Unknown Source)
        at com.mercury.jdbc.sqlserverbase.ddcw.a(Unknown Source)
        at com.mercury.jdbc.sqlserverbase.ddcv.b(Unknown Source)
        at com.mercury.jdbc.sqlserverbase.ddcv.a(Unknown Source)
        at com.mercury.jdbc.sqlserverbase.dde7.a(Unknown Source)
        at com.mercury.jdbc.sqlserverbase.dde7.u(Unknown Source)
        at com.mercury.jdbc.sqlserverbase.ddd2.executeQuery(Unknown Source)
        at com.mercury.jdbc.sqlserverbase.ddd6.executeQuery(Unknown Source)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
        at com.mercury.topaz.cmdb.server.manage.dal.CmdbDalPreparedStatementImpl$FlowControlQueryExecuter.execute(CmdbDalPreparedStatementImpl.java:604)
        at com.mercury.topaz.cmdb.server.manage.dal.CmdbDalPreparedStatementImpl$FlowControlQueryExecuter.execute(CmdbDalPreparedStatementImpl.java:596)
        at com.mercury.topaz.cmdb.shared.manage.flowmanagement.api.FlowManager.execute(FlowManager.java:227)
        at com.mercury.topaz.cmdb.server.manage.dal.CmdbDalPreparedStatementImpl.executeQuery(CmdbDalPreparedStatementImpl.java:356)
        ... 55 more
Tue Mar 3 22:16:38 UTC 2015

0 Likes
Highlighted
Frequent Contributor.. Frequent Contributor..
Frequent Contributor..

Re: "another thread has interrupted the request thread"

I have fixed the Integration Service issues, but the overall problems of Pattern-Based Business Application models losing CI's daily, and the high sending queues persist.  I'm using Red Hat 6.6 and SQL Server 2008.  Should I move to Windows 2012R2 for app server and SQL 2012 for database?  Would it matter?  This is too slow.

0 Likes
Highlighted
New Member.

Re: "another thread has interrupted the request thread"

I would suggest you take a look at UD (discovery).

I have seen this where the discovery jobs were not completing and the DFM was not posting in sequence.

0 Likes
Highlighted
Frequent Contributor.. Frequent Contributor..
Frequent Contributor..

Re: "another thread has interrupted the request thread"

Thanks for replying, but too vague.  Please be more specific.  The discovery jobs sometimes have trouble completing, but from what I can see they have completed, that's why there are CI's in the probe sending queues.  I need more specific direction.

0 Likes
Highlighted
New Member.

Re: "another thread has interrupted the request thread"

DFMs have a strange (to me at least) method of processing the queues.

You are not guaranteed they will be done in sequence.

If a Job runs again before the previous results are posted it can present this issue.

The Job status may show complete but all of the data may not be posted to UCMDB.

 

I would suggest you spread your jobs out so that they are complete and the data posted.

Especially those that have queued data past the Job completion times.

Not sure what UD Jobs you have or their scheduling. Changing the schedule or adding more DFMs could relieve this issue.

 

 

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.