Highlighted
Honored Contributor.
Honored Contributor.
382 views

UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in customer 10

Hi,

After upgrade to UCMDB 10.22 CUP1 I'm getting this error message in startup.log:

2016-04-19 10:19:30,312 INFO [UCMDB - global framework scheduler] - CMDB Instance is started up for customer [id=100001, id name: State: Authorized; Customer: 1]
2016-04-19 10:19:30,328 FATAL [Controller] - Unexpected failure in customer 100001, id name: State: Authorized; Customer: 1loop()
com.mercury.topaz.cmdb.shared.manage.impl.CmdbCustomerNotExistException: [ErrorCode [-2147483648] undefined error code]
Asking for an instance [customerID=1] that doesn't exist
at com.mercury.topaz.cmdb.server.manage.instance.CustomerManager.getInstance(CustomerManager.java:108)
at com.mercury.topaz.cmdb.server.manage.instance.Controller.getOrCreateCustomerController(Controller.java:142)
at com.mercury.topaz.cmdb.server.manage.instance.Controller.customerLoop(Controller.java:117)
at com.mercury.topaz.cmdb.server.manage.instance.Controller.loopIteration(Controller.java:107)
at com.mercury.topaz.cmdb.server.manage.instance.Controller.run(Controller.java:94)
at java.lang.Thread.run(Thread.java:745)
2016-04-19 10:19:32,215 INFO [UCMDB - global framework scheduler] - Local environment start up for customer [id=1, id name: Default Client]

Can sombody explain to me what does it means and how to fixit.

I have problem with stability of my upgraded UCMDB versio.

p.s  satartup and error logs are attached.

Regards,

Damir

0 Likes
8 Replies
Highlighted
Trusted Contributor.
Trusted Contributor.

Re: UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in custome

Hello Damir,

Hope you are more than great.

 

Could you please let us know how this issue is affecting your environment? 

 

Thank you,

Kind Regards.

0 Likes
Highlighted
Honored Contributor.
Honored Contributor.

Re: UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in custome

Hi Ricardo.

Thank you for your answer.

First and biggest concern is Integration Service. It is not very stable and after some time I'm getting errors in error log that Integration Service is not active any more. Windows service also reports error while stopping it.

This message in startup log is also confusing at startup so I went from there.

System stability is relatively OK until I do try Population from RTSM through Population job/Integration Point over IS. When I switch population job over DFP job is correctly finished.

Using IS when populating Ci's from RTSM for job that functioned correctly on version 10.20 at first part of upgrade process from 10.11 CUP7. With IS version 10.22 CUP1 IS probe goes crazy after starting population job wit tons of this errors:

"<2016-04-18 16:16:54,837> [WARN ] [JobExecuterWorker-0:DS_Import from RTSM IS_TEST] (DBLog.java:53) - Failed com.hp.ucmdb.discovery.library.dal.ProcessRedundantDAO:insertDiscoveryResults select objectid from DDM_DISCOVERY_RESULTS where OBJECTID in () order by result_id for update nowait. Message: [mercury][Oracle JDBC Driver][Oracle]ORA-00936: missing expression"

After that I have to forcibly shut down IS service and after few hours server also stops responding. Main UCMDB page goes to time out when accessing.

I can provide you some additional logs if you want to.

Regards,

Damir

 

0 Likes
Highlighted
Honored Contributor.
Honored Contributor.

Re: UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in custome

I forgot to mention that I also upgraded to CP18.

0 Likes
Highlighted
Absent Member.. Absent Member..
Absent Member..

Re: UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in custome

Customer ID 100001 is for the CM user.  Did you upgrade CM as well?

0 Likes
Highlighted
Honored Contributor.
Honored Contributor.

Re: UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in custome

Hi,

I upgraded all three servers to 10.22 CUP1. UCMDB server, CM server and DFP probe.

For now a shut down IS and redirect CM Adapter integration points to use DFP probe instead IS.

CM is complaining from time to time that IS is not active when it tries to sync with UCMDB..

Is there a way to check that IS is upgraded properly and can I reinstall IS somehow without reinstalling whole UCMDB server?

0 Likes
Highlighted
Honored Contributor.
Honored Contributor.

Re: UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in custome

I tried few more restart and now system is pretty much OK until I started IS.

I I notices IS is very unstable and the probe started to be disconnected from server UI Probe Status.

During IS startup I'm getting thees error messages in WrapperProbeGw.log:

jvm 1 | <2016-04-21 09:33:33,271> 14804 [WARN ] [WrapperSimpleAppMain] (AbstractLifeCycle.java:212) - FAILED org.eclipse.jetty.server.session.SessionHandler@225dd06f: java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@4b63183d in org.eclipse.jetty.security.ConstraintSecurityHandler@30ca0264
jvm 1 | java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@4b63183d in org.eclipse.jetty.security.ConstraintSecurityHandler@30ca0264
jvm 1 | at org.eclipse.jetty.security.authentication.LoginAuthenticator.setConfiguration(LoginAuthenticator.java:76)

jvm 1 | <2016-04-21 09:33:33,287> 14820 [WARN ] [WrapperSimpleAppMain] (WebAppContext.java:514) - Failed startup of context o.e.j.w.WebAppContext@3849c449{/incoming,file:/C:/HPUCMDB/UCMDBServer/integrations/deploy/incoming/,STARTING}{C:\HPUCMDB\UCMDBServer\integrations\bin\..\deploy\incoming}
jvm 1 | java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@4b63183d in org.eclipse.jetty.security.ConstraintSecurityHandler@30ca0264
jvm 1 | at org.eclipse.jetty.security.authentication.LoginAuthenticator.setConfiguration(LoginAuthenticator.java:76)

jvm 1 | <2016-04-21 09:33:33,396> 14929 [WARN ] [WrapperSimpleAppMain] (ConstraintSecurityHandler.java:807) - ServletContext@o.e.j.w.WebAppContext@3718cbce{/jmx-console,file:/C:/HPUCMDB/UCMDBServer/integrations/deploy/jmx-console/,STARTING}{C:\HPUCMDB\UCMDBServer\integrations\bin\..\deploy\jmx-console} has uncovered http methods for path: /callhome
jvm 1 | <2016-04-21 09:33:33,396> 14929 [WARN ] [WrapperSimpleAppMain] (ConstraintSecurityHandler.java:807) - ServletContext@o.e.j.w.WebAppContext@3718cbce{/jmx-console,file:/C:/HPUCMDB/UCMDBServer/integrations/deploy/jmx-console/,STARTING}{C:\HPUCMDB\UCMDBServer\integrations\bin\..\deploy\jmx-console} has uncovered http methods for path: /*
jvm 1 | <2016-04-21 09:33:33,786> 15319 [WARN ] [WrapperSimpleAppMain] (ConstraintSecurityHandler.java:807) - ServletContext@o.e.j.w.WebAppContext@21f8b411{/original,file:/C:/HPUCMDB/UCMDBServer/integrations/deploy/original/,STARTING}{C:\HPUCMDB\UCMDBServer\integrations\bin\..\deploy\original} has uncovered http methods for path: /*
jvm 1 | <2016-04-21 09:33:33,786> 15319 [WARN ] [WrapperSimpleAppMain] (AbstractLifeCycle.java:212) - FAILED org.eclipse.jetty.security.ConstraintSecurityHandler@75554cd: java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@37f63846 in org.eclipse.jetty.security.ConstraintSecurityHandler@75554cd
jvm 1 | java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@37f63846 in org.eclipse.jetty.security.ConstraintSecurityHandler@75554cd
jvm 1 | at org.eclipse.jetty.security.authentication.LoginAuthenticator.setConfiguration(LoginAuthenticator.java:76)

jvm 1 | <2016-04-21 09:33:37,561> 19094 [WARN ] [WrapperSimpleAppMain] (DiscoveryClient.java:186) - The password of Basic Authentication doesn’t meet the policy. This password will be reset.

Regards,

Damir

0 Likes
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Re: UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in custome

We had catstophic data corruption failures after loading CP18.    Fortunately we had snapshots of all the systems and restorepoints in the DB to recover from it.   Support suggested reinstalling 10.22 and going straight to CP19 as CP18 has some known issues with data corruption like we saw.

Highlighted
Honored Contributor.. Honored Contributor..
Honored Contributor..

Re: UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in custome

Ken,

I too have seen issues with CP18

Most of them are with history table, you'll see errors referring to HDM tables and one instance with a class model error

We were able to resolve both of these issues in the JMX

Rey Lejano

effectualsystems.com
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.