UCMDB 10.22 CUp1 Startup.log error message - FATAL [Controller] - Unexpected failure in customer 10

Hi,

After upgrade to UCMDB 10.22 CUP1 I'm getting this error message in startup.log:

2016-04-19 10:19:30,312 INFO [UCMDB - global framework scheduler] - CMDB Instance is started up for customer [id=100001, id name: State: Authorized; Customer: 1]
2016-04-19 10:19:30,328 FATAL [Controller] - Unexpected failure in customer 100001, id name: State: Authorized; Customer: 1loop()
com.mercury.topaz.cmdb.shared.manage.impl.CmdbCustomerNotExistException: [ErrorCode [-2147483648] undefined error code]
Asking for an instance [customerID=1] that doesn't exist
at com.mercury.topaz.cmdb.server.manage.instance.CustomerManager.getInstance(CustomerManager.java:108)
at com.mercury.topaz.cmdb.server.manage.instance.Controller.getOrCreateCustomerController(Controller.java:142)
at com.mercury.topaz.cmdb.server.manage.instance.Controller.customerLoop(Controller.java:117)
at com.mercury.topaz.cmdb.server.manage.instance.Controller.loopIteration(Controller.java:107)
at com.mercury.topaz.cmdb.server.manage.instance.Controller.run(Controller.java:94)
at java.lang.Thread.run(Thread.java:745)
2016-04-19 10:19:32,215 INFO [UCMDB - global framework scheduler] - Local environment start up for customer [id=1, id name: Default Client]

Can sombody explain to me what does it means and how to fixit.

I have problem with stability of my upgraded UCMDB versio.

p.s  satartup and error logs are attached.

Regards,

Damir

  • Hello Damir,

    Hope you are more than great.

     

    Could you please let us know how this issue is affecting your environment? 

     

    Thank you,

    Kind Regards.

  • Hi Ricardo.

    Thank you for your answer.

    First and biggest concern is Integration Service. It is not very stable and after some time I'm getting errors in error log that Integration Service is not active any more. Windows service also reports error while stopping it.

    This message in startup log is also confusing at startup so I went from there.

    System stability is relatively OK until I do try Population from RTSM through Population job/Integration Point over IS. When I switch population job over DFP job is correctly finished.

    Using IS when populating Ci's from RTSM for job that functioned correctly on version 10.20 at first part of upgrade process from 10.11 CUP7. With IS version 10.22 CUP1 IS probe goes crazy after starting population job wit tons of this errors:

    "<2016-04-18 16:16:54,837> [WARN ] [JobExecuterWorker-0:DS_Import from RTSM IS_TEST] (DBLog.java:53) - Failed com.hp.ucmdb.discovery.library.dal.ProcessRedundantDAO:insertDiscoveryResults select objectid from DDM_DISCOVERY_RESULTS where OBJECTID in () order by result_id for update nowait. Message: [mercury][Oracle JDBC Driver][Oracle]ORA-00936: missing expression"

    After that I have to forcibly shut down IS service and after few hours server also stops responding. Main UCMDB page goes to time out when accessing.

    I can provide you some additional logs if you want to.

    Regards,

    Damir

     

  • I forgot to mention that I also upgraded to CP18.

  • Customer ID 100001 is for the CM user.  Did you upgrade CM as well?

  • Customer ID 100001 is for the CM user.  Did you upgrade CM as well?

  • Customer ID 100001 is for the CM user.  Did you upgrade CM as well?

  • Hi,

    I upgraded all three servers to 10.22 CUP1. UCMDB server, CM server and DFP probe.

    For now a shut down IS and redirect CM Adapter integration points to use DFP probe instead IS.

    CM is complaining from time to time that IS is not active when it tries to sync with UCMDB..

    Is there a way to check that IS is upgraded properly and can I reinstall IS somehow without reinstalling whole UCMDB server?

  • I tried few more restart and now system is pretty much OK until I started IS.

    I I notices IS is very unstable and the probe started to be disconnected from server UI Probe Status.

    During IS startup I'm getting thees error messages in WrapperProbeGw.log:

    jvm 1 | <2016-04-21 09:33:33,271> 14804 [WARN ] [WrapperSimpleAppMain] (AbstractLifeCycle.java:212) - FAILED org.eclipse.jetty.server.session.SessionHandler@225dd06f: java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@4b63183d in org.eclipse.jetty.security.ConstraintSecurityHandler@30ca0264
    jvm 1 | java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@4b63183d in org.eclipse.jetty.security.ConstraintSecurityHandler@30ca0264
    jvm 1 | at org.eclipse.jetty.security.authentication.LoginAuthenticator.setConfiguration(LoginAuthenticator.java:76)

    jvm 1 | <2016-04-21 09:33:33,287> 14820 [WARN ] [WrapperSimpleAppMain] (WebAppContext.java:514) - Failed startup of context o.e.j.w.WebAppContext@3849c449{/incoming,file:/C:/HPUCMDB/UCMDBServer/integrations/deploy/incoming/,STARTING}{C:\HPUCMDB\UCMDBServer\integrations\bin\..\deploy\incoming}
    jvm 1 | java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@4b63183d in org.eclipse.jetty.security.ConstraintSecurityHandler@30ca0264
    jvm 1 | at org.eclipse.jetty.security.authentication.LoginAuthenticator.setConfiguration(LoginAuthenticator.java:76)

    jvm 1 | <2016-04-21 09:33:33,396> 14929 [WARN ] [WrapperSimpleAppMain] (ConstraintSecurityHandler.java:807) - ServletContext@o.e.j.w.WebAppContext@3718cbce{/jmx-console,file:/C:/HPUCMDB/UCMDBServer/integrations/deploy/jmx-console/,STARTING}{C:\HPUCMDB\UCMDBServer\integrations\bin\..\deploy\jmx-console} has uncovered http methods for path: /callhome
    jvm 1 | <2016-04-21 09:33:33,396> 14929 [WARN ] [WrapperSimpleAppMain] (ConstraintSecurityHandler.java:807) - ServletContext@o.e.j.w.WebAppContext@3718cbce{/jmx-console,file:/C:/HPUCMDB/UCMDBServer/integrations/deploy/jmx-console/,STARTING}{C:\HPUCMDB\UCMDBServer\integrations\bin\..\deploy\jmx-console} has uncovered http methods for path: /*
    jvm 1 | <2016-04-21 09:33:33,786> 15319 [WARN ] [WrapperSimpleAppMain] (ConstraintSecurityHandler.java:807) - ServletContext@o.e.j.w.WebAppContext@21f8b411{/original,file:/C:/HPUCMDB/UCMDBServer/integrations/deploy/original/,STARTING}{C:\HPUCMDB\UCMDBServer\integrations\bin\..\deploy\original} has uncovered http methods for path: /*
    jvm 1 | <2016-04-21 09:33:33,786> 15319 [WARN ] [WrapperSimpleAppMain] (AbstractLifeCycle.java:212) - FAILED org.eclipse.jetty.security.ConstraintSecurityHandler@75554cd: java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@37f63846 in org.eclipse.jetty.security.ConstraintSecurityHandler@75554cd
    jvm 1 | java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@37f63846 in org.eclipse.jetty.security.ConstraintSecurityHandler@75554cd
    jvm 1 | at org.eclipse.jetty.security.authentication.LoginAuthenticator.setConfiguration(LoginAuthenticator.java:76)

    jvm 1 | <2016-04-21 09:33:37,561> 19094 [WARN ] [WrapperSimpleAppMain] (DiscoveryClient.java:186) - The password of Basic Authentication doesn’t meet the policy. This password will be reset.

    Regards,

    Damir

  • We had catstophic data corruption failures after loading CP18.    Fortunately we had snapshots of all the systems and restorepoints in the DB to recover from it.   Support suggested reinstalling 10.22 and going straight to CP19 as CP18 has some known issues with data corruption like we saw.

  • Ken,

    I too have seen issues with CP18

    Most of them are with history table, you'll see errors referring to HDM tables and one instance with a class model error

    We were able to resolve both of these issues in the JMX