DASHBOARD HAC service wont start on DPS after upgrade from 9.22 to 9.23

[Ben.Shirley]

Our DPS won't enter READY state after the 9.23 upgrade due to the dashboard service not starting.  We have the following in the marble_supervisor log:-

2014-01-14 11:31:08,973 [HAC-LAUNCHER] (DBAdapter.java:533) INFO  - In updateAssignment: Executing prepared statement for DASHBOARD of 1 set state=STOPPED(1) stateDuration=0
2014-01-14 11:31:29,007 [HAC-LAUNCHER] (CheckTasksCommand.java:237) INFO  - Service DASHBOARD for 1 has no collisions.
2014-01-14 11:31:29,008 [HAC-LAUNCHER] (CheckTasksCommand.java:275) INFO  - In updateStatus for 'DASHBOARD for 1' will update since DBState STOPPED vs. STARTING or dbRunID 11 vs. 11
2014-01-14 11:31:29,009 [CTRL-SVCStarter] (ServiceHandlerImpl.java:152) WARN  - Failed to start service DASHBOARD for 1
com.hp.bsm.platform.marble.supervisor.S_SupervisorException: Not ready yet to registerCustomer(1). Missing services: [Loader-0@1]
 at com.hp.bsm.platform.marble.supervisor.S_SupervisorModuleContainer.registerCustomer(S_SupervisorModuleContainer.java:291)
 at com.hp.bsm.platform.marble.supervisor.S_MarbleServiceProvider.startService(S_MarbleServiceProvider.java:40)
 at com.mercury.am.platform.controller.launcher.ServiceStarter.run(ServiceStarter.java:34)
 at java.lang.Thread.run(Thread.java:744)

Has anyone seen this issue?

[Ben.Shirley]

Actually that message was from the hac-launcher.log. The MarbleSupervisor contains:-

2014-01-14 14:44:29,260 [CTRL-SVCStarter] (S_SupervisorModuleContainer.java:289) WARN  - Rejecting registerCustomer(1) since there are missing modules: [Loader-0@1]

[Asaf Shechter]

you should look at your marble_loader logs and see why the loader cannot start.
have you tried to restart DPS?

[Ben.Shirley]

loader.all.log:-

2014-01-15 08:41:03,756 [Marble-ModuleContainerInit] (MetaDataImpl.java:61) INFO  - Got MetaData request: Login: 'com.mercury.am.platform.dataentry.loader.utils.MetaDataProxy'; Query: '/dc/flat'; Type: GET
2014-01-15 08:41:03,763 [Marble-ModuleContainerInit] (AbstractModuleContainer.java:272) ERROR - Failed to initialize Adapter LoadingAdapter of Loader will retry in 5 seconds
com.hp.bsm.platform.marble.common.MarbleException: Failed to init Loader core
 at com.hp.bsm.platform.marble.loader.loading.L_LoadingAdapterImpl.init(L_LoadingAdapterImpl.java:62)
 at com.hp.bsm.platform.marble.common.modules.AbstractModuleAdapter.init(AbstractModuleAdapter.java:22)
 at com.hp.bsm.platform.marble.common.modules.AbstractModuleContainer.initAdapters(AbstractModuleContainer.java:266)
 at com.hp.bsm.platform.marble.common.modules.AbstractModuleContainer.access$000(AbstractModuleContainer.java:41)
 at com.hp.bsm.platform.marble.common.modules.AbstractModuleContainer$1.run(AbstractModuleContainer.java:153)
Caused by: com.mercury.am.platform.dataentry.loader.core.LoaderException: Failed to load metadata, metadata node list length = 0
 at com.mercury.am.platform.dataentry.loader.utils.MetaDataProxy.getDCHelper(MetaDataProxy.java:105)
 at com.mercury.am.platform.dataentry.loader.utils.MetaDataProxy.loadMetaData(MetaDataProxy.java:134)
 at com.mercury.am.platform.dataentry.loader.utils.MetaDataProxy.loadFlatMetaData(MetaDataProxy.java:126)
 at com.mercury.am.platform.dataentry.loader.utils.MetaDataProxy.initComponent(MetaDataProxy.java:270)
 at com.mercury.am.platform.dataentry.loader.core.LoaderCore.init(LoaderCore.java:264)
 at com.hp.bsm.platform.marble.loader.loading.L_LoadingAdapterImpl.init(L_LoadingAdapterImpl.java:51)
 ... 4 more
2014-01-15 08:41:04,461 [Marble-ManagerRegistration] (ModuleLocatorImpl.java:196) INFO  - Attempting to register within ModuleManager

MarbleLoader.log:-

2014-01-15 08:43:04,116 [Marble-ModuleContainerInit] (AbstractModuleContainer.java:272) ERROR - Failed to initialize Adapter LoadingAdapter of Loader will retry in 5 seconds
com.hp.bsm.platform.marble.common.MarbleException: Failed to init Loader core
 at com.hp.bsm.platform.marble.loader.loading.L_LoadingAdapterImpl.init(L_LoadingAdapterImpl.java:62)
 at com.hp.bsm.platform.marble.common.modules.AbstractModuleAdapter.init(AbstractModuleAdapter.java:22)
 at com.hp.bsm.platform.marble.common.modules.AbstractModuleContainer.initAdapters(AbstractModuleContainer.java:266)
 at com.hp.bsm.platform.marble.common.modules.AbstractModuleContainer.access$000(AbstractModuleContainer.java:41)
 at com.hp.bsm.platform.marble.common.modules.AbstractModuleContainer$1.run(AbstractModuleContainer.java:153)
Caused by: com.mercury.am.platform.dataentry.loader.core.LoaderException: Failed to load metadata, metadata node list length = 0
 at com.mercury.am.platform.dataentry.loader.utils.MetaDataProxy.getDCHelper(MetaDataProxy.java:105)
 at com.mercury.am.platform.dataentry.loader.utils.MetaDataProxy.loadMetaData(MetaDataProxy.java:134)
 at com.mercury.am.platform.dataentry.loader.utils.MetaDataProxy.loadFlatMetaData(MetaDataProxy.java:126)
 at com.mercury.am.platform.dataentry.loader.utils.MetaDataProxy.initComponent(MetaDataProxy.java:270)
 at com.mercury.am.platform.dataentry.loader.core.LoaderCore.init(LoaderCore.java:264)
 at com.hp.bsm.platform.marble.loader.loading.L_LoadingAdapterImpl.init(L_LoadingAdapterImpl.java:51)
 ... 4 more

The DPS has been restarted but that has not helped.

[PatWest]

hi Ben,

Please check if there are problems with connectivity to the database. Is there perhaps some latency?

Is the database (server) on the same network segment as the BSM servers?

PatW

[Ben.Shirley]

Hi,

Thanks for the response. The database is available from the DPS by ping and telnetting to the SQL port 1433.

[PatWest]

If there is no latency, and the DB server is in the same network segment, then stop BSM and try a reconnect to database once again.

If the same errors appear in the MARBLE logs, then it is time to open a support case.

[drenze]

For what it's worth, I just did a clean install of 9.23 in my test environment and am having the same issue. Didn't have it prior to wiping the old install and doing the new install. Not having it in my prod environment, but definitely having it in test.

[Remzi Cirit · Accepted solution]

ERROR

2014-01-15 08:43:04,116 [Marble-ModuleContainerInit] (AbstractModuleContainer.java:272) ERROR - Failed to initialize Adapter LoadingAdapter of Loader will retry in 5 seconds
com.hp.bsm.platform.marble.common.MarbleException: Failed to init Loader core

And caused by

Caused by: com.mercury.am.platform.dataentry.loader.core.LoaderException: Failed to load metadata, metadata node list length = 0

It seems  Meta Data failure during upgrade is causing  marble  not to start.

Once the MercuryAS process starts you should be able to access the JMX console on the gateway

http://<gateway>:8080/jmx-console/HtmlAdaptor?action=inspectMBean&name=Topaz%3Aservice%3DMeta-Data+Manager

Invoke the updateMetaData() method and then restart BSM.

To see  if this takesccare of it.

Regards

Remzi

[drenze]

This did not resolve the issue. Sigh.

[Remzi Cirit]

Can you post the error again please if it has changed if it has not we can clear meta data tables and use Jmx method again

[Ben.Shirley]

Hi,

Thank you, the solution given to reload the metadata using the updateMetaData()method worked for me.  All services are now up on the DPS.

However starting the GW and getting error:-

2014-02-26 13:50:54,957 [Reconnect-Bus Monitor] (MonitorBusJMX.java:81) ERROR - Failed to initialize bus monitor

It's a long time since I posted this, now I am unsure if I did actually patch the GW or I just got as far as the DPS. Is there a quick way to check I put the 9.23 patch on the GW. If I did, I am unsure why it won't start.

Thanks