mercuryAS not starting on the DPS

BSM 9.25 on Windows 2008 R2
  * 1x GW server
  * 1x DPS server

On the DPS, first 3 processes show STARTED but mercuryAS hangs at a state of STARTING. DBverify completes ok.
On the GW, all process all get up to STARTED. Tried starting up DPS with the GW disabled (and with GW enabled).

Please suggest next steps for troubleshooting / fixing.

Logfile errors appearing in the following order....

odb_boot.log
2016-04-06 12:22:24,608 - - In updateAssignment: Executing prepared statement for BSM_ODB of 1 set state=STARTING(2) stateDuration=0
2016-04-06 12:22:39,086 - 2016-04-06 12:22:39,086 [Starter for "view" to Active mode. (customer 1, id name: Default Client)] ERROR - Failed to start component [view]
2016-04-06 12:22:39,086 - java.lang.IllegalArgumentException: The received string [$TempObject00000001_dl-ucmdbsrv4012______] representation is inconsistent
2016-04-06 12:22:39,086 - at com.mercury.topaz.cmdb.server.fcmdb.spi.data.query.types.ExternalIdImpl.restoreDataStore(ExternalIdImpl.java:207)

topaz_all.ejb.log
2016-04-06 12:23:22,731 [main] (ApplicationsStartUp.java:78) ERROR - failed to init applications
java.lang.NoClassDefFoundError: com/mercury/infra/appframework/FlavorInfoInt
       at java.lang.ClassLoader.defineClass1(Native Method)

jboss_server.log
2016-04-06 12:23:24,772 [main] (EjbModule.java:341) INFO  - Deploying com.mercury.topaz.tmc.bizprocess.Agent
2016-04-06 12:23:24,792 [main] (ServiceMBeanSupport.java:387) INFO  - Registration is not done -> stop
2016-04-06 12:23:24,809 [main] (ServiceController.java:337) WARN  - Problem creating service jboss.j2ee:service=EjbModule,module=tmcagent-client-server.jar
javax.management.InstanceAlreadyExistsException: jboss.j2ee:jndiName=com.mercury.topaz.tmc.bizprocess.Agent,service=EJB already registered.
       at org.jboss.mx.server.registry.BasicMBeanRegistry.add(BasicMBeanRegistry.java:761)

And current jboss_boot.log shows DPS getting stuck here....
2016-04-06 12:25:14,186 - Importing content from directory C:\HPBSM\excite\data\omi
2016-04-06 12:26:35,124 - 2016-04-06 12:26:35,122 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/hac, warUrl=.../tmp/deploy/tmp1152231860252774972hac-exp.war/
2016-04-06 12:26:35,541 - 2016-04-06 12:26:35,540 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/mam-collectors, warUrl=file:/C:/HPBSM/AppServer/webapps/mam-collectors.war/
2016-04-06 12:26:36,205 - 2016-04-06 12:26:36,202 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/myStatus, warUrl=file:/C:/HPBSM/AppServer/webapps/myStatus.war/
2016-04-06 12:26:36,993 - 2016-04-06 12:26:36,991 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/opr-config-server, warUrl=file:/C:/HPBSM/AppServer/webapps/opr-config-content-server.war/
2016-04-06 12:28:26,058 - 2016-04-06 12:28:26,057 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/opr-admin-server, warUrl=file:/C:/HPBSM/AppServer/webapps/opr-content-manager-server.war/

Whereas back at Feb 17th jboss_boot.log shows DPS starting up fully....
2016-02-17 16:23:17,869 [main] (TomcatDeployer.java:127) INFO - deploy, ctxPath=/hac, warUrl=.../tmp/deploy/tmp8114137663858963826hac-exp.war/
2016-02-17 16:23:18,524 [main] ..../mam-collectors. .../webapps/mam-collectors.war/
2016-02-17 16:23:19,929 [main] ..../myStatus. .../webapps/myStatus.war/
2016-02-17 16:23:44,625 [main] ..../opr-config-server ..../webapps/opr-config-content-server.war/
2016-02-17 16:27:56,061 [main] ..../opr-admin-server. .../webapps/opr-content-manager-server.war/
2016-02-17 16:31:52,730 [main] ..../ucmdb-api ..../webapps/ucmdb-api.war/
2016-02-17 16:31:54,780 [main] ..../jmx-console ..../deploy/jmx-console.war/
2016-02-17 16:32:02,140 [main] ..../topaz/gdeopenapi, warUrl=.../tmp/deploy/tmp8039293819259786968gde-openapi-ejb.ear-contents/gdeopenapi-exp.war/
2016-02-17 16:32:10,801 [main] ..../topaz/topaz_api, warUrl=.../tmp/deploy/tmp1120370964531886372tmc.ear-contents/tmc-exp.war/

Many thanks in advance.

Parents
  • mercuryas is a core BSM process and usually takes time to start. From how long is this service in starting state? To do BSM restart, stop GW first and then DPS and to start services, vice versa. Before we go ahead with troubleshooting, if you can restart BSM servers, go ahead with that. Is your BSM running on tomcat or IIS? If its IIS they need to be started, before starting services.

    Once BSM servers are back online share jvm_statistics.log & nanny_all.log as well.

    Hope that BSM ports are not being blocked by virus scanners or other applications. MercuryAS uses port 80.

    If possible get mercuryas thread dump through JMX http://<BAC_server>:11021. This should give us more info on what is going on

    Foundations
    Foundations:type=NannyManager
    invoke java.lang.String showServiceInfoAsHTML

    locate the line
    mercuryAS MercuryAS STARTED 6
    here click on "Thread dump"

  • Thank-you for your prompt response, Sujith.

    More details as requested:
    1. waiting over 24 hours and mercuryAS process still reports STARTING
    2. stop/starting in order: Stop GW >> Stop DPS >> Start DPS >> Start GW
    3. Tomcat (not IIS)
    4. port :80 appears free (netstat -an)
    5. nanny_all.log has been captured and reviewed -- see attached
    6. jvm_statistics.log has been captured and reviewed  -- see attached
    7. nothing appears to fail in above 2 logfiles
    8. can locate the line mercuryAS MercuryAS STARTING 1 6
    9. cannot see this option: here click on "Thread dump"

    It seems Thread dump isn't possible for mercuryAS stuck at STARTING (not STARTED).
    >> C:\HPBSM\opr\support>opr-support-utils.bat -generateThreadDump mercuryAS
    >> 'mercuryAS' service is not running!

    Please share any extra pointers on what and how to troubleshoot further.

    Thanks again in advance.

  • Thanks. What is the time differce between BSM servers and DB servers. Can you get these log files?

    topaz_all.ejb
    topaz_all
    jboss_boot
    jboss_server
    mercuryas.log

    -- Sujith

  • Also upload the logs form %topaz_home%\log\supervisor. Lets stop services and clear working cache by doing the following. Take a backup before deleting them (usually not required as they are created after restart)

    Go to %HPBAC_HOME%/EJBContainer/server/mercury/tmp/deploy, and delete its contents.
    Go to %HPBAC_HOME%/EJBContainer/server/mercury/tmp/sessions, and delete its contents.
    Delete jboss.web by going to %HPBAC_HOME%/EJBContainer/server/mercury/work

     

Reply
  • Also upload the logs form %topaz_home%\log\supervisor. Lets stop services and clear working cache by doing the following. Take a backup before deleting them (usually not required as they are created after restart)

    Go to %HPBAC_HOME%/EJBContainer/server/mercury/tmp/deploy, and delete its contents.
    Go to %HPBAC_HOME%/EJBContainer/server/mercury/tmp/sessions, and delete its contents.
    Delete jboss.web by going to %HPBAC_HOME%/EJBContainer/server/mercury/work

     

Children
No Data