Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE
Highlighted
Contributor.. Andrewartha Contributor..
Contributor..
3018 views

mercuryAS not starting on the DPS

Jump to solution

BSM 9.25 on Windows 2008 R2
  * 1x GW server
  * 1x DPS server

On the DPS, first 3 processes show STARTED but mercuryAS hangs at a state of STARTING. DBverify completes ok.
On the GW, all process all get up to STARTED. Tried starting up DPS with the GW disabled (and with GW enabled).

Please suggest next steps for troubleshooting / fixing.

Logfile errors appearing in the following order....

odb_boot.log
2016-04-06 12:22:24,608 - - In updateAssignment: Executing prepared statement for BSM_ODB of 1 set state=STARTING(2) stateDuration=0
2016-04-06 12:22:39,086 - 2016-04-06 12:22:39,086 [Starter for "view" to Active mode. (customer 1, id name: Default Client)] ERROR - Failed to start component [view]
2016-04-06 12:22:39,086 - java.lang.IllegalArgumentException: The received string [$TempObject00000001_dl-ucmdbsrv4012______] representation is inconsistent
2016-04-06 12:22:39,086 - at com.mercury.topaz.cmdb.server.fcmdb.spi.data.query.types.ExternalIdImpl.restoreDataStore(ExternalIdImpl.java:207)

topaz_all.ejb.log
2016-04-06 12:23:22,731 [main] (ApplicationsStartUp.java:78) ERROR - failed to init applications
java.lang.NoClassDefFoundError: com/mercury/infra/appframework/FlavorInfoInt
       at java.lang.ClassLoader.defineClass1(Native Method)

jboss_server.log
2016-04-06 12:23:24,772 [main] (EjbModule.java:341) INFO  - Deploying com.mercury.topaz.tmc.bizprocess.Agent
2016-04-06 12:23:24,792 [main] (ServiceMBeanSupport.java:387) INFO  - Registration is not done -> stop
2016-04-06 12:23:24,809 [main] (ServiceController.java:337) WARN  - Problem creating service jboss.j2ee:service=EjbModule,module=tmcagent-client-server.jar
javax.management.InstanceAlreadyExistsException: jboss.j2ee:jndiName=com.mercury.topaz.tmc.bizprocess.Agent,service=EJB already registered.
       at org.jboss.mx.server.registry.BasicMBeanRegistry.add(BasicMBeanRegistry.java:761)

And current jboss_boot.log shows DPS getting stuck here....
2016-04-06 12:25:14,186 - Importing content from directory C:\HPBSM\excite\data\omi
2016-04-06 12:26:35,124 - 2016-04-06 12:26:35,122 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/hac, warUrl=.../tmp/deploy/tmp1152231860252774972hac-exp.war/
2016-04-06 12:26:35,541 - 2016-04-06 12:26:35,540 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/mam-collectors, warUrl=file:/C:/HPBSM/AppServer/webapps/mam-collectors.war/
2016-04-06 12:26:36,205 - 2016-04-06 12:26:36,202 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/myStatus, warUrl=file:/C:/HPBSM/AppServer/webapps/myStatus.war/
2016-04-06 12:26:36,993 - 2016-04-06 12:26:36,991 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/opr-config-server, warUrl=file:/C:/HPBSM/AppServer/webapps/opr-config-content-server.war/
2016-04-06 12:28:26,058 - 2016-04-06 12:28:26,057 [main] (TomcatDeployer.java:127) INFO  - deploy, ctxPath=/opr-admin-server, warUrl=file:/C:/HPBSM/AppServer/webapps/opr-content-manager-server.war/

Whereas back at Feb 17th jboss_boot.log shows DPS starting up fully....
2016-02-17 16:23:17,869 [main] (TomcatDeployer.java:127) INFO - deploy, ctxPath=/hac, warUrl=.../tmp/deploy/tmp8114137663858963826hac-exp.war/
2016-02-17 16:23:18,524 [main] ..../mam-collectors. .../webapps/mam-collectors.war/
2016-02-17 16:23:19,929 [main] ..../myStatus. .../webapps/myStatus.war/
2016-02-17 16:23:44,625 [main] ..../opr-config-server ..../webapps/opr-config-content-server.war/
2016-02-17 16:27:56,061 [main] ..../opr-admin-server. .../webapps/opr-content-manager-server.war/
2016-02-17 16:31:52,730 [main] ..../ucmdb-api ..../webapps/ucmdb-api.war/
2016-02-17 16:31:54,780 [main] ..../jmx-console ..../deploy/jmx-console.war/
2016-02-17 16:32:02,140 [main] ..../topaz/gdeopenapi, warUrl=.../tmp/deploy/tmp8039293819259786968gde-openapi-ejb.ear-contents/gdeopenapi-exp.war/
2016-02-17 16:32:10,801 [main] ..../topaz/topaz_api, warUrl=.../tmp/deploy/tmp1120370964531886372tmc.ear-contents/tmc-exp.war/

Many thanks in advance.

Tags (4)
0 Likes
1 Solution

Accepted Solutions
Contributor.. Andrewartha Contributor..
Contributor..

Re: mercuryAS not starting on the DPS

Jump to solution

Thanks for your suggestions, Sujith.

Solved. DPS is now showing as Started.

Symptom was in error log
odb_boot.log [Starter for "view" to Active mode. (customer 1, id name: Default Client)] (CustomerController.java:413) ERROR - Failed to start component [view]
java.lang.IllegalArgumentException: The received string [$TempObject00000001_xxxucmdbsrv4012______] representation is inconsistent
at com.mercury.topaz.cmdb.server.fcmdb.spi.data.query.types.ExternalIdImpl.restoreDataStore(ExternalIdImpl.java:207)

Solution was
Removal of Cmdb Object  via JMX :21212
(My problematic ObjectID was visible in cmdb.query.detailed.log after turning up loglevel in cmdb.properties to DEBUG)

For the record
* DPS/GW clocks showed the same time
* clearing JBoss cache already tried, as per this post
* I captured the logs suggested by Sujith but haven't uploaded them to my post

Also tried
* Sonic MQ uninstall/install.bat
* run_db_verify.bat and applied output *.sql fixes to the Database

Thanks again

6 Replies
Sujith Kumar C. Acclaimed Contributor.
Acclaimed Contributor.

Re: mercuryAS not starting on the DPS

Jump to solution

mercuryas is a core BSM process and usually takes time to start. From how long is this service in starting state? To do BSM restart, stop GW first and then DPS and to start services, vice versa. Before we go ahead with troubleshooting, if you can restart BSM servers, go ahead with that. Is your BSM running on tomcat or IIS? If its IIS they need to be started, before starting services.

Once BSM servers are back online share jvm_statistics.log & nanny_all.log as well.

Hope that BSM ports are not being blocked by virus scanners or other applications. MercuryAS uses port 80.

If possible get mercuryas thread dump through JMX http://<BAC_server>:11021. This should give us more info on what is going on

Foundations
Foundations:type=NannyManager
invoke java.lang.String showServiceInfoAsHTML

locate the line
mercuryAS MercuryAS STARTED 6
here click on "Thread dump"

Close the thread if your issue is resolved to make the thread meaningful and complete. Do show gratitude to people who helped you by assigning points.
Contributor.. Andrewartha Contributor..
Contributor..

Re: mercuryAS not starting on the DPS

Jump to solution

Thank-you for your prompt response, Sujith.

More details as requested:
1. waiting over 24 hours and mercuryAS process still reports STARTING
2. stop/starting in order: Stop GW >> Stop DPS >> Start DPS >> Start GW
3. Tomcat (not IIS)
4. port :80 appears free (netstat -an)
5. nanny_all.log has been captured and reviewed -- see attached
6. jvm_statistics.log has been captured and reviewed  -- see attached
7. nothing appears to fail in above 2 logfiles
8. can locate the line mercuryAS MercuryAS STARTING 1 6
9. cannot see this option: here click on "Thread dump"

It seems Thread dump isn't possible for mercuryAS stuck at STARTING (not STARTED).
>> C:\HPBSM\opr\support>opr-support-utils.bat -generateThreadDump mercuryAS
>> 'mercuryAS' service is not running!

Please share any extra pointers on what and how to troubleshoot further.

Thanks again in advance.

0 Likes
Sujith Kumar C. Acclaimed Contributor.
Acclaimed Contributor.

Re: mercuryAS not starting on the DPS

Jump to solution

Thanks. What is the time differce between BSM servers and DB servers. Can you get these log files?

topaz_all.ejb
topaz_all
jboss_boot
jboss_server
mercuryas.log

-- Sujith

Close the thread if your issue is resolved to make the thread meaningful and complete. Do show gratitude to people who helped you by assigning points.
0 Likes
Sujith Kumar C. Acclaimed Contributor.
Acclaimed Contributor.

Re: mercuryAS not starting on the DPS

Jump to solution

Also upload the logs form %topaz_home%\log\supervisor. Lets stop services and clear working cache by doing the following. Take a backup before deleting them (usually not required as they are created after restart)

Go to %HPBAC_HOME%/EJBContainer/server/mercury/tmp/deploy, and delete its contents.
Go to %HPBAC_HOME%/EJBContainer/server/mercury/tmp/sessions, and delete its contents.
Delete jboss.web by going to %HPBAC_HOME%/EJBContainer/server/mercury/work

 

Close the thread if your issue is resolved to make the thread meaningful and complete. Do show gratitude to people who helped you by assigning points.
0 Likes
Contributor.. Andrewartha Contributor..
Contributor..

Re: mercuryAS not starting on the DPS

Jump to solution

Thanks for your suggestions, Sujith.

Solved. DPS is now showing as Started.

Symptom was in error log
odb_boot.log [Starter for "view" to Active mode. (customer 1, id name: Default Client)] (CustomerController.java:413) ERROR - Failed to start component [view]
java.lang.IllegalArgumentException: The received string [$TempObject00000001_xxxucmdbsrv4012______] representation is inconsistent
at com.mercury.topaz.cmdb.server.fcmdb.spi.data.query.types.ExternalIdImpl.restoreDataStore(ExternalIdImpl.java:207)

Solution was
Removal of Cmdb Object  via JMX :21212
(My problematic ObjectID was visible in cmdb.query.detailed.log after turning up loglevel in cmdb.properties to DEBUG)

For the record
* DPS/GW clocks showed the same time
* clearing JBoss cache already tried, as per this post
* I captured the logs suggested by Sujith but haven't uploaded them to my post

Also tried
* Sonic MQ uninstall/install.bat
* run_db_verify.bat and applied output *.sql fixes to the Database

Thanks again

Absent Member.. obula Absent Member..
Absent Member..

Re: mercuryAS not starting on the DPS

Jump to solution

HI Sir and madam,
I have tried to set up new environment for BSM 9.26 in windows 2008 , even i am also facing  same issue  , in DPS server mercury AS services are not starting i have tried all steps which  you have provided , but still mercury as service in started mode, so  i need your help, so could you please share ur phone number or email id . or when ever you are free  could you please call me @91-8978889964.
==
-

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.