RUM engine fails to update configuration from BSM

Hi Everyone!

 

I'm having an issue with my rum engine this morning that he's unable to update configuration from the BSM server. The system health within RUM engine shows that area as red. When I try to update the config I get an error and when going into the config.manager log I can see the following:

 

2013-02-01 13:01:49,510 [http-0.0.0.0-8180-Processor8] (ConfigManager.java:162) ERROR - Retrieve configuration failed
com.mercury.rum.engine.configmanager.configretriever.DataTypeRetrieverException: Failed to execute configuration request from BAC for data type [APPLICATIONS]
at com.mercury.rum.engine.configmanager.configretriever.AbstractTMCRetriever.retrieve(AbstractTMCRetriever.java:102)
at com.mercury.rum.engine.configmanager.configretriever.NonApplicationBasedRetriever.retrieve(NonApplicationBasedRetriever.java:36)
at com.mercury.rum.engine.configmanager.configretriever.FileDecoratorRetriever.retrieve(FileDecoratorRetriever.java:41)
at com.mercury.rum.engine.configmanager.configretriever.ApplicationsRetriever.retrieve(ApplicationsRetriever.java:25)
at com.mercury.rum.engine.configmanager.configretriever.ConfigurationRetriever.handleRetrieveConfiguration(ConfigurationRetriever.java:140)
at com.mercury.rum.engine.configmanager.configretriever.ConfigurationRetriever.retrieveConfiguration(ConfigurationRetriever.java:116)
at com.mercury.rum.engine.configmanager.ConfigManager.retrieveConfiguration(ConfigManager.java:132)
at com.mercury.rum.engine.configmanager.ConfigManager.forceGetAllConfiguration(ConfigManager.java:176)
at com.mercury.rum.engine.configmanager.jmx.ConfigManagerConfigurationJMX.forceGetAllConfiguration(ConfigManagerConfigurationJMX.java:77)

 

in the end I can see this error(some strange IO Exception)

 

Caused by: com.mercury.rum.engine.configmanager.tmc.TMCHttpRequestInvokerException: TMC Request [GetMonitoredApplicationsData] failed on IOEXception: java.net.SocketTimeoutException: Read timed out
at com.mercury.rum.engine.configmanager.tmc.TMCRequestInvoker.execute(TMCRequestInvoker.java:72)
at com.mercury.rum.engine.configmanager.configretriever.AbstractTMCRetriever.retrieve(AbstractTMCRetriever.java:100)
... 96 more

Parents
  • Check whether you are able to ping the BSM server. Also try restarting the engine and attach topologyengine.log here!!!

  • Have you made any changes to the Applications associated to this RUM Engine, before problem started?

  • Hi!

     

    I am able to ping the BSM server from the engine and vice versa. I can also open the BSM console on the rum engine server. Tried restarting the server itself but that did not change anything.

    I did not do any changes to application - all was working nice and smoothely untill last week.

     

    Here is the topology log 

     

    at com.mercury.rum.engine.topologyengine.configurationreporter.ConfigurationReporter$ResolveNotificationHandler.handleNotification(ConfigurationReporter.java:222)
    at com.mercury.rum.engine.common.jmx.timer.NotificationAdapter.handleNotification(JMXTimer.java:432)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.mx.notification.NotificationListenerProxy.invoke(NotificationListenerProxy.java:138)
    at $Proxy9.handleNotification(Unknown Source)
    at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
    at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
    at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
    at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
    at javax.management.timer.Timer.sendNotification(Timer.java:1237)
    at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1206)
    at javax.management.timer.TimerAlarmClock.run(Timer.java:1289)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
    2013-02-01 11:13:21,572 [MessageListenerThread - RUMEntryTopic1] (CacheLimitAccumulatorLogger.java:63) WARN - 3816 objects thrown in Page2Bsm Reporter between 11:08:18 and 11:13:21
    2013-02-01 11:18:24,384 [MessageListenerThread - RUMEntryTopic1] (CacheLimitAccumulatorLogger.java:63) WARN - 187 objects thrown in Page2Bsm Reporter between 11:13:21 and 11:18:24
    2013-02-01 11:23:24,903 [MessageListenerThread - RUMEntryTopic1] (CacheLimitAccumulatorLogger.java:63) WARN - 5458 objects thrown in Page2Bsm Reporter between 11:18:24 and 11:23:24
    2013-02-01 11:28:37,162 [MessageListenerThread - RUMEntryTopic1] (CacheLimitAccumulatorLogger.java:63) WARN - 261 objects thrown in Page2Bsm Reporter between 11:23:26 and 11:28:37
    2013-02-01 11:37:33,767 [main] (AbstractTopologyEngine.java:73) INFO - creating TopologyEngine
    2013-02-01 11:37:34,689 [main] (AbstractTopologyEngine.java:79) INFO - starting TopologyEngine
    2013-02-01 11:37:34,768 [main] (AbstractTopologyEngine.java:134) INFO - topologyEngine configuration in cdal is empty
    2013-02-01 11:37:35,080 [main] (AbstractTopologyEngine.java:177) INFO - TopologyEngine started successfully
    2013-02-01 15:50:35,308 [http-0.0.0.0-8180-Processor20] (CMDBHelper.java:81) INFO - connecting to cmdb with http to nt032pbsmapp.ads.swc on port 80 with user name rum_integration_user
    2013-02-01 15:50:35,324 [http-0.0.0.0-8180-Processor20] (CMDBHelper.java:86) INFO - connected to customer Default Client
    2013-02-01 15:50:37,121 [Timer-19] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:ClientHostCmdbResolver
    2013-02-01 18:10:25,015 [JBoss Shutdown Hook] (AbstractTopologyEngine.java:181) INFO - stopping TopologyEngine
    2013-02-01 18:10:25,030 [JBoss Shutdown Hook] (AbstractTopologyEngine.java:194) INFO - TopologyEngine stopped successfully
    2013-02-01 18:10:25,218 [JBoss Shutdown Hook] (AbstractTopologyEngine.java:181) INFO - stopping TopologyEngine
    2013-02-01 18:10:25,218 [JBoss Shutdown Hook] (AbstractTopologyEngine.java:194) INFO - TopologyEngine stopped successfully
    2013-02-01 18:10:25,218 [JBoss Shutdown Hook] (TopologyEngine.java:52) INFO - TopologyEngine destroyed
    2013-02-01 18:11:11,110 [main] (AbstractTopologyEngine.java:73) INFO - creating TopologyEngine
    2013-02-01 18:11:12,313 [main] (AbstractTopologyEngine.java:79) INFO - starting TopologyEngine
    2013-02-01 18:11:12,470 [main] (AbstractTopologyEngine.java:134) INFO - topologyEngine configuration in cdal is empty
    2013-02-01 18:11:12,954 [main] (AbstractTopologyEngine.java:177) INFO - TopologyEngine started successfully
    2013-02-01 18:11:37,033 [Timer-47] (CMDBHelper.java:81) INFO - connecting to cmdb with http to nt032pbsmapp.ads.swc on port 80 with user name rum_integration_user
    2013-02-01 18:11:37,033 [Timer-47] (CMDBHelper.java:86) INFO - connected to customer Default Client
    2013-02-01 18:11:43,018 [Timer-18] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-01 20:15:04,539 [Timer-18] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-02 03:05:11,678 [Timer-19] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:ClientHostCmdbResolver
    2013-02-02 09:14:01,853 [Timer-17] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SubnetCmdbResolver
    2013-02-02 13:35:17,887 [Timer-19] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:ClientHostCmdbResolver
    2013-02-03 01:43:34,915 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-03 05:03:23,972 [Timer-17] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SubnetCmdbResolver
    2013-02-03 07:26:55,493 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-03 12:28:57,355 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-03 17:30:55,886 [Timer-18] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-03 18:12:04,762 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-04 00:36:22,184 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-04 05:02:21,547 [Timer-19] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:ClientHostCmdbResolver

Reply
  • Hi!

     

    I am able to ping the BSM server from the engine and vice versa. I can also open the BSM console on the rum engine server. Tried restarting the server itself but that did not change anything.

    I did not do any changes to application - all was working nice and smoothely untill last week.

     

    Here is the topology log 

     

    at com.mercury.rum.engine.topologyengine.configurationreporter.ConfigurationReporter$ResolveNotificationHandler.handleNotification(ConfigurationReporter.java:222)
    at com.mercury.rum.engine.common.jmx.timer.NotificationAdapter.handleNotification(JMXTimer.java:432)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.mx.notification.NotificationListenerProxy.invoke(NotificationListenerProxy.java:138)
    at $Proxy9.handleNotification(Unknown Source)
    at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
    at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
    at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
    at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
    at javax.management.timer.Timer.sendNotification(Timer.java:1237)
    at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1206)
    at javax.management.timer.TimerAlarmClock.run(Timer.java:1289)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
    2013-02-01 11:13:21,572 [MessageListenerThread - RUMEntryTopic1] (CacheLimitAccumulatorLogger.java:63) WARN - 3816 objects thrown in Page2Bsm Reporter between 11:08:18 and 11:13:21
    2013-02-01 11:18:24,384 [MessageListenerThread - RUMEntryTopic1] (CacheLimitAccumulatorLogger.java:63) WARN - 187 objects thrown in Page2Bsm Reporter between 11:13:21 and 11:18:24
    2013-02-01 11:23:24,903 [MessageListenerThread - RUMEntryTopic1] (CacheLimitAccumulatorLogger.java:63) WARN - 5458 objects thrown in Page2Bsm Reporter between 11:18:24 and 11:23:24
    2013-02-01 11:28:37,162 [MessageListenerThread - RUMEntryTopic1] (CacheLimitAccumulatorLogger.java:63) WARN - 261 objects thrown in Page2Bsm Reporter between 11:23:26 and 11:28:37
    2013-02-01 11:37:33,767 [main] (AbstractTopologyEngine.java:73) INFO - creating TopologyEngine
    2013-02-01 11:37:34,689 [main] (AbstractTopologyEngine.java:79) INFO - starting TopologyEngine
    2013-02-01 11:37:34,768 [main] (AbstractTopologyEngine.java:134) INFO - topologyEngine configuration in cdal is empty
    2013-02-01 11:37:35,080 [main] (AbstractTopologyEngine.java:177) INFO - TopologyEngine started successfully
    2013-02-01 15:50:35,308 [http-0.0.0.0-8180-Processor20] (CMDBHelper.java:81) INFO - connecting to cmdb with http to nt032pbsmapp.ads.swc on port 80 with user name rum_integration_user
    2013-02-01 15:50:35,324 [http-0.0.0.0-8180-Processor20] (CMDBHelper.java:86) INFO - connected to customer Default Client
    2013-02-01 15:50:37,121 [Timer-19] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:ClientHostCmdbResolver
    2013-02-01 18:10:25,015 [JBoss Shutdown Hook] (AbstractTopologyEngine.java:181) INFO - stopping TopologyEngine
    2013-02-01 18:10:25,030 [JBoss Shutdown Hook] (AbstractTopologyEngine.java:194) INFO - TopologyEngine stopped successfully
    2013-02-01 18:10:25,218 [JBoss Shutdown Hook] (AbstractTopologyEngine.java:181) INFO - stopping TopologyEngine
    2013-02-01 18:10:25,218 [JBoss Shutdown Hook] (AbstractTopologyEngine.java:194) INFO - TopologyEngine stopped successfully
    2013-02-01 18:10:25,218 [JBoss Shutdown Hook] (TopologyEngine.java:52) INFO - TopologyEngine destroyed
    2013-02-01 18:11:11,110 [main] (AbstractTopologyEngine.java:73) INFO - creating TopologyEngine
    2013-02-01 18:11:12,313 [main] (AbstractTopologyEngine.java:79) INFO - starting TopologyEngine
    2013-02-01 18:11:12,470 [main] (AbstractTopologyEngine.java:134) INFO - topologyEngine configuration in cdal is empty
    2013-02-01 18:11:12,954 [main] (AbstractTopologyEngine.java:177) INFO - TopologyEngine started successfully
    2013-02-01 18:11:37,033 [Timer-47] (CMDBHelper.java:81) INFO - connecting to cmdb with http to nt032pbsmapp.ads.swc on port 80 with user name rum_integration_user
    2013-02-01 18:11:37,033 [Timer-47] (CMDBHelper.java:86) INFO - connected to customer Default Client
    2013-02-01 18:11:43,018 [Timer-18] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-01 20:15:04,539 [Timer-18] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-02 03:05:11,678 [Timer-19] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:ClientHostCmdbResolver
    2013-02-02 09:14:01,853 [Timer-17] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SubnetCmdbResolver
    2013-02-02 13:35:17,887 [Timer-19] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:ClientHostCmdbResolver
    2013-02-03 01:43:34,915 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-03 05:03:23,972 [Timer-17] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SubnetCmdbResolver
    2013-02-03 07:26:55,493 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-03 12:28:57,355 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-03 17:30:55,886 [Timer-18] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-03 18:12:04,762 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-04 00:36:22,184 [Timer-47] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:SoftwareElementCmdbResolver
    2013-02-04 05:02:21,547 [Timer-19] (CmdbResolver.java:252) INFO - can't resolve now. a former resolving is still in progress:ClientHostCmdbResolver

Children