Highlighted
Trusted Contributor.
Trusted Contributor.
1036 views

Can you monitor ESM (and any other ArcSight components) by JMX?

Jump to solution

Hi,

With ESM, we occasionally have some performance-related issues with rules, data monitors, etc. Whilst we're familiar with the old web management interface, that shows you the various MBeans and allows you to turn performance tracing on and off, ideally we'd like to be able to be continually consuming this information in a separate, more traditional, monitoring platform such as SolarWinds, Nagios, etc.

It's not beyond the wit of man to write a shell-script that can login and hack the relevant parts out of the web-page into a format that can be consumed by one of those tools, however it would be much nicer if that information could be consumed over JMX directly.

Has anyone else done this? I suspect if I just went and found the right properties file/shell script and set the various -Dcom.sun.management.jmx.remote properties, it might 'just work', but I'm interested to hear the experiences of others.

Equally it'd be good to do something similar for the other components we use, such as Logger and Connector. Whilst there is a decent amount of information you can get out of agent:043, agent:050 et al, it would be good to have a second independent system monitoring it as well as we do find that occasionally if we're seeing performance issues somewhere in the data-processing chain, it can take a while for the agent events to arrive, or they don't arrive at all.

Thanks,

Jamie

Labels (1)
0 Likes
1 Solution

Accepted Solutions
Highlighted
Trusted Contributor.
Trusted Contributor.

So - with a bit of digging I've found that it looks as if you can specify additional JVM arguments (for the manager java process at least) as shown by the various wrapper.java.additional properties in /opt/arcsight/manager/config/server.defaults.wrapper.conf


e.g:

wrapper.java.additional.8=-XX:+UseParallelOldGC

                            

you can add more, such as: wrapper.java.additional.12=-Dcom.sun.management.jmx.remote

Doing a fairly trivial google search for "how do you enable JMX in a JVM" turns up plenty of helpful answers which suggest that the following is the minimum (and terribly insecure) required to get it going (development and test only, not fit for production):

-Dcom.sun.management.jmxremote

-Dcom.sun.management.jmxremote.port=9010

-Dcom.sun.management.jmxremote.local.only=false

-Dcom.sun.management.jmxremote.authenticate=false

-Dcom.sun.management.jmxremote.ssl=false

Restart the manager and fire up jconsole. I just aimed it servername:9010 and it worked first time.

Interesting things I was able to find:

Under ArcSight>Live>RulesEngine

  • Was able to turn performance tracing on
  • Was then able to see LoadedRules, RulesEngineTimingStats and RulesEngineMemoryStats, as you can through manage.jsp
  • It might be interesting to be able to script/monitor the collection of rule performance statistics.
  • Only wrinkle seems to be that the data table is stored as a List<String> (list of Strings). The fields are pipe-delimited within that. Whether or not a monitoring system such as Hyperic or Nagios can post-process data that comes back over JMX, I'm not sure yet. Otherwise you'd have to resort to a scripting language to pull the data out and post-process it.

Similarly under Arcsight>MysqlActiveListBroker and Arcsight>MysqlSessionListBroker you can find the monitors for ActiveLists and SessionLists  (high queries/sec changes/sec)

  • Might be interesting to keep an eye on the number of queries/changes per second to some activelist and sessionlists. This has previously identified some badly-behaving rules for us.
  • Similar issues to above, post-processing of Lists of Strings required.

I wouldn't recommend fiddling around with /opt/arcsight/manager/config/server.defaults.wrapper.conf on your production installation. Nor would I recommend those JMX settings either as they are insecure and expose the ability to do things like shutdown the JVM. More thought needed to take this further.

View solution in original post

0 Likes
1 Reply
Highlighted
Trusted Contributor.
Trusted Contributor.

So - with a bit of digging I've found that it looks as if you can specify additional JVM arguments (for the manager java process at least) as shown by the various wrapper.java.additional properties in /opt/arcsight/manager/config/server.defaults.wrapper.conf


e.g:

wrapper.java.additional.8=-XX:+UseParallelOldGC

                            

you can add more, such as: wrapper.java.additional.12=-Dcom.sun.management.jmx.remote

Doing a fairly trivial google search for "how do you enable JMX in a JVM" turns up plenty of helpful answers which suggest that the following is the minimum (and terribly insecure) required to get it going (development and test only, not fit for production):

-Dcom.sun.management.jmxremote

-Dcom.sun.management.jmxremote.port=9010

-Dcom.sun.management.jmxremote.local.only=false

-Dcom.sun.management.jmxremote.authenticate=false

-Dcom.sun.management.jmxremote.ssl=false

Restart the manager and fire up jconsole. I just aimed it servername:9010 and it worked first time.

Interesting things I was able to find:

Under ArcSight>Live>RulesEngine

  • Was able to turn performance tracing on
  • Was then able to see LoadedRules, RulesEngineTimingStats and RulesEngineMemoryStats, as you can through manage.jsp
  • It might be interesting to be able to script/monitor the collection of rule performance statistics.
  • Only wrinkle seems to be that the data table is stored as a List<String> (list of Strings). The fields are pipe-delimited within that. Whether or not a monitoring system such as Hyperic or Nagios can post-process data that comes back over JMX, I'm not sure yet. Otherwise you'd have to resort to a scripting language to pull the data out and post-process it.

Similarly under Arcsight>MysqlActiveListBroker and Arcsight>MysqlSessionListBroker you can find the monitors for ActiveLists and SessionLists  (high queries/sec changes/sec)

  • Might be interesting to keep an eye on the number of queries/changes per second to some activelist and sessionlists. This has previously identified some badly-behaving rules for us.
  • Similar issues to above, post-processing of Lists of Strings required.

I wouldn't recommend fiddling around with /opt/arcsight/manager/config/server.defaults.wrapper.conf on your production installation. Nor would I recommend those JMX settings either as they are insecure and expose the ability to do things like shutdown the JVM. More thought needed to take this further.

View solution in original post

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.