Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..
853 views

conApp appliance / connector high CPU usage over time

One of the conApp appliances here was showing upwards of 28% CPU on the console today and "stuck" there.  I checked the process status view on the conApp console and all of the connectors / containers - in fact everything - was showing ~1% CPU utilization.  So I logged into the CLI and ran the Top command on the box, sorted it by %CPU, and toggled the display to show full command details - this revealed one process that was running between 23 and 28% CPU consistently. 

I'm not sure what the command means:  "/opt/arcsight/connectors/connector_1/current/jre/bin/java -server -XX:MaxNewSize=128m -verbose:g"

A simple restart of that container via the console resolved the issue - so far.  The new PID (24133 in the screen cap below) and other related connector commands seem to indicate everything is quiet again, nothing hogging CPU cycles now.

This is a syslog connector installed (alone) in container 1 with 1GB memory allocated to that container - the appliance has 32 GB total.  None of the other connectors seem to be misbehaving.  This syslog connector was not / is not processing any events, but it was being staged for pending use.  It should be almost a flatline for CPU and memory usage at this time.  Another syslog connector on another conApp is moving 800 EPS and it hardly hits 3% CPU. 

I don't know where to look (specific logs etc.)  for clues as to why this would happen, or what I can do to prevent it from happening again.

As usual any ideas are appreciated.

Labels (2)
0 Likes
Reply
8 Replies
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: conApp appliance / connector high CPU usage over time

Do you happen to have a screencap of the top output PRIOR to restarting the container1?

If you gather up the logs for the container and post them here, there may still be some evidence as to what was wrong. Perhaps a regex got into a recursive-backtrack situation and was pegged the CPU...

0 Likes
Reply
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Re: conApp appliance / connector high CPU usage over time

I thought I had a screen cap prior, but must have fat fingered it with the 2nd capture (my bad).

Are there specific logs to start with?  I see several agent.log.*,   agent.out.wrapper.log.*, plus a couple dozen ThreadDump files currently in the /opt/arcsight/connectors/connector_1/current/logs directory

FWIW looking at the agent.log files, most of the files are filled with this (time stamp prior to the container restart):

[2015-05-21 11:03:27,433][ERROR][default.com.arcsight.util.AgentUtil][dumpException] java.lang.IllegalArgumentException

        at sun.util.calendar.ZoneInfo.getOffset(ZoneInfo.java:378)

        at com.arcsight.util.AgentUtil.getLocalTimezoneOffsetInMinutes(AgentUtil.java:168)

        at com.arcsight.agent.pe.b.e(b.java:128)

        at com.arcsight.agent.pe.b.run(b.java:74)

edit - just attached agent.log.1 which spans the time that I restarted the container (2015-05-21 11:03:29).  The rest of the agent.log series are just filled with repeats of the above ERROR message

also attached agent.out.wrapper.log which also spans / captures the container restart event

0 Likes
Reply
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Re: conApp appliance / connector high CPU usage over time

today a recurrence of the above issue - same conApp appliance but this time it is connector_7 that is the culprit (this is a syslog TCP connector)

CPU on this conApp normally is under 3% total - today shows 25% on the GUI, about 21% of that is PID 32562.   None of the connector processes show anything over 1% on the system admin GUI. 

Once again all the agent.log files are completely filled with the exact same error as before (with connector_1).

[2015-05-27 11:22:18,143][ERROR][default.com.arcsight.util.AgentUtil][dumpException] java.lang.IllegalArgumentException

    at sun.util.calendar.ZoneInfo.getOffset(ZoneInfo.java:378)

    at com.arcsight.util.AgentUtil.getLocalTimezoneOffsetInMinutes(AgentUtil.java:168)

    at com.arcsight.agent.pe.b.e(b.java:128)

    at com.arcsight.agent.pe.b.run(b.java:74)

Any insights?  3 other conApps - all have syslog connectors running, some are much busier that this (EPS) - but none of them exhibit this behaviour.  What's wrong with this conApp?

0 Likes
Reply
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: conApp appliance / connector high CPU usage over time

Is there a specific log source that only this connector collects? (that the other working ones dont?) Any flexconnectors?

I'd also check the TimesZone config on the connector destinations. The error seems to be related to timestamp normalization, which can occur in the parser or on the destination parameters. See the "TimeCorrection/Device Time AutoCorrection/TimeChecking" sections of the destinations for some odd setting.

0 Likes
Reply
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Re: conApp appliance / connector high CPU usage over time

Thanks Richard - the syslog connectors aren't dedicated or allocated to specific log sources - each connector has 4 Logger destinations (1 primary + 1 failover at each of 2 sites).

I will sanity check the timezone configs in the destinations as suggested.

fwiw I just restarted this connector and the CPU has settled down again.  The original issue (with connector_1) is still behaving after the restart @ May 21.

thanks for the help....

EDIT - connector_7 is in fact dedicated to syslog TCP traffic - coming from a McAfee product (which sends syslog CEF format).   However, we have the identical setup (another conApp, syslog TCP connector / container 7, McAfee gateway / CEF source, etc.)  at the other site, and it's fine. 

Also the original issue (May 21 - "connector_1") involved a syslog connector that eats windows, Cisco, other device events. 

The time settings on all 4 Logger destinations appear to be the same (default)

0 Likes
Reply
Highlighted
New Member.

Re: conApp appliance / connector high CPU usage over time

Hello,

I have the same error in the agent.log

[2016-09-11 23:59:59,999][ERROR][default.com.arcsight.util.AgentUtil][dumpException] java.lang.IllegalArgumentException

        at sun.util.calendar.ZoneInfo.getOffset(ZoneInfo.java:391)

        at com.arcsight.util.AgentUtil.getLocalTimezoneOffsetInMinutes(AgentUtil.java:169)

        at com.arcsight.agent.bg.a.a(a.java:128)

        at com.arcsight.agent.bg.a.run(a.java:74)

[2016-09-11 23:59:59,999][ERROR][default.com.arcsight.util.AgentUtil][dumpException] java.lang.IllegalArgumentException

        at sun.util.calendar.ZoneInfo.getOffset(ZoneInfo.java:391)

        at com.arcsight.util.AgentUtil.getLocalTimezoneOffsetInMinutes(AgentUtil.java:169)

        at com.arcsight.agent.bg.a.a(a.java:128)

        at com.arcsight.agent.bg.a.run(a.java:74)

But in my case the error starts at 00:00:00.000 and ends at 23:59:59,999

This means we have these errors 24 hours, and what is also strange it mostly occurs once a week, but each time on a different connector.

Any suggestions?

KInd Regards,
Geurt-Jaap Evers
0 Likes
Reply
Highlighted
New Member.

Re: conApp appliance / connector high CPU usage over time

Just started seeing this...did u get a solution?

I am seeing it may be related to timestamps?

0 Likes
Reply
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Re: conApp appliance / connector high CPU usage over time

hi Carl - is this on an ArcMC appliance or other connector?  I did not get to the root of this.  Over time, everything changed - the syslog flow (retired these source devices), connector versions, conApp migration (to ArcMC), all the ADP stuff there and upstream, new ESM's etc etc etc.  Too many little issues to chase in a dynamic environment, like playing whack-a-mole in the old arcades....  

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.