Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Anybody got some news on that?

On my side, support had me add a Java additional Parameter for ReservedCodeCacheSize and since then, no caching issue. It's been 3 weeks now, it was caching and I had to restart the manager about once a week before.

Curious if anybody had some similar results?

0 Likes
superman Respected Contributor.
Respected Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Caching 'looping' issue still comes up for us once in a while.    Could you share how high support had you set the ReservedCodeCacheSize  ?

-XX:ReservedCodeCacheSize=size

     Sets the maximum code cache size (in bytes) for JIT-compiled code. Append the letter k or K to indicate kilobytes, m or M to indicate megabytes, g or G to indicate gigabytes. This option is equivalent to -Xmaxjitcodesize.

REF: http://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html

bug fixes in JVM 6u60 - Java™ SE Development Kit 6 Update 60 Bug Fixes

GC stuff - g1gc logs - Ergonomics -how to print and how to understand (G1GC Concept and Performance Tuning)

0 Likes
Highlighted
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

You need to add a line in the server.defaults.wrapper.conf file, in the additional parameters section.

My line:  wrapper.java.additional.12=-XX:ReservedCodeCacheSize=256m

The index (12 in my case) needs to be adjusted depending on your file and changes you may have done to it.

View solution in original post

0 Likes
superman Respected Contributor.
Respected Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Thanks,  I will test this out on our LAB system.

Here is a list of Performance options available on arcsight jvm:

-XX:+AggressiveOpts 

-XX:+AggressiveHeap 

-XX:ParallelGCThreads=2 

-XX:InterpreterProfilePercentage=10 

-XX:CompileThreshold=35000 

-XX:OnStackReplacePercentage=80 

-XX:ReservedCodeCacheSize=128m 

-XX:PermSize=256m 

-XX:MaxPermSize=256m 

-XX:AllocatePrefetchStepSize=128 

-XX:AllocatePrefetchStyle=2 

-XX:AllocatePrefetchDistance=1024 

-Xmx, -Xms, -Xmn should have sufficiently large values. (For example, -Xms6600m -Xmn5000m -Xmx6600m or -Xms20000m -Xmn18000m -Xmx20000m.)

0 Likes
Established Member.. raymond.doty
Established Member..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

We got this suggestion from Support too.  I will be testing it in dev soon.  Same size as you mentioned.  I think we also found that there are two other things that have a pretty big impact on insertion performance...  Bad incoming timestamps and the amount of memory assigned to server.wrapper (it seems to slow down the issue, assuming you can handle larger settings).

0 Likes
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

I also noticed that bad or "in the past" timestamps affect the insertion time.

I'm running 16Gb of memory for the manager and we were facing the "caching, needs a restart to fix" issue about once a week. We have 256Gb RAM on the server so I guess I could go higher, but I don't think I'd see a big improvement.

0 Likes
Established Member.. raymond.doty
Established Member..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

So, if you are full GC'ing once an hour and your average peak memory utilization is below 90%, I don't suspect it will help much.  But if your Full GCs are running more often and your peak memory is extremely high, I might consider testing it.

Just make sure that the GCs are running in a sufficient amount of time

0 Likes
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Holy crap, last time I remember checking, Full GCs were about at the one hour mark, but now, they are every 8-9 minutes and take about 3 seconds.

Guess I'll look into this!!

0 Likes
Established Member.. raymond.doty
Established Member..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Ours recently changed as well.  We have been running at 32gb for the past 6 months or so.. In the past month, they went from hourly during peak to ~30-40 minutes at peak, so we are looking at adjusting ours as well.

Also, figured I should rework my GC script... Put it on the original post but, perhaps someone else will find it useful

(updated script again, to get gc mem and time run info)

grep "Full GC" -A1 /opt/arcsight/manager/logs/default/server.std.log* | awk '{print $6 " " $7 " | " $9 " " $10}' | sort

0 Likes
Honored Contributor.. dhartman Honored Contributor..
Honored Contributor..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Of course I read this comment AFTER i clicked the restart rules engine button which of course didnt start the rules engine back up.  It would be handy to know how to start the rules engine when the restart doesnt work...if the restart ever works that is ;-D

0 Likes
Established Member.. raymond.doty
Established Member..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Answer wrote:

You need to add a line in the server.defaults.wrapper.conf file, in the additional parameters section.

My line:  wrapper.java.additional.12=-XX:ReservedCodeCacheSize=256m

The index (12 in my case) needs to be adjusted depending on your file and changes you may have done to it.

NOTE: Do not modify this parameter without first consulting support.  This is a pretty dramatic change and there is definitely a wrong way to do it!

We finally have tested this in both our production and development environments.  We haven't had a reoccurance in dev in about 3 months and in prod in about 1 month.  This seems like a success.  We have had moments where the ESM will start to cache across the connectors, but it seems to recover without a restart.  Thanks for helping everyone, I am sure there may be other symptoms that are similar, but this seems to be resolved on our system.

0 Likes
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Ray, I'm glad your problem was also resolved with this magical line. Our prod box has been running strong for 3 months now. I did also saw occurences where cache would build up across all the connectors, but it quickly goes back to normal, so it's fine with me. Global cache rarely gets over 1 million (with 12-14k EPS), so no worries there.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.