ALERT! The community will be read-only starting on April 19, 8am Pacific as the migration begins. Read more for important details.
ALERT! The community will be read-only starting on April 19, 8am Pacific as the migration begins.Read more for important details.
Absent Member.
Absent Member.
4801 views

Tweaking the amount of memory a SmartConnector uses

Jump to solution

It's my understanding that a SmartConnector can consume up to 256 MB of memory by default. I'm wondering if this can be tweaked down, particularly for a connector that works with a rather small log file. Since I'll be running several connectors on a single server, 256 MB per SmartConnector adds up quickly.

Thanks!

Labels (2)
0 Likes
1 Solution

Accepted Solutions
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Yep - edit the agent.wrapper.conf file (assuming you're running it as a service):

wrapper.java.maxmemory=256

If you reduce it, make sure to change this parameter too:  wrapper.java.initmemory=256  

View solution in original post

0 Likes
20 Replies
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Yep - edit the agent.wrapper.conf file (assuming you're running it as a service):

wrapper.java.maxmemory=256

If you reduce it, make sure to change this parameter too:  wrapper.java.initmemory=256  

View solution in original post

0 Likes
Absent Member.
Absent Member.

Oh you mean that file sitting right under my nose that I missed?

It worked! Thanks (again) for your help, and for the very fast response!

0 Likes

I would strongly advise that you research your connectors heap (memory) needs prior to making tuning adjustments.

You can monitor, at the system level (taskmgr on Windows or top on Unix-like systems), and readily notice that connectors virtual space exceeds that of its physical space, which is very common for processes. The configured memory quota is not the sole indicator of actual (active) memory usage, i.e. a virtual address space does not directly translate to a 1:1 actual resident memory usage and therefore a deployment of four connectors defaulted at a init/max heap of 256MB does not mean that your actual physical (ram) demand is 1GB.

A significant factor in the active use of the heap is the event rate.

Examine you connector('s) agent.out.wrapper.log for both garbage connection and events per second statistics;

- garbage collection log entries look like ".......[GC ######K->######K(######K), #.###### secs]"

- In the above; GC means 'garbage collection'. Note: You do NOT want to see a lot of 'Full GC' entries. What is 'a lot'? I would suggest that any occurance other than one every 30+ minutes, however there isn't a hard and fast rule for this. This is driven by the event rate and the type of connector. If you are seeing 'Full GC' then you need to increase the heap rather than decrease it.

- The first #####K is the heap size at the start of the GC

- The second ####K is the heap size at the end of the GC

- The third ####K is the configured heap.

- Note that the JRE will actually be larger than 256MB, the heap is after all the stack working area, as 'other memory' and the actual java code need also be accounted for so don't be surprised that your 256MB conenctors is actually ~ 384MB. Again virtual and since not all code gets executed over time the operating system will adapt to overall memory demand and page (swap) out what isn't being used.

- The #.#### secs is also very important to monitor. This should normally be on the order of .01-.10, the lesser the better.

- When this starts increasing to multiple seconds of duration you are having severe memory demand and garbage collection is trying to keep up and having a harder time of it. When you get to about 10 seconds then things start falling apart, threads are being starved of access to CPU, and you will probably start seeing real performance issues from lost event (UDP receivers) to connections disconnects (WUC) and true system level memory issues and cpu related issues.

- The second ####K is where you can get your best estimate of what your connectors heap demand is.

- watch this over time to determine its (approximate) nominal value.

- The heap init size should probably start at 2x this value, e.g. 230000K->120000K(256000K) [numbers are for demonstration purposes] shows you that the 'floor' for the heap is about 120MB and therefore 240MB is your minimum heap starting point. BTW: for this example, H'mmm maybe subliminally I wasn't being as 'random' as I intended, I would not even touch the heap initial 256MB.

- The event rate per second, e.g. "......{Eps=####.######, Evts=#####}" [geez I really hate the fractions out to a grillion places of prevision - I mean really?!?!?!], is also interesting.

- connectors by event type, e.g. (Cisco) syslog .vs. (Windows) events, offer differing content.

- Are you aggregating, which 'holds' events for a while (the aggregation time).

- Multiple line parsing?

- all of these things impact heap consumption.

So,....

Please 'diagnose' prior to tuning.

I have seldom, actually I cannot cite a single case where I have, seen any system performance improvement that is based on reducing a connectors heap below the default initial 256MB. This does not mean that it doesn't exist/occur. It is just my observation.

Well, that is some of my thought on this.

Thank you,

Doug

Absent Member.
Absent Member.

Doug,

    Thank you for detailed response.

    I am going to look a little more closely per your suggestion before I do the tuning. I guess it's not as simple as I thought. You're right- I thought it was as simple as "4 Connectors = 1 GB of RAM. Apparently that is not the case.

    Thanks again!

0 Likes
Absent Member.
Absent Member.

Actually, I see 0 incidents of Garbage Collection...

But what I do see are many ThreadDumps. Like every few minutes it appears to be dumping.

That doesn't seem right.

Looks like I have something else to troubleshoot now...

0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

I'm going to have to disagree with parts this...  First off, seeing a Full GC less than 30 minutes apart isn't necessarily bad, and in fact increasing the heap memory can have negative consequences.  Threads are only starved of CPU time during Full GCs.  Since the JRE pauses during a Full GC, the more memory it has to clear, the longer it's paused.  If a connector is taking .25 seconds to do a Full GC every 5 minutes, but takes 10-15 seconds to do one every 30 minutes, thats a longer timespan when the connector isn't processing events - this becomes a much larger issue within ESM itself, since then you're talking about doing a Full GC on gigs worth of memory.  In addition, the number you should be concerned with, according to ArcSight themselves is the Full GC, not the normal(partial?  can't remember the proper term) one you see happening - for more info on this, see Christian's Rockstar ESM Admin presentation from Protect '09. 

Also, just because the JRE takes up memory itself doesn't mean that lowering the amount of heap won't help a system.  If your JRE is allocating 256MB and it uses 100MB itself, you're using 256MB.  If you half the heap (for example...this would probably kill it ), then the connector is still using less memory.  For a low EPS connector you won't need the full 256MB, and on system like connector appliances that have fixed amounts of memory and aren't field upgradable, memory management becomes even more vital.  Yes, you can use swap space, but it's not nearly as fast and if you have a heavily utilized connapp, swapping leads to performance degredation - kswapd being #1 in top for example - bad juju.

The key is to test and figure out what works best.  Allocating 256MB to a connector running at .5 EPS is just wasting memory.  If you have a system with 64GB of RAM, this isn't as much of an issue, but if you're only at 4GB of RAM, it becomes a scarce resource.

Just my $0.01573243

0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

What is causing the thread dumps?  Check the agent wrapper logs, they should give you some indication of what's going on.

0 Likes

Thread dumps are not necessarily an indication of a problem, but then again they can be... It's not that simple.

Thread dumps will 'naturally' occur when the EPS value is varying (significantly). Right off, I do not remember the % of change (over brief) time that the connector uses but I think that it is approximatley 50%. If the connector is seeing a EPS that drops, e.g. is 100EPS, and then rises, e.g. 250EPS, you can expect that a thread dump may be generated.

thread dumps can also be generated wen there are problems.

thread dumps are a diagnostic souce for support.

It is troubling that you are not seeing 'GC' at all. A 'healthy' connector will throw a log entry every second or so, maybe more or less often. it it is not then something is interfering with (at least) that function.

The few lines prior to the thread dump may help provide clues to the source of the thread dumps.

Don't Panic! (all due respect to Hitchhiker's...)

Do some 'grep' (windows findstr) for the current agent.out.wrapper.log for '[GC' and '{Eps=' to see if anything has been getting into the log. Also you can look at older log versions for a longer historical perspective. Also while you are there look for 'Full GC'. These things can be easy to miss if the log is wrapping quickly, which would be an indication of the need to do diagnosis!

Thank you,

Doug

0 Likes
Absent Member.
Absent Member.

I... am not sure. Here's what I have in the log:

INFO   | jvm 1    | 2011/06/30 11:03:26 | JVMDUMP013I Processed dump event "user", detail "".
INFO   | jvm 1    | 2011/06/30 11:03:57 | [Thu Jun 30 11:03:57 EDT 2011] [INFO ] {Eps=0.0, Evts=16}
INFO   | jvm 1    | 2011/06/30 11:03:57 | [Thu Jun 30 11:03:57 EDT 2011] [INFO ] {C=0, ET=Up, HT=Up, N=PSoft Fin Test WebLogic Logs,
S=13, T=0.0}
INFO   | jvm 1    | 2011/06/30 11:04:05 | [Thu Jun 30 11:04:05 EDT 2011] [INFO ] First event from [BEA|WebLogic||] received.
INFO   | jvm 1    | 2011/06/30 11:04:57 | [Thu Jun 30 11:04:57 EDT 2011] [INFO ] {Eps=1.65, Evts=115}
INFO   | jvm 1    | 2011/06/30 11:04:57 | [Thu Jun 30 11:04:57 EDT 2011] [INFO ] {C=0, ET=Up, HT=Up, N=PSoft Fin Test WebLogic Logs,
S=112, T=1.65}
INFO   | jvm 1    | 2011/06/30 11:05:56 | JVMDUMP006I Processing dump event "user", detail "" - please wait.
INFO   | jvm 1    | 2011/06/30 11:05:56 | JVMDUMP007I JVM Requesting Java dump using '/arcsight/FC-WebLogic/current/bin/wrapper/aix/
javacore.20110630.110556.774194.0003.txt'
INFO   | jvm 1    | 2011/06/30 11:05:56 | JVMDUMP010I Java dump written to /arcsight/FC-WebLogic/current/bin/wrapper/aix/javacore.20
110630.110556.774194.0003.txt

0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Need to correct myself there - all GCs cause a pause .  Brain fart.

For more info;  https://protect724.arcsight.com/servlet/JiveServlet/download/1122-5-2001/SN12_Beedgen.pdf page 61 and onward.  There's also a video in there if you want to watch it.

0 Likes
Absent Member.
Absent Member.

I appreciate the Hitchhiker's reference. 🙂

I grep'd the entire log for a GC reference and didn't find one. Numerous Eps's though:

[2011-06-30 11:01:57,430][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=0.21666666666666667, Evts=13}
[2011-06-30 11:02:57,453][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=0.05, Evts=16}
[2011-06-30 11:03:57,475][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=0.0, Evts=16}
[2011-06-30 11:04:57,489][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=1.65, Evts=115}
[2011-06-30 11:05:57,507][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=0.016666666666666666, Evts=116}
[2011-06-30 11:06:57,521][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=0.0, Evts=116}
[2011-06-30 11:07:57,536][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=0.0, Evts=116}
[2011-06-30 11:08:57,547][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=0.0, Evts=116}
[2011-06-30 11:09:57,568][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=0.0, Evts=116}
[2011-06-30 11:10:57,586][INFO ][default.com.arcsight.agent.ai][logStatus] {Eps=0.016666666666666666, Evts=117}

As you can see, it's a pretty quiet system.

(I checked the agent.out.wrapper.log for another FC and I don't see any reference to GC there either... maybe I have something [mis]configured somewhere...?)

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.