Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: Checkpoint caching

Seen this before and it was disk performance!

The first thing that the connector will do is to receive the data via the command line tool from CheckPoint and spool it straight to disk. Then the connector actually reads the files and processes them. As part of the processing, it will then spool the data to the inbound queue and also spool the processed data to the outbound queue! Yes, it is reading and storing the data that many times.

So if you have slow disk, contended disk or running on a VM without a dedicated disk, it will slow down and hence trigger caching!

Also, have you checked in the log files and seeing what it is doing? are you hitting the memory? It should give you some indication around the caching messages and show you what is going on. Also, have you considered running Logfu on the connector to see what the internal operation is?

0 Likes
Highlighted
Acclaimed Contributor.. Acclaimed Contributor..
Acclaimed Contributor..

Re: Checkpoint caching

How frequently is your agent doing a full garbage collection?

Even after the changes recommended by @pganguly you are still only sending 2K EPS?

0 Likes
Highlighted
Respected Contributor.. Respected Contributor..
Respected Contributor..

Re: Checkpoint caching

Client wants all fields in ESM. Wont this filter out some fields in ESM? and in our case during peak hours sent to manager EPS is less than post aggregation EPS. HP engineer suggested us to keep ESM memory twice of working set of memory and in our case working set of memory if 4 GB.

0 Likes
Highlighted
Respected Contributor.. Respected Contributor..
Respected Contributor..

Re: Checkpoint caching

GC is happening in less than 5 secs. HP Engineer suggested us to keep ESM memory twice of working set of memory and if we decrease ESM memory we getting "Memory in red zone error". Most memory is used while rules are updating Active List and since last 15 or 20 days our sent to manager EPS is always less than post aggregation EPS.

0 Likes
Highlighted
Respected Contributor.. Respected Contributor..
Respected Contributor..

Re: Checkpoint caching

And after adding Symantec protection logs the total EPS is around 12k - 15k and this expected

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: Checkpoint caching

Sorry confused now - are you seeing Memory In Red Zone messages on ESM or the connector? And where is the caching occurring also? At ESM or the connector? I have been talking about the connector, but if you have been hitting performance issues at ESM, thats a completely different discussion!

In the case of ESM, disk performance is critical - so I would be digging into this, and also make sure you have plenty of memory - and you mentioned GC is happening less than 5 second - every 5 secs or its taking 5 secs?

0 Likes
Highlighted
Absent Member.
Absent Member.

Re: Checkpoint caching

"Wont this filter out some fields in ESM?":

if aggregated events have some fields are identical, they will be kept, otherwise dropped. The setting "Preserve Common Fields: Yes" is designed for this purpose. If the client would like to see a source port for example, the aggregation is not for you. But this information is not very useful usually.


"HP engineer suggested us to keep ESM memory twice of working set of memory and in our case working set of memory if 4 GB.":

Really??? Well, I am not a HP Engineer but this advise contradicts with my experience. ESM Manager is pretty greedy for memory. The only downside of the huge RAM allocation - JRE garbage collector will spend more time for global cleanup but it invoked pretty rare,usually 1 time per hour.

0 Likes
Highlighted
Absent Member.
Absent Member.

Re: Checkpoint caching

I did not see you mentioned about invoking global garbage collector at the ESM each 5 seconds. Fix this first and I would not be surprised if the Check Point Connector issue will disappear after it.

0 Likes
Highlighted
Respected Contributor.. Respected Contributor..
Respected Contributor..

Re: Checkpoint caching

yes now am seeing GC happening every in every 5 secs and most syslog based connector are caching. sometime back HP suggested us to decrease ESM Java memory to twice of working set of memory (4 GB working set of memory * 2 = 8 to 12 gb ) but if we do this we get error "memory in red zone". so we using 24 GB now.

0 Likes
Highlighted
Absent Member.
Absent Member.

Re: Checkpoint caching

Increase up to 32Gb and talk to your client to enable aggregation at Connector's level.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.