Checkpoint caching

[sumsama1]

Hi, I have one checkpoint Firewall which caches most of the time. I have made the batching to 300:1 and increased the java heap to 1 Gb but still same. what else can I do to decrease the caching on connector

[pbrettle]

Seen this before and it was disk performance!

The first thing that the connector will do is to receive the data via the command line tool from CheckPoint and spool it straight to disk. Then the connector actually reads the files and processes them. As part of the processing, it will then spool the data to the inbound queue and also spool the processed data to the outbound queue! Yes, it is reading and storing the data that many times.

So if you have slow disk, contended disk or running on a VM without a dedicated disk, it will slow down and hence trigger caching!

Also, have you checked in the log files and seeing what it is doing? are you hitting the memory? It should give you some indication around the caching messages and show you what is going on. Also, have you considered running Logfu on the connector to see what the internal operation is?

[Shaun]

How frequently is your agent doing a full garbage collection?

Even after the changes recommended by @pganguly you are still only sending 2K EPS?

[sumsama1]

Client wants all fields in ESM. Wont this filter out some fields in ESM? and in our case during peak hours sent to manager EPS is less than post aggregation EPS. HP engineer suggested us to keep ESM memory twice of working set of memory and in our case working set of memory if 4 GB.

[sumsama1]

GC is happening in less than 5 secs. HP Engineer suggested us to keep ESM memory twice of working set of memory and if we decrease ESM memory we getting "Memory in red zone error". Most memory is used while rules are updating Active List and since last 15 or 20 days our sent to manager EPS is always less than post aggregation EPS.

[sumsama1]

And after adding Symantec protection logs the total EPS is around 12k - 15k and this expected

[pbrettle]

Sorry confused now - are you seeing Memory In Red Zone messages on ESM or the connector? And where is the caching occurring also? At ESM or the connector? I have been talking about the connector, but if you have been hitting performance issues at ESM, thats a completely different discussion!

In the case of ESM, disk performance is critical - so I would be digging into this, and also make sure you have plenty of memory - and you mentioned GC is happening less than 5 second - every 5 secs or its taking 5 secs?

[AlexMuratov1]

"Wont this filter out some fields in ESM?":

if aggregated events have some fields are identical, they will be kept, otherwise dropped. The setting "Preserve Common Fields: Yes" is designed for this purpose. If the client would like to see a source port for example, the aggregation is not for you. But this information is not very useful usually.

"HP engineer suggested us to keep ESM memory twice of working set of memory and in our case working set of memory if 4 GB.":

Really??? Well, I am not a HP Engineer but this advise contradicts with my experience. ESM Manager is pretty greedy for memory. The only downside of the huge RAM allocation - JRE garbage collector will spend more time for global cleanup but it invoked pretty rare,usually 1 time per hour.

[AlexMuratov1]

I did not see you mentioned about invoking global garbage collector at the ESM each 5 seconds. Fix this first and I would not be surprised if the Check Point Connector issue will disappear after it.

[sumsama1]

yes now am seeing GC happening every in every 5 secs and most syslog based connector are caching. sometime back HP suggested us to decrease ESM Java memory to twice of working set of memory (4 GB working set of memory * 2 = 8 to 12 gb ) but if we do this we get error "memory in red zone". so we using 24 GB now.

[AlexMuratov1]

Increase up to 32Gb and talk to your client to enable aggregation at Connector's level.