UPDATE! The community will be go into read-only on April 19, 8am Pacific in preparation for migration on April 21. Read more.
UPDATE! The community will be go into read-only on April 19, 8am Pacific in preparation for migration on April 21.Read more.
Absent Member.
Absent Member.
831 views

Connector Tunning - EPS, Filter, Turbo Mode?

I'm interessted in tunning a lil bit my connectors. I talk especialy about:

  • Checkpoint Firewall
  • Bluecoat Proxy
  • WUC - collecting logs on Domaincontroller

Does anyone have some experience conerning:

  • Setting Filters on Connector Level to filter out not essential events just at this point?
  • Restricting the EPS to not overload ESM during some high peaks?
  • Changing the Turbo Mode?
    • I've read in the AdminGuide of ESM, it is recommended to use "Fastest" Mode on Firewalls. Would this also count for Proxylogs or WUC logs?

I think this is a central topic in every ESM architecture because with every new source we will get some more EPS and so on and on... this will make everyone think about what is really needed what is really helpful to get the most out of the system on one hand and to have a good performance on the other hand.

I would be happy to hear about any recommondations or experience made by everyone.

BR, Silvan

Labels (2)
0 Likes
3 Replies
Absent Member.. Absent Member..
Absent Member..

There are a variety of options depending on your requirements and the capability of the connector in question.

Syslog connectors can benefit from Multi-Threading configurations

Various connectors that get high load can benefit from some additional memory being given to their process in agent.wrapper.properties

Turbo mode is a great idea if you do not need the data fields, it reduces the size of the events being sent

I get a lot of mileage out of the Connector batching settings, default is to send batches of 100 events every 5 seconds. In a high volume connector you can up the batch size to 300 and reduce the time down to 1 second. This can help with overall throughput.

Connector filtering is good in that it prevents events from going into ESM, but it doesn't reduce the connector load at all as the connector has to look at the event do determine if it needs to be sent or not.

These are just some ideas.

Joe Burke posted a good guide to connector tuning on the forum that would be worth a look !

Dean

0 Likes
Commodore
Commodore

Can somebody point me out to that guide please?

Regards.

Blanca Rodriguez
SIEM Engineer
0 Likes
Admiral
Admiral

What Dean said...

Though I'd add:

  • Field Based Aggregation can also reduce the load on your ESM (less effective events flowing to the destination)
  • Limiting Event Processing rate is a handy setting if your nearing the capacity of your ESM DB IO.  This works well for WUCs but is rubbish for other connector types.  That being said - since it works for WUCs I find the setting does alleviate loads at busy moments.  This of course, requires you to understand your typical event flow profiles per WUC instance.
  • Increasing transport thread counts to your destinations will also help with event throughput (assuming your destination can handle it).

Cheers,

Ian.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.