Connector Tunning - EPS, Filter, Turbo Mode?
I'm interessted in tunning a lil bit my connectors. I talk especialy about:
- Checkpoint Firewall
- Bluecoat Proxy
- WUC - collecting logs on Domaincontroller
Does anyone have some experience conerning:
- Setting Filters on Connector Level to filter out not essential events just at this point?
- Restricting the EPS to not overload ESM during some high peaks?
- Changing the Turbo Mode?
- I've read in the AdminGuide of ESM, it is recommended to use "Fastest" Mode on Firewalls. Would this also count for Proxylogs or WUC logs?
I think this is a central topic in every ESM architecture because with every new source we will get some more EPS and so on and on... this will make everyone think about what is really needed what is really helpful to get the most out of the system on one hand and to have a good performance on the other hand.
I would be happy to hear about any recommondations or experience made by everyone.
There are a variety of options depending on your requirements and the capability of the connector in question.
Syslog connectors can benefit from Multi-Threading configurations
Various connectors that get high load can benefit from some additional memory being given to their process in agent.wrapper.properties
Turbo mode is a great idea if you do not need the data fields, it reduces the size of the events being sent
I get a lot of mileage out of the Connector batching settings, default is to send batches of 100 events every 5 seconds. In a high volume connector you can up the batch size to 300 and reduce the time down to 1 second. This can help with overall throughput.
Connector filtering is good in that it prevents events from going into ESM, but it doesn't reduce the connector load at all as the connector has to look at the event do determine if it needs to be sent or not.
These are just some ideas.
Joe Burke posted a good guide to connector tuning on the forum that would be worth a look !
What Dean said...
Though I'd add:
- Field Based Aggregation can also reduce the load on your ESM (less effective events flowing to the destination)
- Limiting Event Processing rate is a handy setting if your nearing the capacity of your ESM DB IO. This works well for WUCs but is rubbish for other connector types. That being said - since it works for WUCs I find the setting does alleviate loads at busy moments. This of course, requires you to understand your typical event flow profiles per WUC instance.
- Increasing transport thread counts to your destinations will also help with event throughput (assuming your destination can handle it).