Limiting EPS at Connector Level

[siddarthtalupula1]

Hello,

Is there a way to limit the EPS at the connector level? I am aware of the following options;

A. Aggregation
B. Filter-Out Events

These options are not open for me as I am dealing with Threat Feed logs.

Is there any other option like modifying any parameters in the agent.properties file and control the EPS?

--
Thanks and Regards,
Siddarth T

[mschleich]

Hi,

Yes, there are other settings like batch size or threads but it is not recommended.
There is also the connector bandwidth limit but it is really not a good idea of using this setting due to the side effects.

ArcSight ESM is a real-time SIEM, if you reduce the pipe line, you connector will start to cache and you will have logs with EndTime not synchronized with ManagerReceiptTime.

I don't understand why you need that. If you need to reduce the feeds logs, you have to work at the source.

You have to ask to send logs by batch, like every 10 min, every hour, etc...
You have to check if it is not possible to separate the feeds in using different connectors in place of one.

Sincerely, I don't understand why you have to limit the EPS, what is your main reason?
Could you please explain me why you are looking for limiting EPS, maybe there is another solution for your issue than doing this? I am really curious.

Normally, it is the opposite, we want to increase and when it is not possible we use Loadbalancing.

Thanks
Kind Regards

Michael

[subindbabu]

Yeah, agree with Michel.
Modifying threads , Batching or bandwidth for reducing EPS out will impact your connector processing performance. If you can explain the scenario , we can suggest a better way

[ianfitz · Accepted solution]

As others have said, this is not a typical thing to want to do. If it is essential (I dont understand why you need to do this), you could look at destination-specific settings such as:

• Network / Limit Bandwidth to..
• Processing / Limit Event Processing Rate

These settings often work for certain connector types only, so experiment in a non-critical test environment first so you can work out what works best for you.

Be aware of flow on decisions here... Slow down the feed such that EPS out < EPS in, and you will cache.

Cache too much and you will drop events.

Good luck!

[manojs]

Best way to limit EPS on a connector is by using "Customized Event Filtering" (read more about it in Connector admin guide). 

It  not only saves EPS but will save your ADP license becasue this method will drop events even before they are parsed and thus saving storage as well 🙂

Its very easy to use and very promising. By using this we have reduced our storage consumption by 40% 🙂

Using built-in filtering and aggrgation will not help with ADP licesning. 

[ianfitz]

Hey thats a cool feature, didnt even know it was there.

Its documented in the Smart Connector Users Guide (not the admin guide) but great pickup!