ismael2251 Respected Contributor.
Respected Contributor.
601 views

Load values Max (EPS, RAM, CPU) for SmartConnectors

Hello,

I have two preocupations :

- Is someone know what is the Load max values for EPS, RAM, CPU on a SmartConnector ?
or it will be very helpfull if i can get documentation on it.

- i need to build a daily report regarding the Min / Max  values of EPS, Ram, CPU  of each SmartConnectors in my ArcSight environment. 

Thanks

BR/

 

0 Likes
8 Replies
MajorW_MF Trusted Contributor.
Trusted Contributor.

Re: Load values Max (EPS, RAM, CPU) for SmartConnectors

Just to clairfy, you are looking for what system resources a SmartConnector is using?

0 Likes
ismael2251 Respected Contributor.
Respected Contributor.

Re: Load values Max (EPS, RAM, CPU) for SmartConnectors

Hi Major,

Im looking for :

- EPS Max value a Smartconnector can support.

- Is there on a Smartconnector a matrix between memory / Cpu  load with EPS ? if yes i need to know .

 

 

0 Likes
MajorW_MF Trusted Contributor.
Trusted Contributor.

Re: Load values Max (EPS, RAM, CPU) for SmartConnectors

I am sure someone that is in expert in connectors, can provide a much more detailed discussion for you.

I have to join a meeting right now. But, I am attaching a rather old but highly relevant document that is a bit more educational on performance tuning connectors. 

0 Likes
ismael2251 Respected Contributor.
Respected Contributor.

Re: Load values Max (EPS, RAM, CPU) for SmartConnectors

Thank you.
Your document is very helpfull.
But We haven't the EPS limitation

0 Likes
mschleich Acclaimed Contributor.
Acclaimed Contributor.

Re: Load values Max (EPS, RAM, CPU) for SmartConnectors

Hi Ismael,

 

What you have to take into account is that it depends on different criteria like:

  • connector type (syslog, winc, fileReader, etc..)
  • parser (key value, regex, jdbc, etc..)
  • use of map files and their complexity
  • log format (cef, syslog, json, xml, etc...)
  • JVM RAM
  • CPU threads (enable multi-threading)
  • log content (firewall, oracle, windows, custom webapp, etc...)
  • use of features (aggregation, filtering, fields obfuscation, splitting, merge, etc...)

Based on those criteria, a connector could work easily at 8000 EPS, it is possible but it was just to parse firewall logs from which there are just IP addresses and Ports.

Thus, I use a global criteria which has permitted to configure 100 different SmartConnectors properly.
Consider that a SmartConnector properly configured can work until 1000 EPS in a normal behavior.

You will size JVM RAM, parser and HTTP threads following your environment but always starts by using default values recommended by Micro Focus for each SmartConnector Type.

I insist on the normal behavior because by example, for parsing CEF proxy Events, the max EPS was 650.
Why only that value? Because I am using a very complex map files to extract Domain based on TLD.

Thus I have used 9 CEF Syslog SmartConnectors configured after a Load-balancer to process 120 million of logs with peak at 6000 EPS.

 

The max EPS of a SmartConnector is not the most important factor, you need to know the average EPS it can maintain. This is more critical for the sizing of your logs collection infra and it is close to 1000 EPS.

If you give me more info about which smartconnector type you need to use and for which logs sources, I could answer you more precisely.

 

Thanks
Kind Regards

Michael

Highlighted
ismael2251 Respected Contributor.
Respected Contributor.

Re: Load values Max (EPS, RAM, CPU) for SmartConnectors

Hello  Mickael,

Thank you very much for your inputs. they are very helpfull

I use differents smartconnectors type. The main are obviously :
Syslog ( with almost 20 subparsers(regex) ), Snare , Flex ( Regex), Checkpoint ...

 

Ismael,

0 Likes
mschleich Acclaimed Contributor.
Acclaimed Contributor.

Re: Load values Max (EPS, RAM, CPU) for SmartConnectors

Hi Ismael,

 

For CheckPoint, you could easily reach 3000 EPS  with 1 connector then I suppose you use aggregation thus EPS will decrease.

For Snare agent you can also reach 1000 EPS with 1 connector.

Now regarding your syslog SmartConnector I am not sure to understand you have 20 subparsers with the same SmartConnector? I hope not because this is not very good and secondly it should be very difficult to filter and categorize events and also to fine-tune the smartConnector.

It will be the flex SmartConnector from which it will be more difficult to reach 1000 EPS, it depends of the complexity of the regex but also if there are many different event or if it always the same event ID.
But you can reach the 500 EPS if you use multithreading.

Sincerely, if you consider 1000 EPS as a relatively good value by SmartConnector it will be not too far from the reality. You have to take into account that it will be better to have less SmartConnector as possible.
Now, sometimes, for different reasons like caching, parsing, filtering, aggregation and/or mapping or even for building custom ArcSight content you need to use dedicated SmartConnectors and I recommend to do it but only in that case when it is necessary.

Less SmartConnectors are less threads use on ESM.
When ArcSight SmartConnectors are properly configured and fine-tuned, no event are lost thus less SmartConnector to monitor is easier to manage and finally it is less chance to forget or to miss a connector down or even worse a device quiet.
For syslog SmartConnectors which are only receiver (push type), if the connector is down for few minutes, the events are lost thus you need to be able to react immediately when this happens thus it is better to have less such SmartConnectors.

Just to give you an idea, we have setup 95 SmartConnectors for 45 Logs Type which represents 1.6 billion of events per day (Loggers Pool Destination) or 400 million of events (ESM Destination)

Use filtering as much as you can to reduce the evnts for ESM in keeping in mind that you filter only what you know  it is legitimate and useless for correlation or what is noisy and also do not bring additionnal info.
By example, I have filtered logoff events for ESM (4634).

But for Proxy, Reverse proxy, SSLVPN, IDS I do not filter anything for ESM destination.
For System logs, I filter unparsed events or when deviceVendor OR deviceProduct IS NULL.

After for the sizing of Connector Host, I don't know if you use or will use VM which is the best as you can easily extend resources.
Start with the minimum recommended and then increase progressively.
If you have any question or if you need help on this, do not hesitate to contact me.

I don't know if have enough answered to your question or if you still need more details.
Just ask me what is missing.

Thanks
Kind regards

Michael

0 Likes
ismael2251 Respected Contributor.
Respected Contributor.

Re: Load values Max (EPS, RAM, CPU) for SmartConnectors

Hello Michael,

Thank you very much. i really appreciate your details .

i'll try with all your suggestions. And i'll revert to you if needed.

 Ismael

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.