ESM 6 and Connector Limit?
Has anyone heard of a limitation to an overall number of SmartConnectors feeding to a single ESM 6.* instance? Supposedly best practices dictate it should be 40 or less due to 'thread exhaustion'. Can anyone confirm or deny this? We have customer environments with 5.5 running over a 100 Connectors, but no 6.*.
Also, if you've seen a 6.* environment running with more than 50 unique Connectors reporting to it, please leave a comment.
Hey Anton -
Yeah, I have had ESM v6.8 up to 50k EPS with a lot of connectors. I would have to agree with the statements above that the only real worry/consideration I have found was threads for the connectors as thread wise, I have had well over 700. Here are some notes about adjusting threads ad what they mean -
If there is an issue with threads you will see something like this in the logs (below in red). Further you will see connectors die off as they will be rejected due to thread exhaustion.
WARNING: '17' agent requests REJECTED because the limit of '512' agent threads was exceeded.
“servletcontainer.jetty311.threadpool.maximum=2048” is the limit for The “ActiveThreadCount” ( value found in the ESM System Information Dashboard)
“agents.threads.max=512” represents how many threads the ArcSight Daemon can handle. This can be found going into your Web UI.
h t t p s : / / <ESM URL>:8443/arcsight/web/login.jsp?origPage=%2Farcsight%2Fweb%2Findex.jsp
From there “System Management” > “Threading” (under java.lang) , look for the value presented for “DaemonThreadCount” .
As a guidance, here is a formula to add more agent threads:
When increasing the agent threads parameter you should allow for 2 connections per Connector.
For example: If you add additional 13 more connectors, then set the below properties:
After you increase the agents.threads.max property as below, please check your CPU utilization to make sure that it is still within acceptable levels. If it does, then you can add even more.
HP support has provided us a similar formula except they say there are 3 connections per connector. Another factor to take into consideration as well is if the Smart Connector(s) have multithreaded turned on and a value set , this will have a tremendous impact in the formula provided. the 3 connections represent -
- Internal events
- System logging
- The logs that content is written for
Using the formula you provided, I come up short. When I change the multiplier to "3" the calculation is spot on knowing also my threading option. Depending on the load of the connector with multithreading turned on, we can expect the value to go up and down in the ESM dashboard. The jetty311 value as well.
agents.threads.max=XX (64+13x3)+N (for starters 231 plus what ever amount of threads are given to the multi-threading on the connector)
As noted in other threads - "A common practice is to exactly double the "servletcontainer.jetty311.threadpool.maximum" as compared to agents.threads.max". I have found this recommendation to be from HP as well.
For your second formula... the "+26" is representing what ? That would be helpful since you are explaining.
I wish HP would update their knowledge base as it is terribly out of date. For dealing with the thread pool being exhausted, they still just tell the customer to send the logs. Details like these other engineers like to know so Protect 24/7 can be self serve.
Hope this helps.
To be more precious, the below is a formula that has been approved and recommended by Support ESM T3 and our FT recently as the best practice for an agent configuration:
- agents.threads.max= (a total of xxx agent x 2) + 30% more
- ervletcontainer.jetty311.threadpool.maximum= agent.threads.max x 2
Seeing some message like "RNING: '17' agent requests REJECTED because the limit of '512' agent threads was exceeded.", is a clear indicator that agent thread queue is full. However, it can be trigged due to over-size on connectors, or other thread process queues in manager are blocked. That's why Support TSE requests to collect manager thread dump and logs to see exact blockage. Most of times, this message is disappeared after fixing some issues on the manager's side without touch the agent configuration.
Well, I 100% agree with you regarding the current situation of KB articles.
Here's an updated post about our environment.
Currently we have 6.8c Patch 2 and 140 Unique connectors on ESM, all of them are reporting and sending events.
System performance is very stable with the following parameters:
I absolutely agree as we have a busy environment and so we have set such parameters on the connector side in the agent.properties as -
We find this very useful to get data to the ESM quickly.
Just to bump this, I've seen data monitors cause blockage and trigger this same alert. So while it may be agents, do not get stuck on it being an agent issue completely.
I have seen multiple ESM 6 environments with over a 100 connectors and a couple with over 200 hundred registered connectors. I have never found the number of connectors to be an issue with 6.X. Usually, it is high EPS with poorly written content or High EPS with a lot of content that cause the most issues.
Agree, I have an environment running with 600 registered connectors to a single ESM without issues.
Most of the problems come from poorly written stock content (rules). Remove the stock content and write your own to fit the environment to avoid performance issues under heavy load.