New Ranks & Badges For The Community!
Notice something different? The ranks and associated badges have gone "Star Fleet". See what they all mean HERE
Highlighted
Absent Member.
Absent Member.
2371 views

ESM 6 and Connector Limit?

Greetings!

Has anyone heard of a limitation to an overall number of SmartConnectors feeding to a single ESM 6.* instance?  Supposedly best practices dictate it should be 40 or less due to 'thread exhaustion'.  Can anyone confirm or deny this?  We have customer environments with 5.5 running over a 100 Connectors, but no 6.*.

Also, if you've seen a 6.* environment running with more than 50 unique Connectors reporting to it, please leave a comment.

Cheers!

A.

Tags (4)
0 Likes
25 Replies
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Hey Anton -

Yeah, I have had ESM v6.8 up to 50k EPS with a lot of connectors.  I would have to agree with the statements above that the only real worry/consideration I have found was threads for the connectors as thread wise, I have had well over 700.  Here are some notes about adjusting threads ad what they mean -

If there is an issue with threads you will see something like this in the logs (below in red).  Further you will see connectors die off as they will be rejected due to thread exhaustion.

WARNING: '17' agent requests REJECTED because the limit of '512' agent threads was exceeded.

servletcontainer.jetty311.threadpool.maximum=2048” is the limit for The “ActiveThreadCount”  ( value found in the ESM System Information Dashboard)

agents.threads.max=512” represents how many threads the ArcSight Daemon can handle. This can be found going into your Web UI.


h t t p s : / / <ESM URL>:8443/arcsight/web/login.jsp?origPage=%2Farcsight%2Fweb%2Findex.jsp


From there “System Management” > “Threading” (under java.lang) , look for the value presented for “DaemonThreadCount” . 

Name

Set MAX

Current usage

servletcontainer.jetty311.threadpool.maximum

2048844

agents.threads.max

1024498

scotty

0 Likes
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

As a guidance, here is a formula to add more agent threads:

When increasing the agent threads parameter you should allow for 2 connections per Connector.

For example: If you add additional 13 more connectors, then set the below properties:

agents.threads.max=90     (64+13x2)

servletcontainer.jetty311.threadpool.maximum=154   (128+26)

After you increase the agents.threads.max property as below, please check your CPU utilization to make sure that it is still within acceptable levels. If it does, then you can add even more.

-Nellie

0 Likes
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Nellie -

HP support has provided us a similar formula except they say there are 3 connections per connector.  Another factor to take into consideration as well is if the Smart Connector(s) have multithreaded turned on and a value set , this will have a tremendous impact in the formula provided. the 3 connections represent -

- Internal events

- System logging

- The logs that content is written for

Using the formula you provided, I come up short.  When I change the multiplier to "3" the calculation is spot on knowing also my threading option.  Depending on the load of the connector with multithreading turned on, we can expect the value to go up and down in the ESM dashboard. The jetty311 value as well.

agents.threads.max=XX   (64+13x3)+N  (for starters 231 plus what ever amount of threads are given to the multi-threading on the connector)

servletcontainer.jetty311.threadpool.maximum=154  (128+26)

As noted in other threads - "A common practice is to exactly double the "servletcontainer.jetty311.threadpool.maximum" as compared to agents.threads.max".  I have found this recommendation to be from HP as well.

For your second formula... the "+26" is representing what ?  That would be helpful since you are explaining.

I wish HP would update their knowledge base as it is terribly out of date.  For dealing with the thread pool being exhausted, they still just tell the customer to send the logs.  Details like these other engineers like to know so Protect 24/7 can be self serve.

Hope this helps.

scotty

0 Likes
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Hi Scotty,

To be more precious, the below is a formula that has been approved and recommended by Support ESM T3 and our FT recently as the best practice for an agent configuration:

  1. agents.threads.max= (a total of xxx agent x 2) + 30% more
  2. ervletcontainer.jetty311.threadpool.maximum= agent.threads.max x 2

Seeing some message like "RNING: '17' agent requests REJECTED because the limit of '512' agent threads was exceeded.", is a clear indicator that agent thread queue is full. However, it can be trigged due to over-size on connectors, or other thread process queues in manager are blocked. That's why Support TSE requests to collect manager thread dump and logs to see exact blockage. Most of times, this message is disappeared after fixing some issues on the manager's side without touch the agent configuration. 

Well, I 100% agree with you regarding the current situation of KB articles.

-Nellie

0 Likes
Highlighted
Absent Member.
Absent Member.

Hello Guys.
Here's an updated post about our environment.

Currently we have 6.8c Patch 2 and 140 Unique connectors on ESM, all of them are reporting and sending events.
System performance is very stable with the following parameters:

server.wrapper.conf:

wrapper.java.initmemory=81920

wrapper.java.maxmemory=81920

server.properties:

servletcontainer.jetty311.threadpool.maximum=4096

agents.threads.max=2048

Cheers.

0 Likes
Highlighted
Absent Member.
Absent Member.

The number of connections per connector depends, . By default, it's just one. You can configure it on your connectors. And only you know what you have configured on your connectors.

0 Likes
Highlighted
Commodore
Commodore

best practice is 64, you'll find that number is a  number of HP KM and documents.  If you want more the best practice would be to relay your connectors.

0 Likes
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Heiko -

I absolutely agree as we have a busy environment and so we have set such parameters on the connector side in the agent.properties as -

http.transport.multithreaded=true

http.transport.threadcount=N

We find this very useful to get data to the ESM quickly.

scotty

0 Likes
Highlighted
Absent Member.
Absent Member.

Just to bump this, I've seen data monitors cause blockage and trigger this same alert. So while it may be agents, do not get stuck on it being an agent issue completely.

0 Likes
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Anton & all -

I gathered up my notes as well as spoke with some of my peers, HP FE's and HP Support and wrote a detailed blog on how the "Connector Limit" works.  It can be found here.

I hope this helps with the subject that seems to be brought up more often than not on Protect 24/7

scotty

0 Likes
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

I have seen multiple ESM 6 environments with over a 100 connectors and a couple with over 200 hundred registered connectors.  I have never found the number of connectors to be an issue with 6.X.  Usually, it is high EPS with poorly written content or High EPS with a lot of content that cause the most issues.

0 Likes
Highlighted

Agree, I have an environment running with 600 registered connectors to a single ESM without issues.

Most of the problems come from poorly written stock content (rules). Remove the stock content and write your own to fit the environment to avoid performance issues under heavy load.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.