sagard1 Trusted Contributor.
Trusted Contributor.
2919 views

How to eliminate connector caching issue ?

Please suggest how to eliminate the connector caching issue.

We have got 2 scenario:

1. Software connector reporting to ESM.

2. Logger (appliance) acting as a connector reporting to ESM.

Labels (4)
0 Likes
6 Replies
man-wai.luk@hpe Absent Member.
Absent Member.

Re: How to eliminate connector caching issue ?

Hi Sagar,

This is a very board question and please understand that when the connector start caching, it indicates some of your system / application components had reached it's limitation.

In short, please ensure the followings in order to avoid the "connector caching"

1. Ensure there is sufficient bandwidth between  Connectoers / ESM or Loggers / ESM

2. Ensure your ESM is running in good shape with sufficient spare capacity (i.e. sufficient CPU / Memory / Storage I/O / Network bandwidth )

3. You have allocated "ONE Physical CPU core" per connector instance

4. The BIOS setting of "CPU hyper-threading" on both the ESM and connector servers are "OFF" .

5. If a particluar connector is overload, install an additional connector and redistribute the events loading amongst all the connectors.

6. Allocate sufficient memory for the Smart/Flexconnector, i.e.

     - Syslog Connector   >= 2GB

     - Windows Connector >= 1 GB

     - Other types of Connector >= 512MBytes

7. Perform additional tuning on Syslog Connectors

8. Review the quality of the Flexconnector parser etc.

Hope the above helps.

Regards

Leo Luk, HPE PS

0 Likes
sagard1 Trusted Contributor.
Trusted Contributor.

Re: How to eliminate connector caching issue ?

Thanks Leo Luk. I will perform the mentioned steps and check if our issue is resolved.

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: How to eliminate connector caching issue ?

Hi Sagar,

The caching of events at any connector is not necessarily a problem unless the cache is perpetually growing or is very large to the point that events eventually arriving at ESM are too old to be useful for normal SIEM purposes.

If you have caching at connectors that appears to be abormal, then this can often be attributed (but not limited to) one of the following:

  • Performance issue with the connector
  • Performance issue with ESM
  • A problem with the transport between the connector and ESM

The possible discussion about these is potentially enormous, but there are some resources that can help you examine your environment to work out where a problem may exist.  As you can imagine, the scope of any "performance issue" can include various aspects, that could lie in the hardware (insufficient or malfunction), or the workload (such as abnormally high EPS or EPS too large for the configured environment), or indeed the software (connector and ESM). For example, within ESM itself, the content itself can lead to performance problems if misconfigured, and the more basic issues of selecting suitable heap sizes for the manager JVM can be implicated.

One great place to start is here:

The above page brings together several documents that helps you to examine your overall ArcSight Environment.  It introduces aspects of ESM health monitoring - for example there are data monitors and dashboards which monitor the ESM health and connector health for you.  Keeping an eye on these can be a good starting point to any investigation.

Other questions to consider from the outset might be:

  1. Do all of the connectors reporting to a single ESM instance exhibit a constant caching problem? If all connectors have a problem and their are various connector types (not all syslog connectors for example), then the problem could be with ESM. If only one or two connectors appear to have a caching problem but all others are fine, then the investigation may benefit from starting at the connector side, particularly for high volume connectors such as syslog connectors
  2. Examine the connector logs.  Look for memory problems and any errors which could indicate problems with the network transport from the connector to the ESM.  There is more information about this in the docs from the page linked above. If you need help with these logs files then consider opening a case with HPE support for assistance.
  3. If the problem tends to be with certain connectors, what is similar about them?  Are they all the same type? Are they all on the same network segment? Check the transport from that segment to ESM, is the connection reliable?  Is the TCP RTT acceptable (around 70-80ms).  Issues with TCP RTT will severely affect connector performance when delivering high event rates to ESM.
  4. From the ESM side, use the health dashboards indicated in the above document to get an overall idea of the ESM and connector performance. Do you see there any issues with Database insert times, the performance of which is central to ESM's ability to process events.
  5. If the caching problem is new, consider what has changed in the environment as a whole and particularly within ESM itself. What content could have been added or adjusted/modified? Rules or data monitors with conditions that match a very large set of events can lead to performance problems.  The ESM health dashboards provide information on rules that fire the most often. If you can track down such suspicious content, try temporarily disabling it one-by-one to assess its affect on overall performance.
  6. Consult the ESM logs (<ARCSIGHT_HOME>/logs/default).  The server.log(s) and server.std.log(s) are most relevant.  Check the server.std.log for heap/memory information.  A JVM heap which shows "memory in the red zone" will need adjustment since your system will be under stress trying to free memory thru garbage collection.  Check for ERROR/FATAL messages that are logged frequently and particularly those occurring since the caching problem started.
  7. If you need help examining the log file and/or resolving a performance problem, please open a case with HPE ArcSight support.

I hope that this information is a useful starting point for you Sagar.  There are many discussions in this area on protect724 which you can also benefit from. For example: https://www.protect724.hpe.com/message/59417?commentID=59417#comment-59417 does contain some further links to some great information that will enable you to troubleshoot your environment. 

Thanks and regards,

Darren Hammond
HPE ArcSight Technical Support

ArcSight Support
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
0 Likes
Highlighted
matteo.stettner Respected Contributor.
Respected Contributor.

Re: How to eliminate connector caching issue ?

Hi Leo,

3. You have allocated "ONE Physical CPU core" per connector instance

--> How do you limit this? Yes, you can allocate one CPU for a specific process but how to you hinder other processes of using this CPU as well.

You mention "physical CPU", what is a solution if we talk about connectors installed on a VM?

KR,

Matteo

0 Likes
Super Contributor.. Alexandros Naoum Super Contributor..
Super Contributor..

Re: How to eliminate connector caching issue ?

Hi,

For Windows environment to limit a process can be done via affinity in task manager. It may be little tricky as the main connector process is just a java process.

For Linux the taskset maybe your tool (again the same case with the java processes)

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: How to eliminate connector caching issue ?

Hi Sagar,

Good morning.  I am following up on this question that you posed in yesterday's ArcSight ESM Expert Day, where you were asking for suggestions on how to troubleshoot connector caching issues.

Did you find the information provided by various contributors useful and a good starting point for your investigation?  If so, would you mind marking this thread as answered?

Thanks and regards,

Darren Hammond

HPE ArcSight ESM Technical Support.

ArcSight Support
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.