Problems with Logger Searches and Caching

[siemops]

For weeks, we have encountered issues with errors or obtaining timely search results from our Loggers.

We have 2 Logger Search Heads which query 12 Back-end Loggers (which actually store our log data).

Prior to this issue, the only significant changes to our overall ArcSight environment was to upgraded to the latest Connector framework version (7.8.0.8070.0) and latest Connector parser version (7.8.1.8077.0). Things were working fine prior to this.

Our symptoms are, during a Logger search (via one of our Logger Search Heads), we receive messages such as:

"[Local]Peer logger name will be skipped in this search because it is not reachable."

or

"[Local] Error:loggername: Remote exeception (java.net.SocketTimeoutExecption:Read timed out)".

Further, it now takes a significantly- longer period of time to receive search results (even if there are no errors displayed on a Logger Search Head).

Things we have done to try and resolve:

1) Added "httpclient.SoTimeout=90000" to logger.properties (extend the timeout period between back-end Loggers peers and Logger Search Heads)

2) Upgraded to Logger verson 6.6.0.8208.0 (Logger 6.6 plus a hot fix from Support)

3) Modified wrapper.conf on all of our connectors (To rectify caching issues from smart connectors to the Logger Pool destination (from 7.8.0 Release Notes for low EPS to Logger destinations))

None of these changes have helped our Logger search issues or connector caching issues to the Logger Pool destination.

Any thoughts? Anyone else experiencing similar issues?

[Lar]

Try checking the following:

1 .index on the loggers are up to date ?

(from summary screen - top of page:
Global Summary: There are X number of events from 2018/05/16 17:46:01:253 CEST to 2018/06/12 14:04:28:146 CEST - this time should be within a couple of minutes of the time displayed in the right hand corner of the UI. If the index is lagging there will be search performance.

2. Check the mysql log - you should be able to identify the query, as the log should display the query also. From this you can identify if it is doing an indexed search (ROS) / or non-indexed search (WOS). This log should also show the number of events searched and the number of results.

3. Storage groups on each logger are identical?

4. any of the storage groups above 90%? (regardless if you are searching them or not)

5. try running a simple search on one of the search heads - select only one peer at a time to identify which peer is causing the issue

[red8383light]

What problem would occur if a storage group is above 90% or if the server as a whole is above 90%?

[sergenynms]

Hi, 

Has your problem been solved? If so, can you tell me how you solved it?