Search through large amounts of data unnecessarily - how to speed up searches in ESM?
Our customer have a fairly large implementation of ArcSight ESM. They have two different consumer groups within their organisation; one group looks mostly at realtime data, using rules etc. The other group looks more at historical data. They used to have two different ArcSight ESM instances, where different kind of logs was sent to the two ESM instances. Infrastructure logs to one ESM, Application logs to the other ESM. But now after a consolidation they only have one environment that they all use. The first group is satisfied, because they only look at realtime data, and the system can handle that without issues. However the second group, experience the consolidation as somewhat bad since their searches now takes a lot longer time (because they have to search through a lot more data).
This was the setup regarding logs in the old divided ESM environment
Infrastructure logs -> ESM1 (A lot more logs, Group 1 only doing realtime rules etc.)
Application logs -> ESM2 (Less logs, faster searches, Group 2 doing searches in historical data)
Now they put all logs in the same ESM, and they divide the logs in two different Storage Groups -> INFRA logs, APP logs. The ESM version is 6.5 SP1 P1. They cannot go further than 6.5 SP1 P2 because of problems with PKCS#11 on 6.8 and since they run SuSE they cannot upgrade to 6.9.x.
I noticed that in Command Center you can specify Storage Group when you search. That increases search results a lot for Group2, however they need to schedule reports and searches, and in those cases I have not been able to specify Storage Group. They use ArcSight Console for their daily work, is there any field of parameter we can use there that would increase the search speed?
Is there something I can do for this customer? Thanks for all suggestions.
Generally, ESM I use as a correlation layer, tired of having different groups killing ESM with poorly built queries, generally, let ESM correlate, have the consumers use logger to investigate. We build the content in logger and they can drill down there.
Environment 40-60k eps in ESM, few hundred k EPS in logger
I'm not saying that I don't agree with you in terms on how to use ESM and Logger, but in the end the question "is there anything I can do for this customer" is what matters to me. Perhaps there is nothing I can do for them but ask them to change their ways of working. But since I started looking and noticed how search speeds can be greatly increased in Command Center with lower impact on the environment, I wanted to see if this could be used in the Console as well. I cannot phantom why it wasn't implemented.
Anyway, thanks for input.
I'm afraid it doesn't because Command Center does not have the necessary tools, schedules searches and reports. If those were available then it would be an OK workaround. But I would prefer to be able to specify Storage Group in ArcSight Console instead 🙂