Absent Member.
Absent Member.
434 views

Search through large amounts of data unnecessarily - how to speed up searches in ESM?

Hi,

Our customer have a fairly large implementation of ArcSight ESM. They have two different consumer groups within their organisation; one group looks mostly at realtime data, using rules etc. The other group looks more at historical data. They used to have two different ArcSight ESM instances, where different kind of logs was sent to the two ESM instances. Infrastructure logs to one ESM, Application logs to the other ESM. But now after a consolidation they only have one environment that they all use. The first group is satisfied, because they only look at realtime data, and the system can handle that without issues. However the second group, experience the consolidation as somewhat bad since their searches now takes a lot longer time (because they have to search through a lot more data).

This was the setup regarding logs in the old divided ESM environment

Infrastructure logs -> ESM1 (A lot more logs, Group 1 only doing realtime rules etc.)

Application logs -> ESM2 (Less logs, faster searches, Group 2 doing searches in historical data)

Now they put all logs in the same ESM, and they divide the logs in two different Storage Groups -> INFRA logs, APP logs. The ESM version is 6.5 SP1 P1. They cannot go further than 6.5 SP1 P2 because of problems with PKCS#11 on 6.8 and since they run SuSE they cannot upgrade to 6.9.x.

I noticed that in Command Center you can specify Storage Group when you search. That increases search results a lot for Group2, however they need to schedule reports and searches, and in those cases I have not been able to specify Storage Group. They use ArcSight Console for their daily work, is there any field of parameter we can use there that would increase the search speed?

Is there something I can do for this customer? Thanks for all suggestions.

Labels (1)
0 Likes
6 Replies
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Generally, ESM I use as a correlation layer, tired of having different groups killing ESM with poorly built queries, generally, let ESM correlate, have the consumers use logger to investigate. We build the content in logger and they can drill down there.

Environment 40-60k eps in ESM, few hundred k EPS in logger

0 Likes
Absent Member.
Absent Member.

I'm not saying that I don't agree with you in terms on how to use ESM and Logger, but in the end the question "is there anything I can do for this customer" is what matters to me. Perhaps there is nothing I can do for them but ask them to change their ways of working. But since I started looking and noticed how search speeds can be greatly increased in Command Center with lower impact on the environment, I wanted to see if this could be used in the Console as well. I cannot phantom why it wasn't implemented.

Anyway, thanks for input.

0 Likes
Absent Member.
Absent Member.

Hi ​, If my understanding is right, you wanted to increase the performance at the console level too. Did you try increasing the heap size on the console. This should help in this case. And can you share what tweak you have done at the command center?

Kindly let me know the progress.

Thanks,

0 Likes
Absent Member.
Absent Member.

Hi Sujan,

Console level is not a problem for us.

Regards Victor

0 Likes
Absent Member.
Absent Member.

Hi ​,

Then you can try the same at command center right!

It should provide you the fix

Regards,

Sujan

0 Likes
Absent Member.
Absent Member.

I'm afraid it doesn't because Command Center does not have the necessary tools, schedules searches and reports. If those were available then it would be an OK workaround. But I would prefer to be able to specify Storage Group in ArcSight Console instead 🙂

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.