New Issue with ESM 6.8 Patch2
There is an issue in a new Patch when Active List when new entries in some active lists are not being populated.
Also it seems like purging Engine suddenly stops after a while.
For example today it happened at 05:46 AM, when purging engine suddenly stopped and ALs got overloaded. The last log with "Purged AL" keyword appeared:
[2015-11-20 05:46:02,468][INFO ][default.com.arcsight.common.activelist.DefaultActiveListCache] Purged AL System - Resource ChangeLog removing 0 entries in 116 msec, new size = 390189
After that all the "Purged AL" logs has dissapeared.
Is anyone experiencing this?
UPDATE: Also have a look at screenshots that I've just taken. Looks kinda funny when playing with TTL. Clearly a bug.
Kinda awful, but we've ran into issue as much as 3 times only during last day.
So as I said before, cleaning the AL cache was only a temporary solution.
I have opened a case with HP and am now dealing with everything that goes along with that. We are experiencing the issue daily. I restart of the manager fixes the problem. The lack of activelist purge events in server.log is a marker for this scenario occurring.
I am having the problem as well in my environment.
What is interesting in my case, is that some of the Active lists go above 100 percent all of a sudden, then the thread stops purging those lists only, but purges the other ones, hits after some time an OOM (OutOfMemory), then all threads stop purging and of course eventually dies.
Did you guys see any out of memory exceptions in the server*log* log files for your manager as well or did just all the thread purging stop all of a sudden?
Did you also see situations where the active lists went above 100 percent in the console?
Curious if what I am seeing is similar to what you guys are seeing as well.
I am also working with support on a resolution.
Anyone still having this issue, do you have Activate Framework installed?
After HP Support pointed at Activate Framework as a source of this issue I just want you to know that I have looked at Activate Framework and particularly its System Monitoring pack which has a Rule named "Categorize Asset as Device" (by default it is in /All Rules/Real-time Rules/ArcSight Activate/Solutions/Product Rules/ArcSight System Monitoring/Device/System and Services Changes/Categorize Asset as Device). This rule is Standard and had many (more than 200k hourly) partial matches. After i disabled this rule everything seems fine so far with active lists for 2 weeks already.
Thanks Nikolay, also to share, support seemed to identify a root cause of my issue to potentially the locking scheme on a multimapped active list, causing the list to overflow. As soon as an official patch is released and validated I will share my experience with you