Highlighted
nmbabkin1 Absent Member.
Absent Member.
1294 views

New Issue with ESM 6.8 Patch2

There is an issue in a new Patch when Active List when new entries in some active lists are not being populated.
Also it seems like purging Engine suddenly stops after a while.

For example today it happened at 05:46 AM, when purging engine suddenly stopped and ALs got overloaded. The last log with "Purged AL" keyword appeared:

[2015-11-20 05:46:02,468][INFO ][default.com.arcsight.common.activelist.DefaultActiveListCache] Purged AL System - Resource ChangeLog removing 0 entries in 116 msec, new size = 390189

After that all the "Purged AL" logs has dissapeared.

Is anyone experiencing this?



UPDATE: Also have a look at screenshots that I've just taken. Looks kinda funny when playing with TTL. Clearly a bug.


beforeTTLchanged.jpg

afterTTLchanged.jpg



Labels (2)
0 Likes
17 Replies
nmbabkin1 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

After some analyzis and searching on Protect, I've found that issue is looking exactly like this:

0 Likes
shezaf1 Acclaimed Contributor.
Acclaimed Contributor.

Re: New Issue with ESM 6.8 Patch2

Hi Nikolay,

Since we consider the issue you refer to as solved in 6.8c, it is interesting (and naturally disturbing) that you have identified it in 6.8c P2. It would be valuable if you report the issue to support. Feel free to pass me the ticket number so I can accelerate in the backend.

Thanks

~ Ofer

0 Likes
nmbabkin1 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

Hi Olef, and thanks for the answer!

Service Request ID is 4652879348

Please proceed with this.

0 Likes
nmbabkin1 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

2015-11-23_10-11-45.jpg

OK, here's an update. We also have this type of error everytime we are logging on to a console. I see this for the second time since Patch2 Install. Maybe there is a relation to the bug.

There was an error while logging into core services of type: class java.io.IOException.

Please consult log files to diagnose the issue.

Login will still continue, but some services may be affected.

0 Likes
nmbabkin1 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

Hello!

Any updates? We have to reboot ESM every 2 days to get rid of it.

It is a very awful situation

0 Likes
Honored Contributor.. dhartman Honored Contributor..
Honored Contributor..

Re: New Issue with ESM 6.8 Patch2

We were the ones who reported the issue initially and havent seen it on 6.8cP2.  My guess though is that support\dev is going to tell you the number of entries in your AL cache (in memory) is too high.  The screenshot you had showed a capacity of 5 million entries and the log message was for 390,000 entries which I can tell you that support is going to probably tell you is too big (may not actually be the issue but they may have you try and reduce those first).  Hopefully they can figure it out, otherwise you may have to cronjob a manager restart (we had to do that last time we had similar issues until we could fix it).

0 Likes
nmbabkin1 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

Hey Guys.
I think we were able to resolve this issue with HP Support by rebuilding AL cache:

/etc/init.d/arcsight_services stop manager

mv /opt/arcsight/manager/tmp/tuple/local/classes/ /opt/arcsight/manager/tmp/tuple/local/classes_old

mv /opt/arcsight/manager/tmp/tuple/local/source/ /opt/arcsight/manager/tmp/tuple/local/source_old

/etc/init.d/arcsight_services start manager



Since then we haven't experienced this issue for a week already.
Will have a look further and post additional information later.

0 Likes
Honored Contributor.. Karlo_Luiten Honored Contributor..
Honored Contributor..

Re: New Issue with ESM 6.8 Patch2

I think this is unrelated.

see https://protect724.hp.com/message/62718

0 Likes
blroberts90 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

I'm seeing the same issue.  Rebuilding the AL cache only solved the issue temporarily.

- Brandon

0 Likes
nmbabkin1 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

Hey Guys.
Thats why i don't mark it as a correct answer. Lets wait for HP to solve this completely.

0 Likes
nmbabkin1 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

Unfortunately we just catched this issue again after 2 weeks of stable work.

Will send the new thread dumps to HP Support.

0 Likes
nmbabkin1 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

Hey Guys,

Kinda awful, but we've ran into issue as much as 3 times only during last day.
So as I said before, cleaning the AL cache was only a temporary solution.

0 Likes
blroberts90 Absent Member.
Absent Member.

Re: New Issue with ESM 6.8 Patch2

I have opened a case with HP and am now dealing with everything that goes along with that.  We are experiencing the issue daily.  I restart of the manager fixes the problem.  The lack of activelist purge events in server.log is a marker for this scenario occurring.

- Brandon

0 Likes
david.damo@hpe. Contributor.
Contributor.

Re: New Issue with ESM 6.8 Patch2

Hey Guys,

I am having the problem as well in my environment.

What is interesting in my case, is that some of the Active lists go above 100 percent all of a sudden, then the thread stops purging those lists only, but purges the other ones, hits after some time an OOM (OutOfMemory), then all threads stop purging and of course eventually dies.

Did you guys see any out of memory exceptions in the server*log* log files for your manager as well or did just all the thread purging stop all of a sudden?

Did you also see situations where the active lists went above 100 percent in the console?

Curious if what I am seeing is similar to what you guys are seeing as well.

I am also working with support on a resolution.

Thanks,

David

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.