katzmandu1 Absent Member.
Absent Member.
983 views

Active List Errors in 6.5c?

Jump to solution

All,

I have a fairly busy 6.5 installation where we have a lot of active list traffic. The active lists themselves aren't that big; the one that is giving me fits had less than 100 entries. We do have other lists with 80-100k of entries.

After a fresh start of ESM, the active list will work for 12-24 hours. Once the system gets busy, I'll notice rule actions fail with "AddToList: Failure" My rules are OK and still fire. Previously the rules were reading from and updating this active list. Now, not so much.

Checking server.log I find:

[2014-01-29 09:20:00,006][WARN ][default.com.arcsight.rulesengine.shared.ListManager] Active list not initialized for add [ID 'H12UCaEMBABCAvdAvXXXXXXX==' Name 'NOYB']

The contents of the active list, all 100-300 entries, are gone. The list is set to have the "default" 10k entries and we expect it to remain around 5k over long-term production use.

If I go and change an attribute on the list (say, expiry from 7 days to 14 days) the list gets "re-initialized." The rule fires work, once again and life goes on for a few more hours, until it breaks again.

I have a Sev1 ticket open on this, but wondering if anyone else has run into this, either in 6.5 or other versions.

Labels (1)
Tags (2)
0 Likes
1 Solution

Accepted Solutions
Honored Contributor.. dhartman Honored Contributor..
Honored Contributor..

Re: Active List Errors in 6.5c?

Jump to solution

FYI, we had similar issues which is officially a bug NGS-8682 which is documented in the ESM 6.5cSP1 release notes under the Open Issues section, there was a thread about this along with other issues https://protect724.hp.com/message/43665#43665 but that thread doesn't seem to be working atm.

In some instances, when an ActiveList is modified at a high rate, the ActiveList

cache can become inconsistent with the underlying database table, with the table

row count exceeding the configured list capacity.  As a result  of this

inconsistency, updates to entries not found in the cache are sent to the DB as

INSERT operations, resulting in a CONSTRAINT VIOLATION exception due to the

entry being present in the DB table.  In addition, multiple ActiveList tables are

updated in the same DB transaction, causing the exception in one  ActiveList to

roll back updates that had previously been made in other ActiveLists.

View solution in original post

0 Likes
6 Replies
Established Member.. Jorge
Established Member..

Re: Active List Errors in 6.5c?

Jump to solution

Hi Jonathan,

We are having an issue where the audit actually says the entry was added to an active list but when you check the AL the entry is not there. However if we restart ESM then everything works for a random amount of time until it fails again and we can restart ESM or clear the active list to get it working again for 5-30mins.

Did support ever find a fix for your issue?

More details about what we are seeing:

0 Likes
katzmandu1 Absent Member.
Absent Member.

Re: Active List Errors in 6.5c?

Jump to solution

Here's a big update, since I forgot I posted this.

1) From what I'm told there are at least a few customers who have this issue.

2) The issue surrounds any kind of manipulation of an active list, reading values, writing values, etc. This has caused content which relies upon active lists to fail, and my customer to miss important security events.

3) For a while Support was having my customer restart the system ever 12 hours.

4) Eventually Support provided some changes to my.cnf which may have solved the issue.

0 Likes
Highlighted
ronaldo Absent Member.
Absent Member.

Re: Active List Errors in 6.5c?

Jump to solution

Jonathan

I hope this issue is solved ... keep us posted please ...

Would you be so kind to share the my.cnf changes that were proposed ?

Thanks a lot

Ronny

0 Likes
SCipriano Absent Member.
Absent Member.

Re: Active List Errors in 6.5c?

Jump to solution

Hi Jonathan.

Let me just share some thoughts:

  • Since the AL is highly used, had you tried to use Session Lists instead? I'm just saying that since SL are more oriented to be used in Memory. It's just a workaround try.
  • I had a similar issue with previous ESM versions but, on my case, it was trend related. Nevertheless the issue was quite similar as yours. Our problem was that ArcSight had database schema issues and the trend crashed when some events were detected. In particular the issue was IPv4 and IPv6 related. The ESM created a table trend with improper column types. Since IPv6 was quite larger than IPv4 values... the hole thing just crashed and it became unavailable.
0 Likes
Established Member.. Christoph
Established Member..

Re: Active List Errors in 6.5c?

Jump to solution

Hi Jonathan,

could you please post the changes that solved this issue? Support doesn´t work nearly fast enough 😞

Thanks, Christoph

0 Likes
Honored Contributor.. dhartman Honored Contributor..
Honored Contributor..

Re: Active List Errors in 6.5c?

Jump to solution

FYI, we had similar issues which is officially a bug NGS-8682 which is documented in the ESM 6.5cSP1 release notes under the Open Issues section, there was a thread about this along with other issues https://protect724.hp.com/message/43665#43665 but that thread doesn't seem to be working atm.

In some instances, when an ActiveList is modified at a high rate, the ActiveList

cache can become inconsistent with the underlying database table, with the table

row count exceeding the configured list capacity.  As a result  of this

inconsistency, updates to entries not found in the cache are sent to the DB as

INSERT operations, resulting in a CONSTRAINT VIOLATION exception due to the

entry being present in the DB table.  In addition, multiple ActiveList tables are

updated in the same DB transaction, causing the exception in one  ActiveList to

roll back updates that had previously been made in other ActiveLists.

View solution in original post

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.