Super Contributor.. ejsimon Super Contributor..
Super Contributor..
1417 views

ESM 6.5c manager service issues

Jump to solution

Our Primary ESM 6.5c system was running fine until about 2 weeks ago (it was up for about 3
months without issues) is having serious issues.  Performance has taken a serious hit and the
manager service appears to be restarting itself about once an hour.  I’m also seeing “raw chunk size” and “Memory
usage in red zone” messages in the server.std.log.   I’ve changed the java Heap size from 8 GB to 16 GB, restarted the services, and even
restarted the Linux box it’s running on. I do have an open ticket with HP Support, and they had me set the following in the server.propreties file:

queue.logger.pre-security-event-persistor.batchsize=5000
queue.logger.pre-security-event-persistor.threshold=5000
 

    

However, we’re still seeing the raw chunk size” and “Memory
usage in red zone” messages in the server.std.log.  I’ve given HP Support several sets of logs, thread dump logs and Java memory dumps, but haven’t receive a resolution to the issue.  Our HA ESM which is receiving the same events has not experience any of the issues (yet). 

Any help or suggestions is welcomed. 

Thanks,

Eric

Labels (2)
0 Likes
1 Solution

Accepted Solutions
Super Contributor.. ejsimon Super Contributor..
Super Contributor..

Re: ESM 6.5c manager service issues

Jump to solution

We found one of the Prebuilt Rules for “resource access” over ran a session list with too many events.   The session list was
only designed to handle 10,000 rows in memory and it had over 700,000 trying to write to it..  This caused the mysql database to use more memory and made the manager service very unstable.  We had to do a sql command to truncate the table.

View solution in original post

0 Likes
8 Replies
Super Contributor.. ejsimon Super Contributor..
Super Contributor..

Re: ESM 6.5c manager service issues

Jump to solution

We found one of the Prebuilt Rules for “resource access” over ran a session list with too many events.   The session list was
only designed to handle 10,000 rows in memory and it had over 700,000 trying to write to it..  This caused the mysql database to use more memory and made the manager service very unstable.  We had to do a sql command to truncate the table.

View solution in original post

0 Likes
Honored Contributor.. dhartman Honored Contributor..
Honored Contributor..

Re: ESM 6.5c manager service issues

Jump to solution

We just ran into the same problem, but with different symptoms.  the server.std.log files kept showing us going into the red zone, which started causing Full GC every few minutes.  A restart would fix it temporarily but then it started again.  In the server.log file we were getting errors about that Access list Resource Access overflowing, it was evicting the events but it couldnt keep up (Session list had a max memory size of 10,000 and it had 5.2 million entries in it)

-06-23 11:48:30,961][WARN ][default.com.arcsight.common.sessionlist.tuple.SessionTupleMap] SL Resource Access still overflowing after pruning, evicting keys

[2014-06-23 11:48:30,961][WARN ][default.com.arcsight.common.sessionlist.tuple.SessionTupleMap] SL Resource Access after evicting 30 keys, count = 7556

Disabling the rule that populate (and clear) that access list seemed to resolve it.   I don't know what the purpose was of that rule, but it seems like every customer will run into this problem since that rule is written such that it will fire on practically every event.  Thanks for this post ;-D

0 Likes
zarysh
Visitor.

Re: ESM 6.5c manager service issues

Jump to solution

Hi Eric, Would you share the sql command to do the same. Regards, Sareesh

0 Likes
Honored Contributor.. dhartman Honored Contributor..
Honored Contributor..

Re: ESM 6.5c manager service issues

Jump to solution

Thanks to for the following queries:

/opt/arcsight/logger/current/arcsight/bin/mysql -u arcsight --password=PutPasswordHere arcsight

then

Find which ones are over limit

    //Active list query sorted by over limit

        SELECT TABLE_NAME, TABLE_ROWS, arc.capacity, ar.name, CASE WHEN TABLE_ROWS > arc.capacity THEN 'OVER' ELSE 'OK' END AS OVER_CAPACITY FROM INFORMATION_SCHEMA.TABLES alltables INNER JOIN arc_active_list arc ON UPPER(alltables.table_name) = UPPER(CONCAT('arc_ald_', arc.data_table_id)) INNER JOIN arc_resource ar ON arc.id=ar.id ORDER BY OVER_CAPACITY DESC;

        Sorted by size

        SELECT TABLE_NAME, TABLE_ROWS, arc.capacity, ar.name, CASE WHEN TABLE_ROWS > arc.capacity THEN 'OVER' ELSE 'OK' END AS OVER_CAPACITY FROM INFORMATION_SCHEMA.TABLES alltables INNER JOIN arc_active_list arc ON UPPER(alltables.table_name) = UPPER(CONCAT('arc_ald_', arc.data_table_id)) INNER JOIN arc_resource ar ON arc.id=ar.id ORDER BY TABLE_ROWS DESC;

    //Session list Query

        SELECT TABLE_NAME, TABLE_ROWS, arc.in_memory_capacity, ar.name, CASE WHEN TABLE_ROWS > arc.in_memory_capacity THEN 'OVER' ELSE 'OK' END AS OVER_CAPACITY FROM INFORMATION_SCHEMA.TABLES alltables INNER JOIN arc_session_list arc ON UPPER(alltables.table_name) = UPPER(CONCAT('arc_sld_', arc.data_table_id)) INNER JOIN arc_resource ar ON arc.id=ar.id ORDER BY OVER_CAPACITY DESC;

    //Trend query:

        SELECT round(((data_length + index_length) / 1024 / 1024), 2) "Size in MB", table_name, arc_trend.id as "Resource ID", arc_resource.name as "Resource Name", TABLE_ROWS as 'Row Count', REPLACE(REPLACE(ExtractValue(trend_xml, '/Trend/MaxRows'), ' ', ''),'\n','') as 'Max Rows', CASE WHEN TABLE_ROWS > CAST(REPLACE(REPLACE(ExtractValue(trend_xml, '/Trend/MaxRows'), ' ', ''),'\n','') AS UNSIGNED) THEN 'OVER' ELSE 'OK' END as Over_Capacity FROM information_schema.TABLES LEFT JOIN (arc_trend,arc_resource) ON (arc_trend.table_id=UPPER(mid(table_name,11,6)) AND arc_trend.id=arc_resource.id) WHERE table_name like 'arc_trend_%' order by OVER_Capacity desc, round(((data_length + index_length) / 1024 / 1024), 2) desc;

then

If you need to truncate (clear table but do a safe check first to make sure it can be cleared, run

mysql> truncate table arc_trend_p5a7d0;    (Will take 15-30 minutes depending on size)

results show say query ok

thank you should be able to run the sql query again and the table should be empty

0 Likes
zarysh
Visitor.

Re: ESM 6.5c manager service issues

Jump to solution

Thanks Dustin, 🙂

0 Likes
eagolli@tetra.a Absent Member.
Absent Member.

Re: ESM 6.5c manager service issues

Jump to solution

Dear Dustin,

I am facing a problem and I am getting logs very similar to yours.

Could you pls let me know what was the specific rule that you disabled at that time?

thanks,

Elton

0 Likes
eagolli@tetra.a Absent Member.
Absent Member.

Re: ESM 6.5c manager service issues

Jump to solution

Hi,

I found a specific rules category under Arcsight Foundation that is called Resource Access and I disabled those 3 rules.

Looks it did the trick. We are not getting those strange events in the server.log anymore and finally events started to normally show in the Arcsight Console.

I have also opened a case with Arcsight support in order to understand what is wrong with those rules.

Thank you very much Dustin.

EltonResource Access Rules.png

0 Likes
ugur.ertabak Absent Member.
Absent Member.

Re: ESM 6.5c manager service issues

Jump to solution
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.