ALERT! The community will be read-only on April 19, 8am Pacific as the migration begins. Read more for important details.
ALERT! The community will be read-only on April 19, 8am Pacific as the migration begins.Read more for important details.
Absent Member.
Absent Member.
1188 views

Annotations not purging, growing system table size CORRE

I have a slowly growing system table size due to annotations not being cleared out from the arc_event_annotation table. Is anyone noticing something similar?

mysql tablesize query

use arcsight;

SELECT concat(table_schema,'.',table_name) as Database_Tablename, table_rows as Rows, concat(round(data_length/(1024*1024),2),'M') DATA, concat(round(index_length/(1024*1024),2),'M') idx, concat(round((data_length+index_length)/(1024*1024),2),'M') total_size, round(index_length/data_length,2) idxfrac FROM information_schema.TABLES where ENGINE='innodb';

I currently have about 122 days of annotations stored.

Labels (1)
0 Likes
3 Replies
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

+1, we have entries in the annotation table dated back to Jan 1st (based on manager_receipt_time).

These events have definitely been purged out from the database through space-based retention, and does not appear to be be preserved in another resource, like a case:

mysql> select event_id, end_time, manager_receipt_time from arcsight.arc_event_annotation order by manager_receipt_time asc limit 1;

+--------------+-------------------------+-------------------------+

| event_id    | end_time                | manager_receipt_time    |

+--------------+-------------------------+-------------------------+

| 114229000597 | 2014-01-01 16:44:01.714 | 2014-01-01 16:44:01.714 |

+--------------+-------------------------+-------------------------+

1 row in set (0.00 sec)

mysql> select * from arcsight.arc_event_annotation_p where event_id='114229000597';                                              

Empty set (0.00 sec)

I would think that this is affecting query times associated with annotation conditions. e.g. the SuperConnector is executing queries that includes a join with this table (on-demand in ESM 6.5) in order to identify correlated base events.

0 Likes
Absent Member.
Absent Member.

David,

I appreciate the response. I have a case open with support to try and understand what is going on. I think something that did not help matters is an analyst using a mark similar annotation condition that really went to town on a lot of events. Curiosity has me now wondering if I could turn off mark similar functionality. Not something I've looked into before.

I am sure that the annotation table being larger is going to impact the retrieval time of queries. That is a good point I did not think about.

0 Likes
Vice Admiral
Vice Admiral

Unknown this question was ever solved. We encountered the same issue with our Dev instance however Production was fine. After some helpful queries from Protect and comparisons between instances, we determined that it is related to the archival process for the storage groups (could be a bug? - we're still on ESM 6.51)

Since our Dev environment didn't need to have it's events archived, we had Archival "ON", but "Follow Schedule" for the storage groups UNCHECKED. We enabled the storage groups to follow the archival schedule, and retroactively archived the scheduled jobs up till the current day. Once the nightly purge job ran, the annotations table began purging up till the appropriate retention point for the storage group(s)... It took approximately 8hrs to purge a years worth of data from the annotation table.

Hope this helps anyone else who has run into the same issue.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.