Anonymous_User Absent Member.
Absent Member.
187 views

Sentinel 7.1.1.2 low disk space 600+GB correlated events


Hello ,

we are encountering a problem with sentinel. The problem is that the
database is growing whitout control.We have setup a number of days limit
(180) but from the correlated events we have events older than a year:


Code:
--------------------

SIEM=# SELECT nspname || '.' || relname AS "relation",
SIEM-# pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
SIEM-# FROM pg_class C
SIEM-# LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
SIEM-# WHERE nspname NOT IN ('pg_catalog', 'information_schema')
SIEM-# AND C.relkind <> 'i'
SIEM-# AND nspname !~ '^pg_toast'
SIEM-# ORDER BY pg_total_relation_size(C.oid) DESC
SIEM-# LIMIT 20;
] relation | total_size
-----------------------------+------------
public.correlated_events | 649 GB
public.evt_rpt_168301124 | 33 GB
public.evt_rpt_733434329 | 3845 MB
public.evt_rpt_88504318 | 1412 MB
public.evt_rpt_1856022934 | 1045 MB
public.evt_rpt_1585747665 | 156 MB
public.evt_rpt_1980956963 | 129 MB
public.evt_rpt_1592882176 | 82 MB
public.raw_data_files_info | 65 MB
public.evt_rpt_1141457857 | 27 MB
public.license_record_hours | 26 MB
public.ixlog_part | 24 MB
public.evt_rpt_613223902 | 12 MB
public.license_record | 12 MB
public.sentinel_plugin | 10152 kB
public.evt_rpt_1975727754 | 10080 kB
public.partition_sync_info | 8792 kB
public.evt_rpt_1829667305 | 8456 kB
public.evt_datasync_info | 6928 kB
public.evt_rpt_316978432 | 5216 kB
(20 rows)

SIEM=# select * from public.correlated_events order by date_created asc limit 10;
parent_evt_id | child_evt_id | parent_evt_time | child_evt_time | date_created |
date_modified | created_by | modified_by | parent_part_id
--------------------------------------+--------------------------------------+------------------------+------------------------+------------------------+----
--------------------+------------+-------------+----------------
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8a7-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8b1-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8b3-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8b5-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8b7-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8bf-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8c7-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8d1-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8d3-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
1c480be3-e738-1030-8893-0050569b3a6d | 646a3ba7-e738-1030-b8d5-0050569b3a6e | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 2013-08-14 20:57:14+03 | 201
3-08-14 20:57:14+03 | | | -1
(10 rows)

The problem is how to clean up these older events ? It is safe to use the clean_db.sh ,and make some how a query to delete old data ?



--------------------


--
CIURI86
------------------------------------------------------------------------
CIURI86's Profile: https://forums.netiq.com/member.php?userid=5735
View this thread: https://forums.netiq.com/showthread.php?t=51801

0 Likes
2 Replies
Anonymous_User Absent Member.
Absent Member.

Re: Sentinel 7.1.1.2 low disk space 600+GB correlated events


Yes, they do grow....as a result of the report tables listed under "Data
Synchronisation".....

Deleted tupels can also take up space. There is a background process to
automatically process tuples, but doesn't seem to work on all tables....

I've found I've recovered 30+GB of disk by running:

VACUUM (FULL,VERBOSE);
VACUUM (ANALYZE,VERBOSE);

Against the entire SIEM database.


--
ScorpionSting
------------------------------------------------------------------------
ScorpionSting's Profile: https://forums.netiq.com/member.php?userid=469
View this thread: https://forums.netiq.com/showthread.php?t=51801

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Sentinel 7.1.1.2 low disk space 600+GB correlated events


btw, the VACUUM FULL command requires twice the disk space of the table
to process as it copies the data to a new file then replaces the old at
the end.


--
ScorpionSting
------------------------------------------------------------------------
ScorpionSting's Profile: https://forums.netiq.com/member.php?userid=469
View this thread: https://forums.netiq.com/showthread.php?t=51801

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.