Lieutenant
Lieutenant
3789 views

ESM Archive Storage Management

I am setting up ESM archives. I do not see any archive partition in my archive page, see attached.

Is my understanding correct:

- I should setup a separate Storage Group for ESM archive

- But I do not have enough disk space for the additional Archive Storage Group

Or do I simply turn on the Archiving and check "Follow Schedule" on the Default Storage Group" to achieve the automatic archiving of event data and resource configuration?

I already do have weekly backup of the database.

ESMDB/bin/arcsight export_system_tables $ESMDBUSER $ESMDBPW $ESMDBTNS

Thanks

Jan

Labels (1)
Tags (1)
0 Likes
8 Replies
Commander
Commander

When archiving is enabled in the ACC storage window the day's event will be saved there along with annotations if used. As shown  in your screen shot the default location is /opt/arcsight/logger/data/archives - this directory is part of the OS.  The storage screen reflects choices made during installation as is part of the default storage group so you should not need to create another storage group.  Once enabled, unless there are other issues, you should have your first arvhice tomorrow morning.

One cautionary note, while automatically saved, are not removed automatically and have to be menaged from the command line. Even though the archive will be off line after the retention period, the directories remain in /opt/arcsight/logger/data/archives and have to be removed via command line.  There are a couple good discussions regarding default storage and will provide links once I can verify.

Regards,

Keith Persons

HPE ArcSight ESM Support

0 Likes
Lieutenant
Lieutenant

Great answer, Keith.

Please share with me, once available, link for CLI-based archive management procedures.

Thank you

Jan

0 Likes
Commander
Commander

Jan,

Glad you found the info so far helpful storage and archives can be one challenging areas partly due to changing perspectives between inside the manager and then from the OS level, trying to stay brief here ;-).   There is one older discussion from a few years ago that has an example bash script run from cron -- it is relevant since it is the logger service governing archives.  Another that is good in general is   from a couple years ago, there's a brief section on Storage and Archives and even htough it is 6.0c the concepts are still valid. If I find anything else relevant today I will post the link, please let us know if this has proven helpful.

Regards,

Keith Persons

HPE ArcSight ESM Support

0 Likes
Commander
Commander

Keith,

Is there a way to increase the default storage size? The PS before me setup storage at 2500 MgB, and I am at 90%. I have manually removed much of last year's data, but have very little room to bring in data when I need to do historical searches.

Thanks!

Steve Cook

0 Likes
Lieutenant
Lieutenant

The default storage group management procedure is as follows:

The out-of-the-box event archive is at:

/opt/arcsight/logger/data/archives

The maximum size of the archive is in:

/opt/arcsight/logger/userdata/logger/user/logger/logger.properties

logger.archive.space.allocated-in-gb=valuehere

The current value of the archive can be found:

du -hs /opt/arcsight/logger/data/archives

As said above, the event archive needs to be cleaned up manually before it reaches its capacity.

Cadet 1st Class Cadet 1st Class
Cadet 1st Class

Need a little more information please... I'm not an expert but I can help or find someone that knows more than I do...

Question: Are you looking for warm storage (able to be used from the system on demand)?

OR are you looking for 'cold' storage (data that has to be manually loaded) into the system after it ages off? EG after 90 days but you want to make sure the data is still present.

If it's the first one, the partition that you're pointing the application at has to be large enough to hold the archived data or you have to point at a partition that is large enough on another defined partition. To do that you have to identify a partition in the OS that the application can access and has permissions to write data to. That partition needs to be identified first in the OS then via the application like your example EG /dev/d0b/blah/blah/blah (disk 02 or whatever you named the the larger partition/array of disks). That partition must have the capacity to store your data for the retention period. If it does not, you're going to run out of space. When that happens, both the OS and the application are going to start barking in the logs, (if they're configured to do so).

If you have enough space on another partition that you can point the system at/to as a storage location that the db and the application can write to (hot storage) point and shoot. If you do not, you'll need to find either a bigger disk/array to put the archives on, or, you'll need to mount a logical disk in read/write mode. Make sure that whatever you point at is RAID configured to give you the speed and redundancy you need AND that, that partition is backed up by whatever process you use to safeguard it from catastrophy/bad events.

If this isn't what you're looking for let me know and I'll do a better job of explaining.

0 Likes
Cadet 1st Class Cadet 1st Class
Cadet 1st Class

Here's what I use in one of our environments in the cron to clean up X days to clean out the older log files.

 

As root: (Edit the current cron tab entry)

#crontab -e

0 01 * * * find /opt/arcsight/logger/data/archives/* -ctime +X -exec rm -fr {} \;

No warranty expressed or implied (use this at your own risk)

What does this do?

At 0100 every day, run a 'find' command under the /opt/arcsight/logger/data/archives/* directory and look for files and directories with a create time of greater than X days from current day (E.G. -ctime +90) [This time expressed in DAYS can be adjusted based on your retention policy}. The process finds and removes (rm) force (no opportunity to say no) files and directories (recursively) under the find parameters.

0 Likes
Cadet 1st Class Cadet 1st Class
Cadet 1st Class

In the -ctime (create time) +X (X= the number of days EG +90 for 90 days OR -ctime +90 )
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.