Allocated space in ESM
I need some help on the allocated space in ESM 6.8. We have configured allocated space to be 1.2 TB, but thye default storage group max.size is only 360 GB , with retention of 30 days.
We have archiving ON
What do these setting mean?
My understanding is as follows, please correct me:
After the default storage group size is exceeded or retention period of 30 days is reached, the event would be stored in the local archive(which is 1200 GB allocated space).
Note that, we do not have SAN configured.
Please help me, if I am understanding this correctly.
If we had turned archiving off then I would have to increase the size of the default storage group to almost 1.2 TB. But, since we have archiving ON, events would be archived locally and use the allocated space of 1.2 TB.
I have struggled a bit as well to understand all the terminology behind ArcSight storage space. From my understanding you have the following:
1. Allocated Size - that is the total space you set aside for your events that are currently in the live (on-line) processing memory. This space is used for your Storage Groups and also for the past events that you bring back into the on-line memory from off-line archives (when you need to do that).
2. Storage Groups - they hold the events that are currently in the on-line memory. They have a maximum storage size and a maximum days interval. When either is reached, the oldest events get flushed away.
3. Off-line Archive - by default, each day's worth of events gets automatically written in the off-line archive at the end of the day. So you don't have to reach the limit of your Storage Group in order to have the events written in the off-line archive, they get written there by default on a daily basis. By default the off-line archive resides in:
The maximum size of your off-line archive can be configured in the following file, editing the mentioned property:
logger.archive.space.allocated-in-gb=35 (this is in GB)
Please take into consideration that what I wrote above is valid for ESM 6.5, but I am almost sure this is the situation with 6.8 as well.
Well, if anyone exceeds the space size on the storage slice... it does not cascade into the next storage group. As far as Archiving set to "ON" what this means is that if there is an archive schedule it will follow that schedule to move the data to its archive slice for off line safe keeping until it is to be discarded. Unless you configured archiving the archive space is probably not being used.
You may want to consider Stefan's comment about archiving. He provided where in the server the archives will be written to.
When you installed your ESM you answered questions such as this below. The ESM installer would have looked at where it was told the storage was and carved it up by default. So given 1,200GB or 1.2TB it probably looked like this-
System Storage Size (GB)[XXXGB]: ESM carves it up automatically
Event Storage Size (GB)[XXXGB]: ESM carves it up automatically
Online Event Archive Size (GB)[XXXGB]: ESM carves it up automatically
Retention Period (Days): default
If you go through the "Command Center" you can adjust archive settings, jobs, and storage amount.
If anyone answers your question mark your question as such so others can spot solutions more easily.
The below information may is helpful. From the ACC User’s Guide:
The Maximum Size of the 'event storage' volume, shown in the center, below the storage groups, is the smaller of:
n The maximum size specified in the ESM license property, “logger.limit.maximum”
n The value is calculated based on disk size and the reserved space (Maximum Size = “Size of /opt/arcsight” x 0.9 – “System Storage” – “Event Archives”)
u The size of the /opt/arcsight partition is controlled by the size of the disk drive.
u You set System Storage Size and Online Event Archive Size when you installed ESM.
Allocated Size refers to the amount of disk space actually set aside for the event storage volume.
(The text that appears if you hover over the question mark next to Allocated Size uses the word “memory.” It should say “disk space.”)
This is the value called Event Storage Size that is set on the CORR-Engine Configuration panel of the Configuration Wizard, during installation.
You can increase this size, but you cannot make it smaller.
Let's me explain with examples. Initially here are the allocations:
- /opt/arcsight size = 1000GB or 1TB
- /opt/arcsight * .9 = 900GB
- Allocated size (event archives) = 300GB
- System size = 200GB
- /opt/arcsight/data/archives = 100GB
Maximum size = 1000GB * 0.9 – 300GB – 100GB -100GB = 400GB
From here you can see that Allocated size (300GB) <= Maximum size (400GB) ----- > everything is OK so far
Then you want to increase the Allocated size to 400GB, then Maximum size = 1000GB * 0.9 –400GB – 200GB -100GB = 200GB
You can now see that Allocated size (400GB) > Maximum size (200GB), which is not allowed.