Highlighted
ucc_sharooq Absent Member.
Absent Member.
3649 views

average event size

Jump to solution

Hi Experts,

i want to identify the average size of getting collected at Manger. Could you please help.

I just want to know the steps i will follow.

Cheers,

UCC

Labels (1)
1 Solution

Accepted Solutions
bwatson1 Trusted Contributor.
Trusted Contributor.

Re: average event size

Jump to solution

Assuming ESM CORRE, you can run the following SQL to get the average compressed byte size per event. While this wont tell you the real average size of each event, it will tell you how much storage per event you are consuming.

/opt/arcsight/logger/current/arcsight/bin/psql rwdb web (Login to Postgres)

select sum(length)/sum(eventcount) from data.chunk;

\q (to exit Postgres)

0 Likes
17 Replies
andreas.suess Absent Member.
Absent Member.

Re: average event size

Jump to solution

Hi,

this depends really on the devices.

I would say the average raw event size is ca. 500 - 600 bytes.

The CEF event size is then in average 1500 bytes

Andreas

0 Likes
kreed7 Absent Member.
Absent Member.

Re: average event size

Jump to solution

If you are running oracle,

here is an interesting thread to explore.

0 Likes
bwatson1 Trusted Contributor.
Trusted Contributor.

Re: average event size

Jump to solution

Assuming ESM CORRE, you can run the following SQL to get the average compressed byte size per event. While this wont tell you the real average size of each event, it will tell you how much storage per event you are consuming.

/opt/arcsight/logger/current/arcsight/bin/psql rwdb web (Login to Postgres)

select sum(length)/sum(eventcount) from data.chunk;

\q (to exit Postgres)

0 Likes
marioc Frequent Contributor.
Frequent Contributor.

Re: average event size

Jump to solution

Hi, I was told by pre-sales that they usually estimate 40 bytes as the compressed size of events.   Hope this helps.

Regards

Mario

0 Likes
Established Member.. anwarrhce1
Established Member..

Re: average event size

Jump to solution

Hi Brook Watson

So this command shows the avg event size for 24hrs or something else ?

can we expect the output from logger and v4 express to be the same ?

I just saw that the avg event size on logger is around 1 and on express is around 9, why that much difference ? both logger and express are storing cef events, does that means logger does better compression than express ?

I know express might have few more data comparing to logger but still it cannot be 9 times greater.

Thanks.

0 Likes
bwatson1 Trusted Contributor.
Trusted Contributor.

Re: average event size

Jump to solution

Anwar,

There are two factors in play that I would say is affecting the avg event sizes you are seeing...

  1. Express does add more meta data to the events like zones and things like that.
  2. Express is running a version of CORRE that is at least two versions behind ESM 6.5 and current release of Logger. There were many efficiencies gained in each new version of CORRE. I would say this is the primary difference you are seeing.

As for the timing of the command, it simply takes the size of each chunk in that table and divides by the number of events in the each chunk row. You can explore that table using toad or another sql tool to dig into the details. It has been a while since I did it and cannot remember exactly how the table is laid out.

Thanks,

Brook Watson

0 Likes
EMazurak1 Absent Member.
Absent Member.

Re: average event size

Jump to solution

The average RAW size of an event varies based on the device sending it.  In Presales, we use a sizing calculator that factors in the different sizes of a raw message, before CEF conversion.  Windows logs are 700 bytes, linux logs are 300 bytes, firewall logs are 250 bytes, and switch logs are 100 bytes. ( and on and on.  we have average size events for every type of device )

To expand on above, 1 windows event log message is about 700 bytes raw.  After normalization into CEF, it ends up being 1765 bytes.(as mentioned above, if it is destined for logger, it may be smaller, because there are less fields in the current logger database schema).  Then it gets compressed ~10:1, so it ends up around 170 bytes.

From a storage perspective, a day worth of 1 windows event server at 1eps is 86,400 events.  That will require ~60MB of storage space per day RAW, and 140MB of storage space if normalized into CEF.

Hopefully this sheds some light.

0 Likes
Established Member.. anwarrhce1
Established Member..

Re: average event size

Jump to solution

Thanks for your response.

Is there any way/query to check the avg event size on corr based esm/express for 24hrs or 30 days?

0 Likes
Established Member.. anwarrhce1
Established Member..

Re: average event size

Jump to solution

Thank you for your reply.

I'm also kinda aware of this calculation.

So the best way is what, have the calculator give u the GB/day amount and then on avg double the size and then apply 10:1 compression which will give you storage requirement on esm.

for example from all diff devices you are about to recieve 10GB/day RAW event and after doubled its 20GB/day.

Then apply the compression so it becomes 2GB storage is required daily.

We are saying that compression happenns as soon as events hits the DB ? Is this correct ?

adding some gents from hp:

0 Likes
Jurgen
Visitor.

Re: average event size

Jump to solution

Hi Anwar,

I do not work for HP but i can point you to the following answers from this thread:

------------

yes compression happens to batches/buffers/bunches of events at a time before being committed to storage. These batches/buffers/bunches of events accumulate in to what I think is called a 'chunk' that occupies a fixed storage size in the CORRE storage and the 'time series' of the chunk gets stored for future efficient lookup and retrieval, the 'time series' being the first/last timestamps of the events in the chunk stored

There is no documentation that verifies about compression in 6.5c. In old ESM 101 (for ESM 5.5 or earlier) the partition management section clearly says the event data will not be compressed for 2 days and after that it will be compressed for 20% for 14 days and after 14 days it will compressed 80%.

But for CORR no documentation, Installation Guide, ESM 101, Admin Guide say events will be compressed. Even the active archive data file is always in chunk of 1GB. It doesn't matter if the data inside a 1 day archive is just 400 MB it's still in a 1GB chunk.

-----------

I think you should specify for what purpose you want to know certain information.

Licensing:

1. You want to measure your GB's per day

Bandwidth or smartconnector performance calculation:

2. You want to measure amount of bandwith used from the smartconnector to the Manager/Logger

Bandwidth or manager performance calculation:

3. You want to measure amount of incoming events in the Manager before data enrichment happens

4. You want to measure amount of incoming events in the Manager after data enrichment happens

Storage and backup archive storage calculation:

5. You want to measure amount of gb's stored on a disk in total per day

6. You want to measure amount of gb's stored in the archives per day

i just put some examples in this one.


Kind regards,

Jurgen

0 Likes
Established Member.. anwarrhce1
Established Member..

Re: average event size

Jump to solution

You already taken that as compliment for working in hp so i'm not sorry about that then

So this is getting interesting, pain is the storage requirements; point number 5.

Also I'm not sure why this is not documented properly anywhere about the way corr handles the event storage and compression.

Suppose we've claculated the license requirement for a customer to be 50GB/day, now the question is how much storage space do we need to store events for 60 days. On avg if we consider little less than doubled of raw events which comes 80GB/day required for storage then we apply 8:1 (Not full 10:1) compression; it becomes 10GB/day that means we need 600GB (Give it 100GB more as buffer) of space to store for 60 days.

Can this be suitable for calculation of storage requirements ? or there is any other big bang theory needs to be applied ?

Can this example be assumed same for Logger 6 as well ? because ESM 6.8c and Logger 6.0 share the almost same storage mechanism as far as I heard.

Adding few gurus to shed some light.

0 Likes
aaronmcallister Absent Member.
Absent Member.

Re: average event size

Jump to solution

Hi ,

I'm a relatively new ArcSight customer and don't have the expertise to comment on your question. Hopefully others in the community are able to help.

Thanks,

0 Likes
Jurgen
Visitor.

Re: average event size

Jump to solution

Hah,

I think Anwar ment:   😉

0 Likes
Established Member.. anwarrhce1
Established Member..

Re: average event size

Jump to solution

Sorry aaronmcallister

my mistake.,I meant AKramer.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.