Honored Contributor.. SIEM-TECH Honored Contributor..
Honored Contributor..
2525 views

Logger archives...a waste of time?

Jump to solution

So we have recently discovered that when the logger archives data, it's basically a non indexed flat file.  Apparently on top of that when you "load" an archive point beyond retention your not actually loading anything, rather just a pointer to the data on external storage (NFS).

As you can imagine, trying to search data over a year old no longer stored locally on the logger has proven ineffective.  Searching a week of data during that time period takes 8 - 13+ hours based on the number of events we ingest, which is likely more than most.  For investigative purposes this just doesn't cut it if I need to search a year or two of archived data.

Anyhow, I'm curious how many are aware of this and if any of you have implemented your own archiving solution?  ArcSight claims they are working this but there is no time table nor a gaurantee it will work with existing stored data, in fact it probably wont.  For us, this is just unnacceptable and we will be looking for alternatives.

-Mike

Labels (1)
0 Likes
1 Solution

Accepted Solutions
ianfitz Outstanding Contributor.
Outstanding Contributor.

Re: Logger archives...a waste of time?

Jump to solution

I agree, the current archiving approach is not a practical solution for our use case either (search speed is unusable for any sort of effective searching and reporting that we might want to run).  This archiving feature is probably suitable for some users needs, but certainly not the end solution for us. Purchasing additional loggers or a site license is the official solution to your problem. Not sure how many customers have deep enough pockets though...

An alternative storage and search solution is possibly a more cost effective solution depending on whether you have a few developers on hand...   A TCP forwarder or two would help you get your CEF data into an alternative search/storage system if you wanted to go in that direction...

This is a frustrating problem. We expected a performance hit on searching archives, but we didnt expect it to be as significant as it is.

Good luck,

Ian.

View solution in original post

0 Likes
29 Replies
ianfitz Outstanding Contributor.
Outstanding Contributor.

Re: Logger archives...a waste of time?

Jump to solution

I agree, the current archiving approach is not a practical solution for our use case either (search speed is unusable for any sort of effective searching and reporting that we might want to run).  This archiving feature is probably suitable for some users needs, but certainly not the end solution for us. Purchasing additional loggers or a site license is the official solution to your problem. Not sure how many customers have deep enough pockets though...

An alternative storage and search solution is possibly a more cost effective solution depending on whether you have a few developers on hand...   A TCP forwarder or two would help you get your CEF data into an alternative search/storage system if you wanted to go in that direction...

This is a frustrating problem. We expected a performance hit on searching archives, but we didnt expect it to be as significant as it is.

Good luck,

Ian.

View solution in original post

0 Likes
Honored Contributor.. SIEM-TECH Honored Contributor..
Honored Contributor..

Re: Logger archives...a waste of time?

Jump to solution

Ian,

You are dead on.  We are currenlty developing a solution it's just unfortunate you can't use the millions of dollars worth of equipment you already purchased and use.

Thanks for the response.

0 Likes
Tom_Chris1 Absent Member.
Absent Member.

Re: Logger archives...a waste of time?

Jump to solution

I find it extremely hard to believe that there is no way to archive the index file(s) daily, just like you archive your data.  My analysts have to conduct searches going back 1 year.  If a logger fails and all I have is the configuration backup and the data (no indexing), I am going to tell my client that it would probably take months to conduct a search going back 1 year.

I promise they won't be happy about that...

0 Likes
Honored Contributor.. SIEM-TECH Honored Contributor..
Honored Contributor..

Re: Logger archives...a waste of time?

Jump to solution

Tom,

Try loading an archive and searching data that is older than the retention period (not still searchable on the logger).  You'll get your answer. It's not even about the logger crashing.  Logger doesn't index archived data.   Better you know now than later.  -Mike

0 Likes
Vini Acclaimed Contributor.
Acclaimed Contributor.

Re: Logger archives...a waste of time?

Jump to solution

I will have to agree with you all, it is a big inconvenience.

You can still claim you have the data but search through it is a different story.

I have also heard that you cannot run reports on archived data that you have re-attached.

I believe that this problem will be resolved when Logger is shipped with the new version of the CORR engine.

0 Likes
sparky1 Absent Member.
Absent Member.

Re: Logger archives...a waste of time?

Jump to solution

Will be interesting to see if this change is made with the updated engine, cos right now the Express using CORR does index its archive data. I've no performance metrics around it, but presume its going to be better than what exists now 😉

0 Likes
ngreen1 Absent Member.
Absent Member.

Re: Logger archives...a waste of time?

Jump to solution

The only practical use I have found to archiving is to meet our requirement to store a backup of data offsite. The compression of the archive is so inefficient we are keeping more data on our logger than we are archiving.

I am glad to hear that Express archives the index as well as the data as we are about to implement Express.

Has anyone figure what format the logger archive is in? I have had a brief look at a file with a text editor and could not figure our anything that made sense in terms of CEF, which is what I was expecting.

0 Likes
sparky1 Absent Member.
Absent Member.

Re: Logger archives...a waste of time?

Jump to solution

Just be aware that express has a 200GB limit for archive retention. You can of course move the archives off to another location, but if needed you'll need to copy them back when needed. I've written a shell script to handle this if anyone is interested.

0 Likes
rpatel21 Absent Member.
Absent Member.

Re: Logger archives...a waste of time?

Jump to solution

Hi Vini,


We will be addressing the archiving indexing issue when Logger moves to CORRE. Similar to Express though, only a certain amount of data will be indexed and accessible. This is still very fluid and other options are also being considered for how indexed archived data would be exposed to the user.

-Roopak

0 Likes
Mat1 Absent Member.
Absent Member.

Re: Logger archives...a waste of time?

Jump to solution

The compression of the archive is so inefficient we are keeping more data on our logger than we are archiving.

Archives are not commpressed at all !

0 Likes
ngreen1 Absent Member.
Absent Member.

Re: Logger archives...a waste of time?

Jump to solution

Logger is kind enough to zip the archives, so there is some mild compression

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.