Absent Member.
Absent Member.
17 views

Logger Archive Extractions

I've developed a simple script that allows one to export CEF events directly from archive files.

Brief disclaimer - this tool is officially not supported and not maintained.  I'm providing this script here in the hopes someone finds it useful.  If you make any improvements, please feel free to share them back with the community.  What follows is the README from the tarball, as it has some good examples and instructions.

lacat

This is a simple utility that exports CEF records from a Logger archive file. It prints them to stdout by design, allowing the user to redirect them to a file or pipe them into something else (grep, awk, whatever) for further manipulation.

Written in python (targetting 2.6.x) and using only the standard libraries that should be available on all RHEL installations, this should be fairly self contained.

Usage

$ ./lacat -h

Usage: lacat [options] path_to_dat path_to_meta

Extracts cef events from Logger Archive files to stdout

THIS SOFTWARE IS NOT SUPPORTED.  USE AT YOUR OWN RISK.

Why is it called lacat?

    Because "Logger_Archive_cat" was too long to type.


Options:

  -h, --help            show this help message and exit

  -j, --json            export as json instead of raw cef

  -f FILTER, --filter=FILTER

                        specify a key=val to filter records by. multiple -s

                        k=v allowed

The usage is hopefully quite straightforward and the implementation fast enough.  I'm still optimizing it a bit to squeeze a bit more performance so check back here for revisions.

Installation

Place the file lacat in your path and make the file executable:

chmod +x lacat

Examples

Export raw CEF and capture in the file outfile.cef


./lacat ArcSight_Data_1_0504403158265495556.dat ArcSight_Metadata_1_504403158265495556.csv  > outfile.cef

Export all CEF records, one per line in JSON format, and capture in outfile.json

./lacat -j ArcSight_Data_1_0504403158265495556.dat ArcSight_Metadata_1_504403158265495556.csv  > outfile.json

Filter results by limiting output to destination IP 10.0.0.1

./lacat -f dst=10.0.0.1  ArcSight_Data_1_0504403158265495556.dat ArcSight_Metadata_1_504403158265495556.csv

Filter results by limiting output to destination IP 10.0.0.1 and UDP events only.

./lacat -f dst=10.0.0.1 -s proto=UDP  ArcSight_Data_1_0504403158265495556.dat ArcSight_Metadata_1_504403158265495556.csv

Notes

Multiple -s options can be specified to create an AND condition.  You can always specify -j to get each record output in JSON for ease of parsing with other languages.

35 Replies
Highlighted
Outstanding Contributor.
Outstanding Contributor.

Re: Logger Archive Extractions

Hi Jeremy,

First of all thank you for sharing this with us, extracting CEF events from Logger archive files is clearly something which can be very useful. That being said I tried to run lacat with some archive files and the script didn't return anything. I guess the files were fine as I didn't get any error message and the script ran about 40 seconds. I'm not very familiar with logger archive files so I could have missed something obvious. Any idea on what could be the problem ?

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Gaetan - not sure what's going on.  What do the contents of your archive directory look like and what cmd line did you use to run the script?

0 Likes
Reply
Highlighted
Outstanding Contributor.
Outstanding Contributor.

Re: Logger Archive Extractions

Jeremy,

This is how it looks like:

lacat.jpg

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Gaetan - I'm going to private message you for followup.  That's odd.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Great tool. Thank you for the contribution. Marked for future use.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

looks like an excellent tool. Are the archive files stored in raw cef? I wonder if 'lacat' could be adapted to dump json from a connector or manger? Connectors and the manager using the Forwarding connector ( Super Agent) can output cef via syslog. I am intrested in seeing that cef turn into json.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Jeremy,

I seem to have the same problem as Gaetan.

Did you manage to solve it ?

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Samuel - The archive files are not raw format, exactly...  At first, each chunk of events is output in cef but with special binary delimiters that connote a couple of different types of records.  Then, each chunk is gzipped independently and bundled together into a larger dat file and the metadata for each chunk is written to the csv file.  Each chunk is also padded with binary metadata.

Karl - we're exchanging emails now trying to track down the issue.  My initial hunch is that the binary delimiter being used may be changed slightly with different Logger versions.

If you're interested in seeing CEF turn to JSON, we should talk.  I have this working in my lab at the moment for an as of yet unannounced project (hopefully SOON) but I would be happy to give you code snippets and pointers to move in this direction.

Could you do me a favor?

Line 104 is

if not r.startswith( cef_type):

    continue

Could you change it to be

if not r.startswith(cef_type):

   print hexlify(r[:60])

   continue

And share that?  That will print out in hex the actual bytes of the first 60 characters of the record it thinks it's found and I can then compare that to the simple parser's criteria.

I should probably add something like that as a debugging mode... 






0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

These patterns don´t seem to match what I have in my raw file.

I followed your instructions and uploaded out.hex.

rec_delim = "\xab\x6c\x77\x00\x00\x00\x00"
tail_delim = "\x00\x00\x01\x43"
cef_type = "\x0a\x00\x00\x25"

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Jeremy,

Thanks for the response. I am interested in turning CEF into JSON. Code snippets, pointers and guidance would be much appreciated. As of now I am sending CEF via syslog-ng to another system but Json is what I really want. I was looking around in protect 24x7 to see if there was any progress in this area. I came across your posting and also across the REST Flex connecter for json coming back into ESM.

Sam

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Sam - alright you've inspired me.  I'll try to post a blog post early to mid next week with some code samples that should help you get this going.  It's not difficult, there's just a couple of moving pieces that can be tricky to get working together.

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.