Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Karl - thx for the output.  I looked at it a bit.  I'll carve out sometime soon to dig into what's going on here.  There's definitely something odd going on with the record parsing and this data sample.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Thanks Jeremy, don´t hesitate to contact me again if you need me to test something.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Wanted to circle back real quick.  I've gotten pulled into a couple of things that are consuming my time.  I will eventually fix this lacat issue, but it's going to be a couple of more days at least.

Sorry all if you're waiting on it.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Thanks Jeremy. I look forward to you post.

0 Likes
Reply
Highlighted
New Member.

Re: Logger Archive Extractions

very nice.

I got it working by modifying your find_cef_in_raw_gz funciton:

def find_cef_in_raw_gz(s):

    """returns generator to clean up records"""

    rec_delim = "\x00\x00\x00\x00\x00\x00\x00\x00"

    tail_delim = "\x00\x00\x01"

    cef_type = "\x43\x45\x46\x3a\x30"

    for r in s.split(rec_delim):

        if not r.startswith(cef_type):

            # print('not a cef event')

            continue

        # rec = r[12:] # strip off the cef binary header

        rec = r

        # strip off the tail delimeter, etc and emit our cef

        # print(rec[:rec.rfind(tail_delim)], '\n')

        yield rec[:rec.rfind(tail_delim)]

you should post this guy to github so I can submit changes.  I could also add multiprocessing amongst other changes, like working entirely in memory instead of using a tempfile.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Hey Jared,

I'm out and about for the next couple of days but when I get back in the office, I'll sit down and create a github repo.

Pull requests are awesome and earn you a beverage on me at Protect or some other place we'd end up being colocated.

-j

0 Likes
Reply
Highlighted
Contributor.
Contributor.

Re: Logger Archive Extractions

Hi Jared,

that works for me as well with archives from Logger 5.2 GA.

Jeremy, great post. this is exactly what I've been looking for. We see pretty slow performance on archived searches and I'd like to play around with post processing these archives to separate events by customer URI.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Hi Jared,

I tried exporting some archives from Logger 5.3SP1 and it works fine.

Thanks for the update.

0 Likes
Reply
Highlighted
Outstanding Contributor.
Outstanding Contributor.

Re: Logger Archive Extractions

Working fine for me too. Many thanks

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

I went ahead and put it up on github as requested to make it easier for anyone to contribute

hpsec/lacat · GitHub

Jared - I haven't integrated your changes yet.  I figure it'd be a simple way to get the pull requests started and you'd get the credit for contributing if you were interested.

0 Likes
Reply
Highlighted
New Member.

Re: Logger Archive Extractions

sweet man.  i updated mine to python3 using concurrent.futures (multiprocessing).  I also have a few hundred TB to convert and enrich.

-Jared

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Logger Archive Extractions

Jared -

Yezzz.  There are quite a few features I "missed" when I decided to lock it to py2.6 to play nice with RHEL.  Missing DictComprehensions bit me a few times.

If you want, create a pull request for a new branch if you're interested.  That way, maybe some others not as "OS Constrained" could use your changes if you don't mind sharing.  OR, better yet, just fork it on github.

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.