Question regarding Logger Log retrieval

Hi everyone,

I got a scenario that i need to use logger to retrieve all the logs from all the connectors for the past 6 months. Tried manual export and found out it is too troublesome as 1 hour of logs already exceed 1million rows limit, schedule search is not practical either as i will need to have 24 schedule searches just for 1 day of logs.

The environment have about 50GB of data per day, i did some research using RESTful API seems to be the better way to do it, and found the arcsightrest.py by Marius very useful (still trying to figure out some minor stuffs), plan to cut the 50GB into 12 files, 2 hours each.

My question for everyone is - is there a better approach of achieving this?

Thanks,

Keo

 

  • Keo,

    I'm little confused by your question. Are you asking if there is a way to retrieve 60 day logs stored in Logger? What is your Use Case? Since everything is in Logger, why are you trying to export every events in Logger? You can serach events of interest in Logger, but I don't understand why you want to export 60 days of logs. I can see a reson for total event counts in 60 days or certain user activity in 60 days or etc..., but could you clarify why you are trying to export every single logs for the past 60 days?

    thanks,

    Brian Chong

  • Hi Brian,

    Thanks for replying.

    My use case is to have all the logs stored in Logger, export it out and send it to another tools that i'm currently working on.

    To start with, i will need the past 60 days of data (all logs).

    Thanks,

    Keo

  • Keo,

    Now that I know what your end goal is, I recommend that you approach it differnently. Instead of export events, I would use the Logger forwarding connector to forward logs to your tool. Assuming your tool accepts CEF and I don't see reason not to since CEF is parsed and normalized. Now within, the forward connnector, there is a date you can specify so that you can go back several days, weeks or months to query the data and forward it to the destination. It will still take some time to send all 60 days worth of logs, but it's better approach than exporting it.

    thanks,
    Brian Chong

  • Hi Brian,

    Thanks for the reply!

    I didn't aware there is "Logger forwarding connector" as an option.

    Is this configurable from the Logger GUI under Configuration->Forwarder? Or it needs a forwarding connector setup file similar to ESM forwarding connector?

    Thanks!

    Keo

  • Hi Keo,

    I've double check this on Logger v6.4 and this can only be done on Logger -> ESM. I aplogize for the confusion.  I thought Logger can have other destination besides the ESM to forward., but it seems like ESM is the only destination  you can configure. as a destination.

    When you create or edit existing logger forwarder, there is check box named "filter my time range" and this option allows the user can specify time frame. However, this can only be done by forwarding logs to ESM only. No other desintation AFAIK.

    thanks,
    Brian Chong

  • Hello,

    just wanted to add as additional information.

    1) Logger 6.6 Administrator's Guide:
    https://community.softwaregrp.com/t5/Logger/Logger-AdminGuide-6-6/ta-p/1642065

    2) On page 395 topic "Forwarders" starts and on page 396 you can see that there more types of Forwarders, for example:
    a) UDP Forwarder UDP forwarders forward events by using the User Datagram Protocol
    b) TCP Forwarder: TCP forwarders forward events by using the Transmission Control Proto
    c) ArcSight ESM CEF Forwarders: ArcSight ESM CDF forwarders send Common Event Format (CEF) events to an ESM Destination. The built-in connector on Logger is used to forward these events to ESM.

    Regards,

    Marijo

  • Hi Marijo,

    Thanks!

    I managed to get logs out, and it's alot faster! 

    But now i'm facing another issue, whereby the logs forwarded out from Logger is not consistent.

    For example, i create a TCP Forwarder, using the same query, same date range, and same destination, i run this TCP forwarder setting few times, the logs size forwarded to the syslog server have different sizes on each attempts.

    Not sure whether you experienced this issue before, I've opened a case with ArcSight support to clarify further.

    Thanks,

    Keo