logging and searching unstructured data

[marcony1]

Our customer wants to log some data from the billing system applicatin level. The log files generated by these systems are not well suited for parsing with smart or flex connectors, due to some reasons...

As these systems are considered as high priority systems, the customer's log management policy require that log files from these systems should be logged on the existing logger appliance. So the questions are:

1. Is it possible to save these log files on logger appliance, in original form (the log files are textual files), using file transfer receivers?

2. If original log files can be saved on logger appliance, in original form, without any parsing and transformation, would it be possible to use logger appliance search capabilities, to search for some word phrases on these files?

3. Would it be possible to make search on a portion of files, based on some time criteria (from date1 to date2)?

Milan

[GCA]

Milan,

1 : I would say yes.  For instance, you could consider each line as being one new entry.  You can use a regex like : (.*)$ and store it in the $name field

2: I would say yes too, logger search capability is quite good and you can search from the raw event or from the field(s) defined in 1)

3. I would say yes if you use the time the logger has recorded the event ( this means you should push these logs to your logger on a regular basis to avoid having the real event time being too different from the logger time )

I never tried that but it should be feasible.  That being said, even a very basic parsing could be useful if you plan to make search in these logs.

HTH

[marcony1]

Hi GCA,

That would of course be one possible solution!

What might be under the question, is the size of the line. As these are some customer application logs, I cannot be sure that some line would not be greater then 4000 bytes, tha is the max size for event.rawEvent field. On the other side, event.name field can hold up to 512 bytes, what might be not enough in some cases.

As there is no other info that I can get about these log files, and the way they are created, I initially wanted to use file transfer receiver, on the logger appliance. But again, I am not sure what are limits for size of the files that would be ftp-ed or scp-ed from source system to the logger appliance. I shall contact Arcsight support to check that.

Anyway, your idea is worth to be tested. Thanks,

Milan