Captain
Captain
371 views

logging and searching unstructured data

Our customer wants to log some data from the billing system applicatin level. The log files generated by these systems are not well suited for parsing with smart or flex connectors, due to some reasons...

As these systems are considered as high priority systems, the customer's log management policy require that log files from these systems should be logged on the existing logger appliance. So the questions are:

1. Is it possible to save these log files on logger appliance, in original form (the log files are textual files), using file transfer receivers?

2. If original log files can be saved on logger appliance, in original form, without any parsing and transformation, would it be possible to use logger appliance search capabilities, to search for some word phrases on these files?

3. Would it be possible to make search on a portion of files, based on some time criteria (from date1 to date2)?

Milan

Labels (3)
0 Likes
2 Replies
Admiral
Admiral

Milan,

1 : I would say yes.  For instance, you could consider each line as being one new entry.  You can use a regex like : (.*)$ and store it in the $name field

2: I would say yes too, logger search capability is quite good and you can search from the raw event or from the field(s) defined in 1)

3. I would say yes if you use the time the logger has recorded the event ( this means you should push these logs to your logger on a regular basis to avoid having the real event time being too different from the logger time )

I never tried that but it should be feasible.  That being said, even a very basic parsing could be useful if you plan to make search in these logs.

HTH

0 Likes
Captain
Captain

Hi GCA,

That would of course be one possible solution!

What might be under the question, is the size of the line. As these are some customer application logs, I cannot be sure that some line would not be greater then 4000 bytes, tha is the max size for event.rawEvent field. On the other side, event.name field can hold up to 512 bytes, what might be not enough in some cases.

As there is no other info that I can get about these log files, and the way they are created, I initially wanted to use file transfer receiver, on the logger appliance. But again, I am not sure what are limits for size of the files that would be ftp-ed or scp-ed from source system to the logger appliance. I shall contact Arcsight support to check that.

Anyway, your idea is worth to be tested. Thanks,

Milan

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.