Highlighted
mkoy Absent Member.
Absent Member.
657 views

Event Field Mapping for Apache Logs Through Syslog Connector

Hi All,

For one of our Apache servers, we get logs through a syslog connector.

Sorrily, all valuable fields are mapped to Name field on ESM side, which, I guess, prevents a proper correlation.

A sample Name field from ESM is below:

95.7.61.126 - user1 [29/Apr/2011:13:49:42 +0000] "GET /images/img_none.png HTTP/1.1" 304 -

Isn't it possible to make a better mapping, just as mapping the IP to Attacker Address,...etc.

Best Regards,

Labels (2)
0 Likes
Reply
4 Replies
aneeshpskadavil1 Honored Contributor.
Honored Contributor.

Re: Event Field Mapping for Apache Logs Through Syslog Connector

Hi,

Did you find a solution to this problem.

regards,

Aneesh

0 Likes
Reply
ianfitz Outstanding Contributor.
Outstanding Contributor.

Re: Event Field Mapping for Apache Logs Through Syslog Connector

If it is a syslog daemon connector, you should just need to modify syslog.properties and add the appropriate subagent first in the list for the device of interest. You may also need to check your custom subagent list settings in agent.properties to see that you didnt accidentally exclude it.

You can find the correct subagent name/details by examining an unzipped version of syste/agent/*.aup (rename to zip and extract).

If the problem is that the data is in a syslog file you are following, and the event is being read as a syslog event (Unix/Unix), then you can create a Unix parser override that, based on a regex condition (or otherwise), can conditionally spawn an additional apache processor (specify details as found above in the unzipped aup).

Both methods work, depends on the specifics of your connector/data sources/streams.

HTH,

Ian.

0 Likes
Reply
danneaffholder1 Absent Member.
Absent Member.

Re: Event Field Mapping for Apache Logs Through Syslog Connector

Ian Fitz

first let me apologize for being a newb.

Can you explain more regarding unzipping the AUP and spawning an additional apache processor --- I cannot find the details that are trying to reference in the unzipped AUP file - I am working with  SmartConnectors build version 5.2.7.6474.0 --- what should I be looking for

Thanks

0 Likes
Reply
ianfitz Outstanding Contributor.
Outstanding Contributor.

Re: Event Field Mapping for Apache Logs Through Syslog Connector

Hi there,

Sorry it is in $CONNECTOR_HOME/system/agent/fcp/*.aup

Create a subdir and move the file into the subdir, rename it to x.zip

Unzip x.zip and you will see a whole lot of directories created.

Within each directory are the parsers for each type (normally named ...subagent....)

So from this you will find the required path to the parser, and the parser name.  You need to know the path and the filename up until .sdkrfilereader.properties, so if it were apache.subagent.sdkrfilereader.properties, and it was in the apache directory, the string you need is: apache/apache.subagent

So... then somewhere along the line you will need to identify that it is an apache log.  You can either do this in your flexagent, or if your connector is reading in all lines as generic_syslog, you could add a parser override (and put it at user/agent/fcp/<the path/filename to the normal generic syslog subagent that you can also find above))

To see how it works in the override, start with something silly like:

#if you expect to see the string GET or POST in there somewhere, and with to use your subparser, then

regex=(<some regex that will match the prefix to all events>(?:(GET|POST){0,1}).*)

token.count=2

token[0].name=all

token[1].name=identifier

event.flexString1=identifier

event.flexString2=all

# i forget the exact syntax right now - doing it from memory, but its something like:

extraprocessor.count=1

extraprocessor.type=regex

extraprocessor.folder=path/filename  <<< ie the path to the apache subagent you found above

extraprocessor.conditionfield=event.flexString1

extraprocessor.conditionvalues=GET,POST

extraprocessor.field=event.flexString2

extraprocessor.subagent=true

extraprocessor.clearfieldafterparsing=false

extraprocessor.flexagent=true

So... you can probably find the exact syntax for extraprocessors in the flex connector dev guide or on protect 724, hopefully pretty close to the above.  What it all means is:

  • If there was a match where identifier  was GET or POST (ie flexString1 was GET or POST here), then call the system parser for the apache (or whatever) parser - and you pass the whole event into the system parser (ie the field you pass to the parser is stored in flexString2)
  • if not, then worst that will happen is flexString1 and 2 contain some text that was not there before
  • if you wanted you could use the additionalregexparsing/.../regex.0.sdkrfilereader.properties concept to clear out these fields again.

Hope this gets you started!

Cheers,

Ian.

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.