Highlighted
saurabh.rohra Absent Member.
Absent Member.
783 views

Extraprocessor only using extraprocessor[0] to parse event. It is not using extraprocessor[1],2,3.. for parsing log file

Hello Everyone!

    I have a flex with 5 extra processors which will parse different log type from a single log file.

My issue is when I run the connector, only extraprocessor[0] is being used to parse the events and for logs which it is not able to parse it is giving below warning warning message in agent.logs

[2016-10-14 17:35:26,131][WARN ][default.com.arcsight.agent.sdk.c.r][parseTokensNow] Message [[Thu Aug 11 13:15:21 2016] [error] (20014)Internal error: [client 10.0.0.103:61731] AH01102: error reading status line from remote server demo.testfire.net:80] did not match the common regular expression [\[(\S+ \S+ \d+ \d+:\d+:\d+ \d+)\] \[error] \[client (\d+\.\d+\.\d+\.\d+)\] ModSecurity: (.*?)\. Pattern match (.*?)\. \[file (.*?)\] \[line "(.*?)"\] \[id "(\d+)"\] \[msg "(.*?)\"\] \[data "(.*?)\"\] \[severity "(\S+)"\] \[tag "(.*?)\"\] \[hostname "(.*?)"\] \[uri "(.*?)\"\] \[unique_id "(.*)"]], ignoring...

Ideally when the extraprocessor[0] is not able to parse the event, it should check extraprocessor[1],extraprocessor[2],extraprocessor[3],extraprocessor[4] to parse the event.

Kindly find the parser files and log file attached.

Please help me how i can solve this issue.

Thank you.

Saurabh R.

0 Likes
1 Reply
maystrovichva Super Contributor.
Super Contributor.

Re: Extraprocessor only using extraprocessor[0] to parse event. It is not using extraprocessor[1],2,3.. for parsing log file

You defined 5 extraprocessors, but all they are the same for connector. You have not point the connector how they differ. So the connector use only first one (extraprocessor[0]).

Option 1. You can point connector to use conditional selection. Read Using Extraprocessors in FlexConnectors.docx.

conditionfield - Specifies the condition field the extra processor uses.

conditiontype - Specifies how the condition field relates to the condition values.

conditionvalues - Specifies condition values. Use commas to separate multiple values.

For example,

extraprocessor[0].type=regex
extraprocessor[0].filename=extra2
extraprocessor[0].field=event.rawEvent

extraprocessor[0].conditionfield=event.deviceCustomString1

extraprocessor[0].conditiontype=equals

extraprocessor[0].conditionvalues=some_string
extraprocessor[0].clearfieldafterparsing=true
extraprocessor[0].flexagent=true

Read it as if event.deviceCustomString1 equals some_string then use extraprocessor[0].

Regex example,

extraprocessor[0].type=regex
extraprocessor[0].filename=extra2
extraprocessor[0].field=event.rawEvent

extraprocessor[0].conditionfield=event.deviceCustomString1

extraprocessor[0].conditiontype=regex

extraprocessor[0].conditionvalues=Security.*
extraprocessor[0].clearfieldafterparsing=true
extraprocessor[0].flexagent=true

Read it as if regex pattern "Security.*" matches string in event.deviceCustomString1 then use extraprocessor[0].

So you can define something like this:

# FlexAgent Regex Configuration File
do.unparsed.events=false

regex=(.*)
token.count=1

token[0].name=Message_00
token[0].type=String

event.deviceVendor=__getVendor(Indusface)
event.deviceProduct=__getVendor(WAF)
event.rawEvent=Message_00
event.deviceCustomString1=__stringConstant("BaseProc")

#token[0].name=Timestamp
#token[0].type=TimeStamp
#token[0].format=EEE MMM dd HH\:mm\:ss yyyy

#token[1].name=message
#token[1].type=String

#event.startTime=Timestamp
#event.rawEvent=message

extraprocessor.count=5

extraprocessor[0].type=regex
extraprocessor[0].filename=extra2
extraprocessor[0].field=event.rawEvent

extraprocessor[0].conditionfield=event.rawEvent

extraprocessor[0].conditiontype=regex

extraprocessor[0].conditionvalues=ModSecurity\: Warning\\. Pattern match

extraprocessor[0].clearfieldafterparsing=true
extraprocessor[0].flexagent=true

extraprocessor[1].type=regex
extraprocessor[1].filename=extra3
extraprocessor[1].field=event.rawEvent

extraprocessor[1].conditionfield=event.rawEvent

extraprocessor[1].conditiontype=regex

extraprocessor[1].conditionvalues=ModSecurity\: Warning\\. Matched phrase
extraprocessor[1].clearfieldafterparsing=true
extraprocessor[1].flexagent=true

extraprocessor[2].type=regex
extraprocessor[2].filename=extra4
extraprocessor[2].field=event.rawEvent

extraprocessor[2].conditionfield=event.rawEvent

extraprocessor[2].conditiontype=regex

extraprocessor[2].conditionvalues=ModSecurity\: .*?\\. Match of
extraprocessor[2].clearfieldafterparsing=true
extraprocessor[2].flexagent=true

extraprocessor[3].type=regex
extraprocessor[3].filename=extra5
extraprocessor[3].field=event.rawEvent

extraprocessor[3].conditionfield=event.rawEvent

extraprocessor[3].conditiontype=regex

extraprocessor[3].conditionvalues=ModSecurity\: .*?\\. Pattern match
extraprocessor[3].clearfieldafterparsing=true
extraprocessor[3].flexagent=true

extraprocessor[4].type=regex
extraprocessor[4].filename=extra6
extraprocessor[4].field=event.rawEvent

extraprocessor[4].conditionfield=event.rawEvent

extraprocessor[4].conditiontype=regex

extraprocessor[4].conditionvalues=\\[(\\S+ \\S+ \\d+ \\d+\:\\d+\:\\d+ \\d+)\\] \\[error] (.*)
extraprocessor[4].clearfieldafterparsing=true
extraprocessor[4].flexagent=true

Option 2. You can do the same things with submessages.

# FlexAgent Regex Configuration File
do.unparsed.events=false

regex=(.*)
token.count=1

token[0].name=Message_00
token[0].type=String

event.deviceVendor=__getVendor(Indusface)
event.deviceProduct=__getVendor(WAF)
event.rawEvent=Message_00
event.deviceCustomString1=__stringConstant("BaseProc")

#token[0].name=Timestamp
#token[0].type=TimeStamp
#token[0].format=EEE MMM dd HH\:mm\:ss yyyy

#token[1].name=message
#token[1].type=String

#event.startTime=Timestamp
#event.rawEvent=message

#submessage.messageid.token=

submessage.token=Message_00

# Default submessage

submessage.count=1

submessage[0].pattern.count=5

submessage[0].pattern[0].regex=regex_from_extra2.sdkrfilereader.properties

submessage[0].pattern[0].fields=fields_mapping_like_in_extra2.sdkrfilereader.properties

submessage[0].pattern[1].regex=regex_from_extra3.sdkrfilereader.properties

submessage[0].pattern[1].fields=fields_mapping_like_in_extra3.sdkrfilereader.properties

submessage[0].pattern[2].regex=regex_from_extra4.sdkrfilereader.properties

submessage[0].pattern[2].fields=fields_mapping_like_in_extra4.sdkrfilereader.properties

submessage[0].pattern[3].regex=regex_from_extra5.sdkrfilereader.properties

submessage[0].pattern[3].fields=fields_mapping_like_in_extra5.sdkrfilereader.properties

submessage[0].pattern[4].regex=regex_from_extra6.sdkrfilereader.properties

submessage[0].pattern[4].fields=fields_mapping_like_in_extra6.sdkrfilereader.properties

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.