syslog connector multiline regex parse problem

Hello all!

I'm developing a parser for logs which contains multiline and single line records aswell. It catches the multline records finely as long as i use the 'regex=(.*)' statement, but with it, this parser also captures logs it shouldnt. There are about ~10 different devices that would send logs to this connector, i could separate them by submessages, but it came to discussion that there will be linux audit logs through syslog, and since i have the (.*) regex, the parser captures it instead of a default parser.

The relevant parts of the parser:

# FlexAgent Regex Configuration File
multiline.starts.regex=^\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}.\\d{3}\\s\\s\\S .*
#multiline.singleline.nowaiting=True
multiline.ends.regex=^\\}$
do.unparsed.events=False

regex=(.*)

token.count=1

token[0].name=Message
token[0].type=String

 

submessage[0].pattern.count=11
submessage[0].pattern[0].regex=(\\d \\-\\d \\-\\d \\s\\d\\d\:\\d\\d\:\\d\\d\\.\\d ) .*?(\\S ) (\\d )(.*?)\\[(.*?)\\]\\s(.*?)\:(.*)
submessage[0].pattern[0].fields=event.deviceReceiptTime,event.deviceSeverity,event.name,event.message,event.deviceCustomString3
submessage[0].pattern[0].names=$1,$2,$6,$7,$5
submessage[0].pattern[0].mappings=$1|$2|$6|$7|$5
submessage[0].pattern[0].types=TimeStamp,String,String,String,String
submessage[0].pattern[0].formats=yyyy-MM-dd HH:mm:ss.SSS,null,null,null,null

The multiline event:

2019-11-05 12:51:44.758 INFO 1 --- [processname] component : data: {
"0": {
"someId": 5,
"someMoreId": "201",
"someVersion": "3.5"
},
"1": {
"someId": 1,
"someMoreId": "2033",
"someVersion": "2"
},
"2": {
"someId": 5,
"someMoreId": "7564",
"someVersion": "1"
}
}

PS: i also tried to modify the agent.properties, and put the generic_syslog before the flexagent_syslog
agents[0].customsubagentlist=cef_syslog|linux_auditd_syslog|generic_syslog|flexagent_syslog

Parents Reply Children