Highlighted
Super Contributor.. Super Contributor..
Super Contributor..
12773 views

Flex Connector || Multi line pharsing

Jump to solution

Hi Everyone,

I am trying to use a Flex connector for a Linux application log

But, I find that some of the logs are in one line

And some are in two or more line

 

I have find some idea, seem need to use multi-lines parsing 

But what should I use at multiline.ends.regex?

https://community.microfocus.com/t5/ArcSight-User-Discussions/Advanced-multiline-parsing-text-log-with-FlexConnector/td-p/1663232

https://community.microfocus.com/t5/ArcSight-User-Discussions/Parsing-for-mixed-single-and-multi-line-logs-Callenge-Help/td-p/1650207

 

My expression : \[(\S+)\]\-\[(\d+:\d+:\d+.\d+)\]\s(\S+)\s\:\s(\d+)\s(\S+)\:(\d+)\s\[(.*)\]

multiline.starts.regex=^([INFO]|[EBUG])
multiline.ends.regex=

20.png21.png

 

Regards

Tony

0 Likes
1 Solution

Accepted Solutions
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Hi Tony,

"multiline.ends.regex=" is not mandatory, but it's a good practice to have it set. In your case, don't use it, the parser will consider and end of line when it sees the "multiline.starts.regex=" again.

Mustapha

View solution in original post

13 Replies
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Hi Tony,

"multiline.ends.regex=" is not mandatory, but it's a good practice to have it set. In your case, don't use it, the parser will consider and end of line when it sees the "multiline.starts.regex=" again.

Mustapha

View solution in original post

Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Hi,

 

May the Regex tester not working with multi line?

22.PNG

Regards

Tony

0 Likes
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

I don't think it recognises multiline, but I might be wrong as I personally don't use it.

If you still want to use the Regex tester, I suggest that you play around using your text editor to convert your sample log to a single line event. Then, Regex tester should read this easily, it's just a matter of adding "multiline.starts.regex=" to your flex later on.

Note: Even if it's a multi-line, you still need to have "regex=...." matching the whole event from start to end, including what you've got in  "multiline.starts.regex="

Hope this helps.

Mustapha
0 Likes
Highlighted
Honored Contributor.. Honored Contributor..
Honored Contributor..

Hi,

Hope you are doing fine.

would you please tell me how to parse multiline sys log file?

kindly have a look at the sample log.

<188>2019-12-18 06:50:34 DC-FW-01 %%01SEC/4/SESSION(l): -DevIP=10.1.1.19; Protocol:tcp; 45.116.232.45:9059; -->202.83.164.173:80; 10.2.9.102:80; [2019/12/18 11:50:19 - 2019/12/18 11:50:34] Src VPN ID:0 Dst VPN ID:0; status:1

User name:45.116.232.45;
<188>2019-12-18 06:50:34 DC-FW-01 %%01SEC/4/SESSION(l): -DevIP=10.1.1.19; Protocol:tcp; 113.197.54.162:8880; -->103.63.3.227:445; [2019/12/18 11:49:6 - 2019/12/18 11:50:34] Src VPN ID:2 Dst VPN ID:2; status:1

User name:113.197.54.162;
<188>2019-12-18 06:50:34 DC-FW-01 %%01SEC/4/SESSION(l): -DevIP=10.1.1.19; Protocol:udp; 34.245.205.135:8664; -->202.83.164.168:53; 10.2.3.100:53; [2019/12/18 11:50:1 - 2019/12/18 11:50:34] Src VPN ID:0 Dst VPN ID:0; status:0

User name:34.245.205.135;

waiting for your kind response.

Regards

0 Likes
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Hi  MikeAlpha, 

 

As a starter, I will do the follow

1. Make the log in one line

2. Use the connector Redex to make Flex connector 

3. Add "multiline.ends.regex=xxx" to the config file as last

 

regards

Tony

0 Likes
Highlighted
Honored Contributor.. Honored Contributor..
Honored Contributor..

Hi Tony,

Thank you so much for your kind response.

I have created a flex connector for following multiline syslog. But it is successfully assigning values in 
Flex agent regex tester. But it is not parsing the values when installed. I can view the complete message in event.name field. Please help to resolve the issue and  help to confirm whether multiline work for syslogs or not.   

LOG:

***********************************
<188>2019-12-18 06:50:34 DC-FW-01 %%01SEC/4/SESSION(l): -DevIP=10.1.1.19; Protocol:tcp; 45.116.232.45:9059; -->202.83.164.173:80; 10.2.9.102:80; [2019/12/18 11:50:19 - 2019/12/18 11:50:34] Src VPN ID:0 Dst VPN ID:0; status:1

User name:45.116.232.45;

***********************************

Below is the code

multiline.starts.regex=.*\\d+\\-\\d+.*
multiline.ends.regex=.*User\\s+name\\\:.*\\;
do.unparsed.events=true
regex=(\\d+\\-\\d+\\-\\d+\\s+\\d+\\\:\\d+\\\:\\d+)\\s+(\\w+\\-\\w+\\-\\d+).*DevIP\=(\\d+\\.\\d+\\.\\d+\\.\\d+)\\;\\s+Protocol\\\:(\\w+)\\;\\s+(\\d+\\.\\d+\\.\\d+\\.\\d+)\\\:(\\d+)\\;.*\\>(\\d+\\.\\d+\\.\\d+\\.\\d+)\\\:(\\d+)\\;\\s+(\\d+\\.\\d+\\.\\d+\\.\\d+)*\\\:*(\\d+)*\\;*\\s*+\\[(\\d+\\/\\d+\\/\\d+\\s+\\d+\\\:\\d+\\\:\\d+)\\s+\\-\\s+(\\d+\\/\\d+\\/\\d+\\s+\\d+\\\:\\d+\\\:\\d+)\\]\\s+Src\\s+VPN\\s+ID\\\:(\\d+)\\s+Dst\\s+VPN\\s+ID\\\:(\\d+)\\;\\s+status\\\:(\\d+)(.*User.*)

token.count=16

token[0].name=timestamp
token[0].type=String

token[1].name=DevName
token[1].type=String

token[2].name=DevIP
token[2].type=IPAddress

token[3].name=Protocol
token[3].type=String

token[4].name=SourceIP
token[4].type=IPAddress

token[5].name=SourcePort
token[5].type=String

token[6].name=DesIP1
token[6].type=IPAddress

token[7].name=DesPort1
token[7].type=String

token[8].name=DesIP2
token[8].type=String

token[9].name=DesPort2
token[9].type=String

token[10].name=TimeReq
token[10].type=String

token[11].name=TimeEstb
token[11].type=String

token[12].name=SourceVpnId

token[12].type=String

token[13].name=DstVpnId
token[13].type=String

token[14].name=Status
token[14].type=String

token[15].name=SubMsg
token[15].type=String


submessage.messageid.token=Status
submessage.token=SubMsg

 

event.deviceAddress=DevIP
event.deviceVendor=__stringConstant("Huawei")
event.deviceProduct=__stringConstant("NTC12AM")
event.destinationAddress=DesIP1
event.transportProtocol=Protocol
event.name=DevName
event.sourceAddress=SourceIP


#l10n.filename.prefix=

submessage.count=2

submessage[0].messageid=0
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=.*User name\:(\\d+\\.\\d+\\.\\d+\\.\\d+);
submessage[0].pattern[0].fields=event.sourceUserId

submessage[1].messageid=1
submessage[1].pattern.count=1
submessage[1].pattern[0].regex=.*User name\:(\\d+\\.\\d+\\.\\d+\\.\\d+);
submessage[1].pattern[0].fields=event.sourceUserId

kindly help in resolving the issue.

Regards

0 Likes
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Hi MikeApha,

 

There is a sample at "current\user\agent\contrib\flexagent" -> "multiline.sample.sdkrfilereader.properties"

Maybe you can take a look on it

 

Regards

Tony

 

0 Likes
Highlighted
Respected Contributor.
Respected Contributor.

Hi @tonyssbear @MikeAlpha @mustapha_arakji ,

I have a problem while writing flex connector for some logs not getting parsed. Some of the logs are in single line while some are in two lines with second line having only doubled inverted comma.

Please find the log pattern below:

Pattern 1:

<180> 03/18/2020:15:34:56 XYZ <not blocked>

Pattern 2:

<180> 03/18/2020:15:34:56 XYZ <blocked>

"

Hope you notice the difference between the pattern 1 and pattern 2. The first pattern doesn't have double inverted comma in the second line. 

I am blank on this. Since I am not a multiline parser expert and I have no idea on how I can proceed further, I direly need your help.

Thanks in Advance.

Regards,

Mitesh Agrawal

 

 

 

0 Likes
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Hi Mitesh,

Your flex should work fine event with this double quote line (you will get errror in the agent.log indicating that the regex doesn't match) but all should be good for the the other lines and parsing should work fine. Double check your regex and make sure it 100% matches without the double quote line.

To clean the double quote out, you can use this property (line.ignore.regex=) which is a regular expression that when matches a line, the line is excluded and not processed.

Based on the example provided, you don't have to use the multiline.starts.regex property, still even if you decide to use it, you can still ignore the line using the property above.

 

Hope this helps.

Mustapha
Highlighted
Respected Contributor.
Respected Contributor.
Thanks for a quick reply. This is for syslog connector and when I added the line below it didn't work:
line.ignore.regex=\\"

As of now since this is syslog, I guess the above suggestion didn't work. Please help.

Thanks in Advance.
Regards,
Mitesh Agrawal
0 Likes
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

I see, you're right, this might not match as it doesn't have the syslog headers; which makes me wonder, how did you extract the raw logs to get these samples look like this...

Is the double quote being paser as a single event?

Anyway, If it was me, I wouldn't be much concerned now about the double quote, you can always include .* at the end of the regex if you think the double quote will be part of the data somewhere.

If you could share the regex you're using, I might be able to help more.

Mustapha
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.