Frequent Contributor.. Samuel_DS Frequent Contributor..
Frequent Contributor..
395 views

Syslog flex not working

Hello Guys,

So I installed a syslog connector to integrate an application. It listens on port 516 TCP. However, the logs are not being properly parsed, so I decided to use a syslog subagent to parse the logs to desired fields. But my subagent hasn't worked quite well either.  regex2.PNG

So it's like my submessage fields are being ignored.  I have attached a sample of my regex below. 

# FlexAgent Regex Configuration File
do.unparsed.events=true

regex=(\\S+)\: (.*)

token.count=2

token[0].name=hpot
token[0].type=String

token[1].name=restofmessage
token[1].type=String


submessage.messageid.token=hpot
submessage.token=restofmessage

 

event.deviceVendor=__stringConstant(Symantec)
event.deviceProduct=__stringConstant(MTNloki)
event.name=restofmessage


#l10n.filename.prefix=

submessage.count=1


submessage[0].pattern.count=1
submessage[0].pattern[0].regex=\\w+\\\:(\\w+.\\w+.\\w+)\\s+\\w+\\\:(\\S+)\\s+\\w+\\\:(\\S+)\\s+\\w+\\\:\\s+\\w+\\\:(\\w+.\\w+.\\w+.\\w+)\\s+\\w+\\\:(\\S+)\\s+\\w+\:(\\S+)\\s+\\w+\\\:(\\d+)\\s+\\w+\\\:\\d+
submessage[0].pattern[0].fields=event.domainString1,event.sourceHostName,event.deviceCustomString1,event.destinationHostName,event.deviceCustomString2,event.deviceCustomString3,event.externalId
submessage[0].pattern[0].extramappings=event.name\=__stringConstant("MTNN Loki Event")

for the sample logs below:
<14>Jun 06 12:45:17 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV002.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821504
<14>Jun 06 12:45:06 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV03.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821488
<14>Jun 06 12:44:46 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV03.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821472
<14>Jun 06 12:44:27 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV002.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821454
<14>Jun 06 12:44:17 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV03.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821439
<14>Jun 06 12:43:46 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV03.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821420
<14>Jun 06 12:43:26 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV002.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821397
<14>Jun 06 12:43:06 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV03.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821373
<14>Jun 06 12:42:37 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV03.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821352
<14>Jun 06 12:42:26 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV002.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821335


Please I am in dire need of assistance.

 

Thanks,

Samuel.

0 Likes
9 Replies
Respected Contributor.. yakupisler Respected Contributor..
Respected Contributor..

Re: Syslog flex not working

Hello,

There should be a line that contains the matched value, add the following line to parser and delete syslog.properties file in the /user/agent/ path.

submessage[0].messageid=Javelin 

0 Likes
Honored Contributor.. simon.simcic@sr Honored Contributor..
Honored Contributor..

Re: Syslog flex not working

I would suggest using a key value parser with syslog.

https://www.youtube.com/watch?v=MI9b5CKF_Is

The messages in your logs look structured.

S

0 Likes
Frequent Contributor.. Samuel_DS Frequent Contributor..
Frequent Contributor..

Re: Syslog flex not working

Hello Yakupisler,
I have tried this but to no avail.
Thanks a lot for your response.
0 Likes
Respected Contributor.. yakupisler Respected Contributor..
Respected Contributor..

Re: Syslog flex not working

Did you edit agent.properties file with following parameters?

agents[0].usecustomsubagentlist=true
agents[0].customsubagentlist=flexagent_syslog|ciscopix_syslog... bla bla.

After editing this delete syslog.properties and /user/agent/agentdata then restart connector.

0 Likes
Frequent Contributor.. Samuel_DS Frequent Contributor..
Frequent Contributor..

Re: Syslog flex not working

Yes I have done all that.

It seems like the connector keeps ignoring my submessage fields but parses my base fields as the console screenshot shows. 

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Syslog flex not working

So i see the regex issue, but first just to be sure, my recommendation is to always have a syslog connector available that is not used by any other production systems. You can have multiple on the same server even.

When developing Flexparsers I always have a syslog connector specifically for that purpose, the main reason is that i remove all entries under customsubagentlist, it should only have the flexagent definition there like so:

agents[0].customsubagentlist=flexagent_syslog

Also set this option to get some more logging:

agents[0].unparsedevents.log.enabled=true

 And if you have this setup towards an ESM, ensure that "preserve raw events" are enabled under Connector-Configure-Default, and all the way at the bottom.

The reason for this is that in many cases the raw event coming in might be a bit different than you are expecting (if you use rsyslog for example it adds a tag to the start of the log by default).

Now that we have a connector setup for this we delete the syslog.properties file in /user/agent folder and start it up while tailing the agent.log.

It should now pop up with either of 3 errors:

1. Unable to parse line: XXX (This means that even your base regex did not hit).

2. Submessage with token XXX did not parse (your base regex hits but not the submessage token

3.No token found for submessage XXX (meaning that either your token is missing while there is no default catch-all token, or you are mapping the wrong field to submessagetoken).

 

Sorry for the long writeup, so let's get down to the issue:

First of all you do not need to escape ":" characters here, and if you need to escape them they still need double escaping.

Current version:

Log message:
Jun 06 12:45:17 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV002.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821504

Regex:
(\\S+)\: (.*)

Here you are currently trying to escape the ":" character, while the only characters you actually need to escape is:

. $ ^ { [ ( | ) ] } * + ? \ 

You are also missing the submessage token reference, which would just make it the default catch-all (you are not telling it about the token id Javelin).

Just quickly, though not tested, this should work, though feel free to update it a bit:

#Community sample file, created by Marius Iversen
replace.defaults=true
trim.tokens=true
comments.start.with=#

#Jun 06 12:45:17 OJTADSEC02 Javelin: domain:mtn.com.ng hostName:\\GOP1102153LT alarmType:ExternalNetUser accounts: destination:OJTSRV002.mtn.com.ng objectName:web dm:localhost:30031/JSM1 AlertId:4741 timeStamp:1559821504
regex=.*?(\\S*):\\s(.*)

token.count=2

token[0].name=submessagetokenid
token[0].type=String
token[1].name=submessagetoken
token[1].type=String

additionaldata.enabled=false

event.deviceVendor=__stringConstant(Symantec)
event.deviceProduct=__stringConstant(MTNloki)
event.name=stringConstant("Symantec MTNloki Event")

submessage.messageid.token=submessagetokenid
submessage.token=submessagetoken

submessage.count=1


# Pick this one if all messages always have the same format and order
submessage[0].messageid=Javelin
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=.*?domain:(\\S*)\\shostName:(\\S*)\\salarmType:(\\S*)\\saccounts:\\s(\\S*\\s)?destination:(\\S*)\\sobjectName:(\\S*)\\sdm:(\\S*)\\sAlertId:(\\d+)\\stimeStamp:(\\d+)
submessage[0].pattern[0].fields=event.DOMAIN,event.HOSTNAME,event.AlarmType,event.Account,event.destination,event.objectname,event.dm,event.alertid,event.timestamp
submessage[0].pattern[0].types=String,String,String,String,String,String,String,String,String
submessage[0].pattern[0].format=Null,Null,Null,Null,Null,Null,Null,Null,Null
submessage[0].pattern[0].extramappings=event.name\=__stringConstant("Some Name")


# Pick this one for a more generic one:
submessage[0].messageid=Javelin
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=.*?\\S*:(\\S*)\\s\\S*:(\\S*)\\s\\S*:(\\S*)\\s\\S*:\\s(\\S*\\s)?\\S*:(\\S*)\\s\\S*:(\\S*)\\s\\S*:(\\S*)\\s\\S*:(\\d+)\\s\\S*:(\\d+)
submessage[0].pattern[0].fields=event.DOMAIN,event.HOSTNAME,event.AlarmType,event.Account,event.destination,event.objectname,event.dm,event.alertid,event.timestamp
submessage[0].pattern[0].types=String,String,String,String,String,String,String,String,String
submessage[0].pattern[0].format=Null,Null,Null,Null,Null,Null,Null,Null,Null
submessage[0].pattern[0].extramappings=event.name\=__stringConstant("Some Name")

 

Few notes that are more specific to my style:

1. I tend to leave a ".*?" at the start, the reason for this is there has been so many times in which parsing does not work basically because there is a weird tab/space or any non UTF-8 character at the start of the message. This is solely a personal taste though.

2. Greedy parsers are quicker but of course might be more error prone. I left 2 submessage examples, one if the log format is always the same, that one is always recommended but has possibilities to not hit if a small thing changes, but then you ensure you know exactly what you are parsing in. The second option is a much greedy version utilizing \S* at any text/number/special character representation, this is mostly to ensure a special character does not mess up your \w for example. This works well on these specific syslog usercases in which values are always separated by a space afterwards.

3. I did not perform any mapping, so i just added the name of the event as it appears in the logmessage, so you could change it to anything you want.

4. Timestamp conversion was not added, currently set as string, but it's just a basic epoch time timestamp.

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
Frequent Contributor.. Samuel_DS Frequent Contributor..
Frequent Contributor..

Re: Syslog flex not working

Capture.PNG

 

Thanks a lot Marius for your detailed response. 

But sadly this didn't work either. 

Best Regards,

Samuel

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Syslog flex not working

Please enable raw logging as mentioned below and show me an example, if it matches with the raw logs you posted before.

The raw logs you posted in your initial post, was that directly from your syslog server or from raw event field on the ESM?

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Frequent Contributor.. Samuel_DS Frequent Contributor..
Frequent Contributor..

Re: Syslog flex not working

 

Raw logging has always been enabled.

I got the raw logs I posted ab initio from the raw event field.


logf.PNG

Regards,

Samuel

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.