This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Custom Regex Parser For Specific Logs On Syslog SmartConnector

I have a syslog SmartConnector which is collecting logs from few devices.

I have a custom log from the same source device which is not getting parsed. 
I wrote a regex parser for specific log however, I would like to know how to apply that parser to specific log that I am trying to target without impacting all other existing parsing. 

  • For the Syslog Connector, the parsers will be evaluated in the order of the custom subagent list that is listed in the agent.properties file.  For the custom parser that you wrote, it would be evaluated as the flexagent_syslog subagent.  If your parser is unique enough it should not be captured by your other device sources, however, if you are concerned about that then you can verify that your flexagent_syslog subagent is listed after all your other source devices (or you can edit the custom subagent list to make sure that it is). 

  • Or just create a new dedicated smart connector to collect those events. So, you will be 100% sure it wont impact the production connector you are running on. 

  • I created a product_syslog.subagent.sdkrfilereader.properties file with the regex utility of ArcSight. The file was placed under /flexagent/syslog.

    Removed the syslog.properties and restarted the SmartConnector. 

    The logs are still appearing as unparsed events. Although, I revalidated my regex parser through the utility and it looks okay.

  • If you're willing to share your parser and some sample events, I can take a look at it to see if I can find any issues. 

  •    Thank you for the help.

    Raw logs look like this.

    <158>Feb  5 14:55:56 <F5Hostname> info perl[27116]: 01310053:6: ASMConfig change: URL <URL> [update]: Start enforcing element. { audit: policy = <F5PolicyName> username = <AccountName> client IP = <IPAddress> }"

    <158>Feb 5 14:58:33 <F5Hostname> info perl[27116]: 01310053:6: ASMConfig change: URL <URL> [delete] { audit: policy = <F5PolicyName> username = <AccountName> client IP = <IPAddress> }"
    <158>Feb 5 14:58:33 <F5Hostname> info perl[27116]: 01310053:6: ASMConfig change: URL <URL> [delete] { audit: policy = <F5PolicyName> username = <AccountName> client IP = <IPAddress> }"
    <158>Feb 5 14:58:33 <F5Hostname> info perl[27116]: 01310053:6: ASMConfig change: URL <URL> [delete] { audit: policy = <F5PolicyName> username = <AccountName> client IP = <IPAddress> }"

    # FlexAgent Regex Configuration File
    do.unparsed.events=true

    regex=.*(ASMConfig\\schange)\:\\s(\\S+)\\s(.*?)\\[(\\S+?)\\].*policy\\s\=\\s(.*?),\\susername\\s\=\\s(.*?),.*

    token.count=6

    token[0].name=Message
    token[0].type=String

    token[1].name=ObjectName
    token[1].type=String

    token[2].name=ObjectValue
    token[2].type=String

    token[3].name=Action
    token[3].type=String

    token[4].name=PolicyName
    token[4].type=String

    token[5].name=Username
    token[5].type=String


    #submessage.messageid.token=
    #submessage.token=

    event.deviceCustomString3=Action
    event.deviceCustomString2=ObjectValue
    event.deviceCustomString1=ObjectName
    event.destinationUserName=Username
    event.name=Message
    event.deviceCustomString4=PolicyName

    event.deviceVendor=__stringConstant("F5")
    event.deviceProduct=__stringConstant("f5asm")


    #l10n.filename.prefix=

  • Looks like "info perl[27116]: 01310053:6:" isn't part of the syslog header and not being picked up in the regex of the parser.  Using .* upfront or in the middle can give some mixed results, so I'd suggest the following to account for the first part:

    regex=info perl\\[\\d+]\\\:\\s+\\d+\\\:\\d+\\\:\\s+(ASMConfig\\schange)\:\\s(\\S+)\\s(.*?)\\[(\\S+?)\\].*policy\\s\=\\s(.*?)\\susername\\s\=\\s(.*?).*