FlexConnector Error : Regex file for handling long message

Hi All,

I tried to create Regex configuration for below event.

"2014-02-12 10:35:11,258  ; UserId:26; Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details:

Trigger Condition:Send the alert if transactions fail  or transactions response time is greater than 10 seconds  or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once."

For above event i declared three tokens and below is config file. When i run the standalone application,

But in my CEF output file, i am getting three logs. Means, Message information is splitted into two more logs as below.


1. Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details:

2. Trigger Condition:Send the alert if transactions fail

3. or transactions response time is greater than 10 seconds

4. or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once.


Below is my Regex file. Please advise anything i need to change.

# FlexAgent Regex Configuration File

do.unparsed.events=true

regex=(\\d \\-\\d \\-\\d \\d\\d\:\\d\\d\:\\d\\d,\\d )  ; UserId\:(\\d ); (.*)\\.

token.count=3

token[0].name=Time_Of_Event

token[0].type=TimeStamp

token[0].format=yyyy-MM-dd HH\:mm\:ss,SSS

token[1].name=UserId

token[1].type=String

token[2].name=Message

token[2].type=String

#submessage.messageid.token=

#submessage.token=

event.name=Message

event.deviceReceiptTime=Time_Of_Event

event.sourceUserId=UserId

#l10n.filename.prefix=

Thanks

Jayakrishnan

.

  • Hi,

    I have checked the Raw logs & modified the parser Slightly.

    While testing in my test Environment & i am able to Parse the Logs Properly.

    Kindly find below the Updated Parser:

    --------------------------------------------------------------------------------------------------------------------------------------------

    #2014-02-12 10:35:11,258  ; UserId:26; Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details: Trigger Condition:Send the alert if transactions fail  or transactions response time is greater than 10 seconds  or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once.

    #2014-02-12 10:35:11,258  ; UserId:28; Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details: Trigger Condition:Send the alert if transactions fail  or transactions response time is greater than 10 seconds  or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once.

    #2014-02-12 10:35:11,258  ; UserId:27; Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details: Trigger Condition:Send the alert if transactions fail  or transactions response time is greater than 10 seconds  or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once.

    do.unparsed.events=true

    regex=(\\d \\-\\d \\-\\d \\s \\d \\:\\d \\:\\d \\,\\d )\\s \\;\\s UserId\\:(\\d )\\;\\s (.*\\s .*)\\.

    token.count=3

    token[0].name=Time_Of_Event

    token[0].type=TimeStamp

    token[0].format=yyyy-MM-dd HH\:mm\:ss,SSS

    token[1].name=UserId

    token[1].type=String

    token[2].name=Message

    token[2].type=String

    event.message=Message

    event.deviceReceiptTime=Time_Of_Event

    event.sourceUserId=UserId

    event.deviceVendor=__stringConstant("UnknownA")

    event.deviceProduct=__stringConstant(“UnknownB”)

    ---------------------------------------------------------------------------------------------------------------------------------------------------

    Let me know if you have any queries on the same.

    Regards,

    karthik

  • Hi Karthik,

    Thanks for your reply. It works fine.

    My issue is, we don't have access to alter the raw logs.

    There may be few events with tab space as above.

    Is there any method to trim, remove space and modify in parser itself?

    Thanks in Advance

    Jayakrishnan

  • Hi Jaya Krishnan,

    In Your main regex=(\\d \\-\\d \\-\\d \\s \\d \\:\\d \\:\\d \\,\\d )\\s \\;\\s UserId\\:(\\d )\\;\\s (.*\\s .*)\\.


    In the Highlighted part of the above Regex, You can increase the "Any whitespace character[s]"

    based on the tab space in your Raw logs.


    Else if you are having multiple events with different Tab Space, I would suggest you to create multiple patterns in your parser for corresponding Tab Spaces.


    Else if your raw log itself is getting split-ted into multiple lines, You can initially use a "multiline.starts.regex" function & later apply your parser.

    Regards,

    karthik

  • Hi Karthi,

    Will do accordingly. I have one more issue and need your help.

    Below are two events

    2014/07/22 01:01:12 WARNING: Discovery Unable to find any Agents in the discovery area.

    2014/07/22 00:01:44 : Scheduler Next scheduled action is Discover_Neutron.exe at 07/22/14 00:30:00

    First one have WARNING and another one don't have.

    I would like to declare three tokens like Time_of_event, Severity and Message.

    My second token may or may not have value. So i wrote as below.

    # FlexAgent Regex Configuration File

    do.unparsed.events=true

    regex=(\\d\\d\\d\\d\\/\\d\\d\\/\\d\\d \\d\\d\:\\d\\d\:\\d\\d)\\s (.*)\\\:\\s (.*)

    token.count=3

    token[0].name=Time_Of_Event

    token[0].type=TimeStamp

    token[0].format=yyyy/MM/dd HH\:mm\:ss

    token[1].name=Message

    token[1].type=String

    token[2].name=Action

    token[2].type=String

    submessage.messageid.token=Message

    #submessage.token=

    event.deviceProdcut=__stringConstant("Reporter")

    event.deviceVendor=__stringConstant("HP Overview")

    event.deviceAction=Action

    event.deviceReceiptTime=Time_Of_Event

    event.message=Message

    #l10n.filename.prefix=

    But in my CEF output, i am not able to see any information except date.

    Anything wrong with my declaration.

    Please advise.

    Thanks

    Jayakrishnan

  • Hi Karthi,

    I used below two lines.

    multiline.starts.regex=\\d \\-\\d \\-\\d \\d\\d\:\\d\\d\:\\d\\d,\\d (.*)

    regex=(\\d \\-\\d \\-\\d \\d\\d\:\\d\\d\:\\d\\d,\\d )\\s ;\\s UserId\\\:(\\d )\\;\\s (.*\\s .*)

    It works perfectly for my original log file which has multi line events and tab spaces inbetween.

    Thanks a lot for your advise,

    Jayakrishnan

  • Hi Jaya Krishnan,

    Kindly use the below parser for your 2nd Query, I have already tested it in my setup.

    --------------------------------------------------------------------------------------------------------------------------------------------

    do.unparsed.events=true

    regex=(\\d \\/\\d \\/\\d \\s \\d \\:\\d \\:\\d )\\s (.*)\\:\\s (.*)

    token.count=3

    token[0].name=Time_Of_Event

    token[0].type=TimeStamp

    token[0].format=yyyy/MM/dd HH:mm:ss

    token[1].name=Message

    token[1].type=String

    token[2].name=Action

    token[2].type=String

    event.deviceReceiptTime=Time_Of_Event

    event.name=Action

    event.message=Message

    event.deviceProduct=__stringConstant("Reporter")

    event.deviceVendor=__stringConstant("HP Overview")

    -----------------------------------------------------------------------------------------------------------------------------------------------

    Kindly let me know if you have any Queries.