Absent Member.
Absent Member.
888 views

FlexConnector Error : Regex file for handling long message

Hi All,

I tried to create Regex configuration for below event.

"2014-02-12 10:35:11,258  ; UserId:26; Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details:

Trigger Condition:Send the alert if transactions fail  or transactions response time is greater than 10 seconds  or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once."

For above event i declared three tokens and below is config file. When i run the standalone application,

But in my CEF output file, i am getting three logs. Means, Message information is splitted into two more logs as below.


1. Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details:

2. Trigger Condition:Send the alert if transactions fail

3. or transactions response time is greater than 10 seconds

4. or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once.


Below is my Regex file. Please advise anything i need to change.

# FlexAgent Regex Configuration File

do.unparsed.events=true

regex=(\\d+\\-\\d+\\-\\d+ \\d\\d\:\\d\\d\:\\d\\d,\\d+)  ; UserId\:(\\d+); (.*)\\.

token.count=3

token[0].name=Time_Of_Event

token[0].type=TimeStamp

token[0].format=yyyy-MM-dd HH\:mm\:ss,SSS

token[1].name=UserId

token[1].type=String

token[2].name=Message

token[2].type=String

#submessage.messageid.token=

#submessage.token=

event.name=Message

event.deviceReceiptTime=Time_Of_Event

event.sourceUserId=UserId

#l10n.filename.prefix=

Thanks

Jayakrishnan

.

Labels (2)
0 Likes
6 Replies
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Hi,

I have checked the Raw logs & modified the parser Slightly.

While testing in my test Environment & i am able to Parse the Logs Properly.

Kindly find below the Updated Parser:

--------------------------------------------------------------------------------------------------------------------------------------------

#2014-02-12 10:35:11,258  ; UserId:26; Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details: Trigger Condition:Send the alert if transactions fail  or transactions response time is greater than 10 seconds  or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once.

#2014-02-12 10:35:11,258  ; UserId:28; Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details: Trigger Condition:Send the alert if transactions fail  or transactions response time is greater than 10 seconds  or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once.

#2014-02-12 10:35:11,258  ; UserId:27; Message:Alert BPM Alert (id: d800120df270de4543f2163ebc44c5e8) was updated with these details: Trigger Condition:Send the alert if transactions fail  or transactions response time is greater than 10 seconds  or transactions response time relative to configured thresholds is as specified  when trigger conditions occur even once.

do.unparsed.events=true

regex=(\\d+\\-\\d+\\-\\d+\\s+\\d+\\:\\d+\\:\\d+\\,\\d+)\\s+\\;\\s+UserId\\:(\\d+)\\;\\s+(.*\\s+.*)\\.

token.count=3

token[0].name=Time_Of_Event

token[0].type=TimeStamp

token[0].format=yyyy-MM-dd HH\:mm\:ss,SSS

token[1].name=UserId

token[1].type=String

token[2].name=Message

token[2].type=String

event.message=Message

event.deviceReceiptTime=Time_Of_Event

event.sourceUserId=UserId

event.deviceVendor=__stringConstant("UnknownA")

event.deviceProduct=__stringConstant(“UnknownB”)

---------------------------------------------------------------------------------------------------------------------------------------------------

Let me know if you have any queries on the same.

Regards,

karthik

0 Likes
Absent Member.
Absent Member.

Hi Karthik,

Thanks for your reply. It works fine.

My issue is, we don't have access to alter the raw logs.

There may be few events with tab space as above.

Is there any method to trim, remove space and modify in parser itself?

Thanks in Advance

Jayakrishnan

0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Hi Jaya Krishnan,

In Your main regex=(\\d+\\-\\d+\\-\\d+\\s+\\d+\\:\\d+\\:\\d+\\,\\d+)\\s+\\;\\s+UserId\\:(\\d+)\\;\\s+(.*\\s+.*)\\.


In the Highlighted part of the above Regex, You can increase the "Any whitespace character[s]"

based on the tab space in your Raw logs.


Else if you are having multiple events with different Tab Space, I would suggest you to create multiple patterns in your parser for corresponding Tab Spaces.


Else if your raw log itself is getting split-ted into multiple lines, You can initially use a "multiline.starts.regex" function & later apply your parser.

Regards,

karthik

0 Likes
Absent Member.
Absent Member.

Hi Karthi,

Will do accordingly. I have one more issue and need your help.

Below are two events

2014/07/22 01:01:12 WARNING: Discovery Unable to find any Agents in the discovery area.

2014/07/22 00:01:44 : Scheduler Next scheduled action is Discover_Neutron.exe at 07/22/14 00:30:00

First one have WARNING and another one don't have.

I would like to declare three tokens like Time_of_event, Severity and Message.

My second token may or may not have value. So i wrote as below.

# FlexAgent Regex Configuration File

do.unparsed.events=true

regex=(\\d\\d\\d\\d\\/\\d\\d\\/\\d\\d \\d\\d\:\\d\\d\:\\d\\d)\\s+(.*)\\\:\\s+(.*)

token.count=3

token[0].name=Time_Of_Event

token[0].type=TimeStamp

token[0].format=yyyy/MM/dd HH\:mm\:ss

token[1].name=Message

token[1].type=String

token[2].name=Action

token[2].type=String

submessage.messageid.token=Message

#submessage.token=

event.deviceProdcut=__stringConstant("Reporter")

event.deviceVendor=__stringConstant("HP Overview")

event.deviceAction=Action

event.deviceReceiptTime=Time_Of_Event

event.message=Message

#l10n.filename.prefix=

But in my CEF output, i am not able to see any information except date.

Anything wrong with my declaration.

Please advise.

Thanks

Jayakrishnan

0 Likes
Absent Member.
Absent Member.

Hi Karthi,

I used below two lines.

multiline.starts.regex=\\d+\\-\\d+\\-\\d+ \\d\\d\:\\d\\d\:\\d\\d,\\d+(.*)

regex=(\\d+\\-\\d+\\-\\d+ \\d\\d\:\\d\\d\:\\d\\d,\\d+)\\s+;\\s+UserId\\\:(\\d+)\\;\\s+(.*\\s+.*)

It works perfectly for my original log file which has multi line events and tab spaces inbetween.

Thanks a lot for your advise,

Jayakrishnan

0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Hi Jaya Krishnan,

Kindly use the below parser for your 2nd Query, I have already tested it in my setup.

--------------------------------------------------------------------------------------------------------------------------------------------

do.unparsed.events=true

regex=(\\d+\\/\\d+\\/\\d+\\s+\\d+\\:\\d+\\:\\d+)\\s+(.*)\\:\\s+(.*)

token.count=3

token[0].name=Time_Of_Event

token[0].type=TimeStamp

token[0].format=yyyy/MM/dd HH:mm:ss

token[1].name=Message

token[1].type=String

token[2].name=Action

token[2].type=String

event.deviceReceiptTime=Time_Of_Event

event.name=Action

event.message=Message

event.deviceProduct=__stringConstant("Reporter")

event.deviceVendor=__stringConstant("HP Overview")

-----------------------------------------------------------------------------------------------------------------------------------------------

Kindly let me know if you have any Queries.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.