Mide1 Regular Contributor.
Regular Contributor.
760 views

Help with multiline Regex with truncated messages

Al,

I've been through the loads of multiline regex FlexConnectors, threads on here and unfortunately, they have not been able to solve my problem. I have a requirement to integrate AWS windows VMs via cloudwatch agents. The Windows OS logs when picked up by cloudwatch are sent through other AWS services and then aggregated in an AWS S3 bucket for central collection. As a result of the extra hops, there are additional info that are appended to the original windows OS message. These info includes the AWS account number and the AWS instance id which are both valuable. These info are in JSON format. So, in summary, we end up with the orginal windows message enclosed in  information that are in JSON format.

To parse this out, I have written a JSON flex which parses out the JSON fields and passes the rest of the windows message to a regex extraprocessor. I use a regex extraprocessor and not a keyvalue becuase the Windows OS logs are no longer keyvalue pairs. They "=" have been replaced with "\t", "\n", or "\r" . The other catch and the main challenge and main reason for these thread is that the Windows message are multiline. Therefore, I have been challenged with a way to parse.The messages are truncated and so, I'm struggling with the "multiline.ends.regex" property. I only need to parse out the event id and the hostname field from the Windows OS message.

Please see my parser below and the attached screen shot of the Windows OS message that is passed to the regex extraprocessor. This has been driving me bonkers for the last 3 days. Please help!

# FlexAgent Regex Configuration File
do.unparsed.events=true

multiline.starts.regex=\\[Security\\]\\s+\\[\\d+\\]\\s+\\[Microsoft-Windows-Security-Auditing\\]\\s+\\[\\S+\\]\\s+\\[.*\\.

multiline.ends.regex=(\\n|\\r)
multiline.max.count=3

regex=\\[Security\\]\\s+\\[(\\d+)\\]\\s+\\[Microsoft-Windows-Security-Auditing\\]\\s+\\[(\\S+)\\]\\s+\\[.*


token.count=2

token[0].name=eventid
token[0].type=String

token[1].name=hostname
token[1].type=String

additionaldata.enabled=true


#submessage.messageid.token=
#submessage.token=

event.deviceCustomString1=hostname
event.deviceCustomString1Label=__stringConstant(" Windows Hostname")
event.externalId=eventid


#l10n.filename.prefix=

Regards,

Mide.

 

 

 

0 Likes
2 Replies
Mide1 Regular Contributor.
Regular Contributor.

Re: Help with multiline Regex with truncated messages

I should also add that my latest error message from agents.log has to do with the infamous "NullPointerException" which doesn't help me troubleshoot whatsover.
0 Likes
Highlighted
Mide1 Regular Contributor.
Regular Contributor.

Re: Help with multiline Regex with truncated messages

Bump! I'm sure someone has to have a suggestion or two
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.