Help with multiline Regex with truncated messages
I've been through the loads of multiline regex FlexConnectors, threads on here and unfortunately, they have not been able to solve my problem. I have a requirement to integrate AWS windows VMs via cloudwatch agents. The Windows OS logs when picked up by cloudwatch are sent through other AWS services and then aggregated in an AWS S3 bucket for central collection. As a result of the extra hops, there are additional info that are appended to the original windows OS message. These info includes the AWS account number and the AWS instance id which are both valuable. These info are in JSON format. So, in summary, we end up with the orginal windows message enclosed in information that are in JSON format.
To parse this out, I have written a JSON flex which parses out the JSON fields and passes the rest of the windows message to a regex extraprocessor. I use a regex extraprocessor and not a keyvalue becuase the Windows OS logs are no longer keyvalue pairs. They "=" have been replaced with "\t", "\n", or "\r" . The other catch and the main challenge and main reason for these thread is that the Windows message are multiline. Therefore, I have been challenged with a way to parse.The messages are truncated and so, I'm struggling with the "multiline.ends.regex" property. I only need to parse out the event id and the hostname field from the Windows OS message.
Please see my parser below and the attached screen shot of the Windows OS message that is passed to the regex extraprocessor. This has been driving me bonkers for the last 3 days. Please help!
# FlexAgent Regex Configuration File
event.deviceCustomString1Label=__stringConstant(" Windows Hostname")