Valued Contributor.. sathariel Valued Contributor..
Valued Contributor..
708 views

Force usage of flex parser

Hi,

i have a strange Problem with one Flex Parser i've written. The situation is the following:

  • 8 devices (same vendor) send Messages via syslog to this connector
  • all devices send "normal" syslog Messages as Linux that get parsed as Unix/Unix (which is not too much of a Problem) regulary
  • 6 devices send application alerts in CEF via syslog - one is pretty active (every couple of minutes), the others send only some times a day.
  • those devices now also log other alerts via syslog (the header looks like the one of the CEF Messages) - here i need a submessage Parser, since it can't be sent in CEF and is also not real syslog.
  • Events in CEF and for the new alerts are sent from the same app_Sender daemon process.
  • in the Agent.properties the customsubagentlist is my_flexparser|cef_syslog

From the device that is pretty active, events are parsed completely fine. CEF is interpreted correct, my Parser gets the mappings that i want from the events and even the Linux syslog messages are fine. But on the devices that are not so active, my parser is always ignored (generic_syslog is used instead which ends up in a completely wrong message!) and even sometimes CEF Messages are displayed without formatting!

I am testing since a week and i don't know what i can do. I browsed all documentation, searched the whole Protect Forums, i always remove the syslog.properties file and clean up agentdata when i restart the syslog connector, i set usecustomsubagentlist to false/true/yes.

Looking in the syslog.properties files shows that it "learned" flexagent_syslog|cef_syslog|generic_syslog for one device, the one that does everything correct has only cef_syslog|generic_syslog - the others have only generic_syslog.

I really need an option to have my Parser used first, then cef_syslog and then generic_syslog as order. This would ensure that all Messages are parsed like they have to be parsed.

The connector Version is 7.4.0.7963.0. I could also give some examples (in a redacted form) but since the Parser itself is working, i don't think that this is necessary.

Thanks

Thorsten

Labels (3)
0 Likes
12 Replies
Highlighted
subindbabu Honored Contributor.
Honored Contributor.

Re: Force usage of flex parser

Hi,

Can you please try to edit the syslog.properties file manually ? For each device, edit the parser file name, in the way you want to prioritize.

--SUBIN--

--SUBIN--
0 Likes
Valued Contributor.. sathariel Valued Contributor..
Valued Contributor..

Re: Force usage of flex parser

Hi,

this looks better. Will syslog re-read this file from time to time or are changes applied immediatly? After the change i received one event from one device sill not correctly parsed (stupid - it is alerted as "su failed" while in the original message no su nor failed is occuring!). But for another device it did it correctly.

And still one thing bothers me here...the first line in syslog.properties:

#Automatically generated by syslog

Is there a way to define this outside of syslog.properties for the devices?

Thanks

Thorsten

0 Likes
subindbabu Honored Contributor.
Honored Contributor.

Re: Force usage of flex parser

Connector is referring this file to locate the corresponding parser file , when each event comes.

You just try to edit the file and restart the connector.

--SUBIN--
0 Likes
Valued Contributor.. sathariel Valued Contributor..
Valued Contributor..

Re: Force usage of flex parser

Hi,

i restarted the connector and enabled raw Events again. I will monitor the whole thing and report how it went. My first impression was, that it looks better now even without the restart.

Cheers

Thorsten

0 Likes
subindbabu Honored Contributor.
Honored Contributor.

Re: Force usage of flex parser

...good day

--SUBIN--
0 Likes
Acclaimed Contributor.. Shaun Acclaimed Contributor..
Acclaimed Contributor..

Re: Force usage of flex parser

Try setting "forwardmode" to "true" in your agent.properties.

0 Likes
Valued Contributor.. sathariel Valued Contributor..
Valued Contributor..

Re: Force usage of flex parser

Hi,

i found the setting. However i am unsure how this will work. I only have 2 syslog Parsers defined in my subagentlist (cef_syslog and my_flex_syslog) and it still uses generic_syslog as a Parser. The document says it will try every available Parser - so it has to go through all 100+ Parsers before choosing the right one? Performancewise this is a horror and i am not sure if this will work later (tomorrow i will extend the logging of events on the devices and there will be much more noise).

Btw. the editing of the syslog.properties was a great idea, Arcsight only choose to switch the cef_syslog parser with mine which is fine. So now i have cef_syslog|my_syslog|generic_syslog for the devices. But there is no way to configure this outside so in case the syslog.properties is deleted or a new device is added, this will be done automatically?

Cheers

Thorsten

0 Likes
pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: Force usage of flex parser

Couple of comments - have to run in a minute:

1) You cant control the sequence of the loading of the parsers - its regex and all builds together into a one large decision tree (this is the power of Regex) and there is a lot of logic that goes into this process. So its the logic that determines the order, not a list in a properties file - sorry.

2) You can have multiple subagent parsers in the syslog flex agent folder - if you cant hit each message with one subagent parser, just use multiple subagents. Thats fine. Its not the most efficient way to do this, but its fast, simple and usually gets around the complexity of sub parsers and so on. If its multiple subagents per message, thats really inefficient, but if it helps, its a simple way to solve this.

3) For the regex it must be a full 100% match of the message - this is how the decision tree is created. The message is run through the decision tree and it will select that parser based on the full match of the message. If it is getting hit by the incorrect parser, then it is making a full match based on that incorrect regex. So take a closer look at the message and try to figure out what the options are around making it a full match on the correct regex / parser combination. If its hitting the CEF one for example, see what you can do to make the message change / exactly hit the message.

4) As a final step, you can have multiple syslog connectors - not ideal, but if its an option you can have multiple ones with different parser setups in it to solve this.

5) You can change the header processing - its in the default properties file and you can see how it processes the header - its usually this that causes incorrect regex parsing because it cuts out the differentiator in the message (in the header). WARNING THOUGH - change this and ALL OTHER STANDARD PARSERS WILL BREAK. So only do this if you are only specifically looking at a couple of messages and you have them covered with FlexConnectors or existing parsers.

0 Likes
avihaigr Trusted Contributor.
Trusted Contributor.

Re: Force usage of flex parser

To elaborate to the 4'th solution  that Paul Brettle has suggested -

You can set a new destination to the unparssed messages and sent these messages to different connector.

(And filter out these messages from the orig destination).

It's a work around but this will make the work.

0 Likes
stefan.oancea Outstanding Contributor.
Outstanding Contributor.

Re: Force usage of flex parser

Hi Paul,

I particularly like your point number 1; finally I see someone clearly stating what I always felt like happening considering the testing I've done with the Syslog Connector. I was never really sure though, because I've also seen so many suggestions about changing the order of the parsers that in the end I was wondering if it is not actually me who is missing something .

So thanks for that!

One more question I would be happy to get a quick confirmation  for: if you completely remove specific parsers from the list in agent.properties, agents[0].customsubagentlist=, in that particular case is it true that the connector will completely skip those removed parsers during the processing of the logs? There are a few situations in which I found this useful, tried it and from what I recall it seemed to work. But again, a second opinion would be great.

Thanks,

Stefan

0 Likes
pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: Force usage of flex parser

Hi Stefan,

Yes, that is correct - if you remove the parser name from the list, the connector will then not load it. If you want to confirm this, just check the agent.log file when the connector loads and it will list out the parsers as it loads them (verbose logging is a little extreme) and importantly you can see it not in the list.

Does that make sense?

0 Likes
stefan.oancea Outstanding Contributor.
Outstanding Contributor.

Re: Force usage of flex parser

Hello Paul,

Yes, it surely does. Thanks for confirming that.

All the best,

Stefan

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.