Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Knowledge Partner Knowledge Partner
Knowledge Partner
2843 views

"Boy Meets ArcSight" or "the fairy tales of a ill-treated ArcSight admin" --- Epsiode 1.0 - custom syslog parser ---


This tale is about me fighting a deep fight agains "Percy the Custom Syslog-Parser".

the epilog to this series can be found here

Once upon a time...

I got the task to write a parser for data which will arrive via a syslog connector.

Easy task i thought -  but far wrong.

The data did not have a syslog header... so I thought okay, give it a try, write a parser and see what happens.

And it came as it was expected - bad dragons all over the place.

First of all, I was unsure if my custom parser got picked up. So I read the magic book called   - over and over again, but yes i was right,

  • it told me to put my file in ARCSIGHT_HOME\user\agent\flexagent\syslog
  • and i should give it the name <vendor>.subagent.sdkrfilereader.properties

and than everything should work.

No chance!

My data was always picked up by "IBM|AIX_audit" parser, a very evil small parser... (ahh by the way I used SmartConnector Flame-Work Version 7.4)

So my next idea was not to fight against all syslog parsers in one fight, but just let my parser run alone.

I checked tha magic book (see above) and found out that I can limit the dragons(parsers) I need to fight with

  • agents[x].customsubagentlist
    • "Set this property to the restricted subagent list based on device types in your environment."

and I should

  • set agents[x].usecustomsubagentlist=true

in the agent.properies file.

But my sword was a dull saber, as my parser called serioussam never was picked up by the SmartConnector-flamework whem I set

agents[0].customsubagentlist=serioussam and

agents[0].usecustomsubagentlist=true

I checked varios tales about custom syslog parser an a invite only dragon hunt community called "protect724" - found a lot of helpless combatants like

the later one desciebed my isseu as well, but nobody was able to help this lost soul.

So i went back and set

agents[0].usecustomsubagentlist=false

and I checked the all-knowing oracle (not Oracle) called the agent.log which lives in the dark corner of ARCSIGHT_HOME\current\logs.

Some black magician told me once to move this file to another place or rename it befor I ask my question, to have a clear view on the logs

And so I did, and after sending some test data to the syslog-dragon, I could see

  • [INFO ][default.com.arcsight.common.config.AgentPropertiesFileConfiguration][customInitialization] customInitialization() - read properties from file [C:\arcsight\74x\current\user\agent\flexagent\syslog\serioussam.subagent.sdkrfilereader.properties

So I asked myself whats wong with the usecustomsubagentlist and the customsubagentlist

and then i saw the light at the ond of the tunnel - and it was not a train:

  • agents[0].customsubagentlist

seems only be handling self written parsers if you add another dragon here, I chose

  • agents[0].customsubagentlist=flexagent_syslog|serioussam

and set

  • agents[0].usecustomsubagentlist=true

And they lived happily ever after, my parser was picked up now and the AIX monster was elimated forever.

Lessons Learned

  1. custom parsers need to live in ARCSIGHT_HOME\user\agent\flexagent\syslog
  2. custom parsers need to be named  <vendor>.subagent.sdkrfilereader.properties
  3. it is not needed to that event.deviceVendor=__getVendor(SecondEncounter) (in parser file) needs to match the <vendor> portion of the properties file
  4. you can set agents[0].usecustomsubagentlist=true, but its not mandatory
  5. if you set it, don't put only your custom parser in the customsubagentlist, but add flexagent_syslog and generic_syslog as well
  6. |([^|]*)| means "catch-all" between two pipes, but not a pipe itself
  7. You can enable massive debug output setting this in agent.properties
    • log.channel.file.property.package.com.arcsight=0
    • log.global.debug=true
  8. You dont need a proper syslog header, just a good regex! to write a parser
Labels (3)
8 Replies
anurag2631 Absent Member.
Absent Member.

Re: "Boy Meets ArcSight" or "the fairy tales of a ill-treated ArcSight admin" --- Epsiode 1.0 - custom syslog parser ---

Interesting...waiting for more upcoming Episode Keep it up Andreas

Honored Contributor.. Karlo_Luiten Honored Contributor..
Honored Contributor..

Re: "Boy Meets ArcSight" or "the fairy tales of a ill-treated ArcSight admin" --- Epsiode 1.0 - custom syslog parser ---

Like your style

0 Likes
maystrovichva Super Contributor.
Super Contributor.

Re: "Boy Meets ArcSight" or "the fairy tales of a ill-treated ArcSight admin" --- Epsiode 1.0 - custom syslog parser ---

Interesting stuff))) I just want to make some correction for 6 point: regex is wrong, you omit an opening parenthesis. Regex should be like this:

|([^|]*)|

Knowledge Partner Knowledge Partner
Knowledge Partner

Re: "Boy Meets ArcSight" or "the fairy tales of a ill-treated ArcSight admin" --- Epsiode 1.0 - custom syslog parser ---

your right, ​, will correct it in the tail, Thanks for mentioning

Acclaimed Contributor.. Shaun Acclaimed Contributor..
Acclaimed Contributor..

Re: "Boy Meets ArcSight" or "the fairy tales of a ill-treated ArcSight admin" --- Epsiode 1.0 - custom syslog parser ---

Let me see if I can clarify some of these for you (from my experience):

2. you can name the override file anything you want as long as it ends with .subagent.sdkrfilereader.properties

3. you can set the vendor/product to whatever you want, it does NOT have to match the override filename

4. flexagent_syslog is what picks up the custom flexagent parsers, it already exists in the default subagentlist

5. if you only want the syslog smartconnector to pick up your custom flexagent parsers, then set customsubagentlist=flexagent_syslog and set usecustomsubagentlist=true

8. sometimes you just need to test with a very basic flexagent syslog parser [regex=(.*)] to figure out exactly what part of the syslog header the smartconnector will match and/or remove before being passed to your custom parser.

Martyn Hill Honored Contributor.
Honored Contributor.

Re: "Boy Meets ArcSight" or "the fairy tales of a ill-treated ArcSight admin" --

Hi Shaun and Andreas

Good clarifications below.

One small point on terminology - we should differentiate between an 'override' and a full custom flex parser. It doesnt change the gist of Shaun's last update, except...

For a custom (Syslog) flex, it is indeed true that it doesn't matter what you choose to call the stub-name - although using 'vendor' is the preferred/best practice (for when you're maintaining loads of flex's in the same framework/JVM.). And indeed, they are all enumerated as part of the 'flexagent_syslog' sub-agent type, so no explcit additons are required to the customsubagentlist.

For an 'override' on the otherhand - which is designed to augment/replace parts (or all) of an existing sub-agent - the naming has to match precisely the stock parser (which is semi-hard coded in to the framework, and sequenced against other sub-agents very specifically to avoid one gobbling another's event stream.) I know that's not what Shaun was trying to say, but the use of the term override should probably be avoided in this (flex) context.

A note on 'sequencing' of sub-agents... You have no (supported) control over the sequencing of the sub-agents - beyond enabling/disabling individual sub-agents through customsubagentlist (and its associated switch property.) If you must get eyes on how we have sequenced the long list of sub-agents (occasionally useful for Syslog or SNMP Unified), then send me a PM. Its a bit too knarly to expose here - no intent to sound patronising... But, then I am British 🙂

And yes, our Syslog framework (Flex or stock-parser) is capable of consuming events with or without RFC compliant Syslog headers. If they are present and match a very broad range of (lauighably 'compliant') RFC forms, then they will be silently gobbled-up ahead of the parser ever seeing that part - leaving you with just the 'message body' to play-with. What has been gobbled up automagically wil be made avauilable in the pre-defined fields '_SYSLOGxxx' from withij your flex parser to map to something intelligent in the ARST schema.

If you want to see what the Syslog framework attempts to match in a header, browse agent.defaults.properties. Its not especially transparent, but you'll see what you need if you care enough to look. Oh, and PLEASE don't go adjusting agent.defaults.properties - its a sure-fire way to confuse anyone else who then tries to debug or maintain your connectors - Flex or otherwise. I won't therefore tell you where it lives... 🙂

 

M.

Outstanding Contributor.. rtoni1 Outstanding Contributor..
Outstanding Contributor..

Re: "Boy Meets ArcSight" or "the fairy tales of a ill-treated ArcSight admin" --

Late to the party, but wanted to say thanks to Andreas, Shaun, Martin for the clarifications.

The notion that flexagent_syslog must be included in the customsubagentlist was not abundantly clear to me for sure.  I'm not a flexconnector guru by any stretch, and spent a bit of time agonizing over this same issue (not having that item included in my config) before coming back to research a bit more here and seeing that dragon slayed.  

In the flexconnector guide, the only mention of flexagent_syslog is in the example of what the syslog.prpoerties might look like (when doing a flex connector sanity check), which is a reasonable "clue", but IMHO it would be extremely helpful to explicitly note the need to include this in the customsubagentlist along with the flex itself - and highlight this in big bold font (for us old folk :-).

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: "Boy Meets ArcSight" or "the fairy tales of a ill-treated ArcSight admin" --

Found an interesting read in relation to the overall topic, of building your own parsers and cutomization around existing parsers, external mapping and an good overview of filenames, foldernames for flex-connectors

"FlexConnector deep dive and best practices" - from Protect 2014, see attached PDF.

 

I liked this table: 

Folder in ${ArcSight Connector Home}/current/user/agentpurpose
/flexagentparser
/acpevent categorization
/fcpstandard parser / parser override
/mapmap file
/extmapexternal mapper
/libjdbc driver

 

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.