Flexconnector Parser question

[klackey]

Hello,

I am trying to write a custom parser for a couple of events we are getting from a PBX into a syslog connector. The generic syslog parser handles the most of the events, but there are a couple that it is not handling correctly which I am trying to parse.

I have create the following 2 parser files:

NortelPBXLoginSucess.sdkrfilereader.properties &

NortelPBXLogout.sdkrfilereader.properties

and placed them in:

/home/ArcSightSmartConnectors/syslogconnector/current/user/agent/flexagent/syslog

and changed their file permissions to 777.

The first looks for logout events:

Dec  9 10:19:05 foo INFO: : 10.95.3.201: Info: User goo logout from server foo.boo.com:443.
Dec  9 10:20:55 foo INFO: : 10.95.3.201: Info: User goo logout from server foo.boo.com:443.
Dec  9 10:23:41 foo INFO: : 10.95.3.201: Info: User goo logout from server foo.boo.com:443.
Dec  9 10:25:25 foo INFO: goo: 10.95.3.201: Info: User goo logout from foo.boo.com:443.
Dec  9 10:25:38 foo INFO: goo: 10.95.3.201: Info: User goo logout from foo.boo.com:443.
Dec  9 10:25:49 foo INFO: goo: 10.95.3.201: Info: User goo logout from foo.boo.com:443.
Dec  9 10:26:32 foo INFO: goo: 10.95.3.201: Info: User goo logout from server foo.boo.com:443.
Dec  9 12:32:56 foo INFO: goo: 10.95.3.201: Info: User goo logout from server foo.boo.com:443.

The parser file for the logout event is as follows:

# FlexConnector Regex Configuration File
do.unparsed.events=false

regex=(\\w{3}\\s+\\d{1,2}\\s+\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\s+INFO:\\s*(\\w*):\\s+(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}):\\s+Info:\\s+(.*):(\\d+)

token.count=6

token[0].name=Timestamp
token[0].type=TimeStamp
token[0].format=MMM  dd HH\:mm\:ss
token[1].name=HostName
token[1].type=String
token[2].name=UserName
token[2].type=String
token[3].name=DestinationAddress
token[3].type=IPAddress
token[4].name=Message
token[4].type=String
token[5].name=DestinationPort
token[5].type=Integer

event.deviceReceiptTime=__useCurrentYear(Timestamp)
event.destinationAddress=DestinationAddress
event.destinationPort=DestinationPort
event.destinationHostName=HostName
event.destinationUserName=UserName
event.destinationUserId=UserName
event.name=__stringConstant("Logout")
event.message=Message
event.deviceVendor=__stringConstant("Nortel")
event.deviceProduct=__stringConstant("PBX")

I stopped the connector, deleted the syslog.properties file, and restarted the service but it is still not parsing.

Idaes?

Thanks!

[klackey]

The 2nd parser is a multi line file used to aggregate two events in order to extract enough info for it to be worth wile.

Events:

Dec  8 17:11:23 boo INFO: : 10.95.3.201: Info: id=goo
Dec  8 17:11:23 boo INFO: : 10.95.3.201: Info: Login Success

parser:

# FlexConnector Regex Configuration File
do.unparsed.events=false

multiline.starts.regex=(\\w{3}\\s+\\d{1,2}\\s+\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\s+INFO:\\s+:\\s+(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}):\\s+Info:\\s+id=(\\w+)
multiline.ends.regex=(Info:\\sLogin\\sSuccess)
regex=(\\w{3}\\s+\\d{1,2}\\s+\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\s+INFO:\\s+:\\s+(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}):\\s+Info:\\s+id=(\\w+)

token.count=4

token[0].name=Timestamp
token[0].type=TimeStamp
token[0].format=MMM  dd HH\:mm\:ss
token[1].name=HostName
token[1].type=String
token[2].name=DestinationAddress
token[2].type=IPAddress
token[3].name=UserName
token[3].type=String

event.deviceReceiptTime=__useCurrentYear(Timestamp)
event.destinationAddress=DestinationAddress
event.destinationHostName=HostName
event.destinationUserName=UserName
event.destinationUserId=UserName
event.name=__stringConstant("Authen OK")
event.message=__stringConstant("Login Sucess")
event.deviceVendor=__stringConstant("Nortel")
event.deviceProduct=__stringConstant("PBX")

thanks!

[klackey]

So I have been fighting with this and have made no real progress. The current version of my logout parser looks like:

# FlexAgent Regex Configuration File
do.unparsed.events=false

regex=.*INFO\:\\s+(\\S+)\:\\s+(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\:\\s+Info\:\\s(.*)server\\s(.*)\:(\\d+).

token.count=5

token[0].name=UserName
token[0].type=String

token[1].name=DestinationAddress
token[1].type=IPAddress

token[2].name=Message
token[2].type=String

token[3].name=HostName
token[3].type=String

token[4].name=DestinationPort
token[4].type=Integer

#submessage.messageid.token=

#submessage.token=

event.destinationPort=DestinationPort
event.deviceVendor=__stringConstant("Nortel")
event.name=__stringConstant("Logout")
event.destinationUserName=UserName
event.destinationAddress=DestinationAddress
event.message=__concatenate(Message,"server ", HostName, ":", DestinatoinPort)
event.destinationHostName=HostName
event.deviceProduct=__stringConstant("PBX")
event.destinationUserId=UserName
event.deviceSeverity=_SYSLOG_PRIORITY
event.sourceAddress=_SYSLOG_SENDER
event.deviceReceiptTime=__useCurrentYear(_SYSLOG_TIMESTAMP)

Works fine in the Regex tester on the following evetns:

goo.us.com INFO: boo: 10.5.1.187: Info: User boo logout from server goo.us.com:443.

or

DEC 14 17:32:35 goo.us.com INFO: boo: 10.5.1.187: Info: User boo logout from servergoo.us.com:443.

event log looks like:

[2010-12-14 17:26:20,679][INFO ][default.com.arcsight.agent.p.e][getInputStream] Resource [syslog/NortelPBXLogout.sdkrfilereader.properties] not found
[2010-12-14 17:26:20,679][INFO ][default.com.arcsight.agent.p.e][getInputStream] Resource [syslog/NortelPBXLogout.sdkrfilereader.properties] not found (AUP file ignored)
[2010-12-14 17:26:20,679][INFO ][default.com.arcsight.common.config.AgentPropertiesFileConfiguration][customInitialization] customInitialization() - read properties from file [/apps/arcsight/ArcSightSmartConnectors/syslogconnector/current/user/agent/flexagent/syslog/NortelPBXLogout.sdkrfilereader.properties].
[2010-12-14 17:26:20,680][INFO ][default.com.arcsight.agent.sdk.d.n][init] Successfully Parsed properties from file [syslog/NortelPBXLogout]

Any advice is appreciated.

[klackey]

Well, I figured out where my events are going. They are parsing as Cisco IronPort evetns. I tried changing the precedence in the syslog.properties file, but the connector seems to ignore my changes and then overwrites them restoring the old order.

Ideas?