Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Highlighted
mat053241 Super Contributor.
Super Contributor.
1008 views

Question regarding regex syslog flexconnector

Currently trying to create a parser for the Cisco ISE product with Arcsight, but its a bit complicated...at least for my level of expertise with this product for the moment.

I've been parsing the raw logs for events we would like to see, but the logs aren't in a "normal" syslog format, making this process difficult.  The problem i'm having is regarding how to parse these things since the log files are split into different pieces, even though they have the same ID.  Some of the individual logs have over 10 parts to each individual ID.  The lines below reflect the first portion of the header...what would be a good way to parse this using the regex flex connector?

                                                 ID             Part

CISE_Passed_Authentications 0002079847 4 0

CISE_Passed_Authentications 0002079847 4

CISE_Passed_Authentications 0002079847 4

CISE_Passed_Authentications 0002079847 4

Full sample:

Sep  4 00:02:23 t-na-e-dc-4 t-na-e-dc-4 CISE_Failed_Attempts 0002079848 3 0 2012-09-04 00:02:23.307 -05:00 0069172882 5400 NOTICE Failed-Attempt: Authentication failed, ConfigVersionId=14, Device IP Address=10.2.25.10, Device Port=32768, DestinationIPAddress=10.7.1.28, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=mat67746, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ti-svc-wi_b-dc-3, User-Name=test, NAS-IP-Address=10.2.25.160, NAS-Port=29, Service-Type=Framed, Framed-MTU=1300, State=37CPMSessionID=0ae4ffa00003898250457d50\;43SessionID=ts-na-e-c-4/134430/295\;, Called-Station-ID=00-00-00-00-00-00, Calling-Station-ID=0c-04-00-00-00-00, NAS-Identifier=tis-w_bw-dc-3, NAS-Port-Type=Wireless - IEEE 802.11, Tunnel-Type=(tag=0) VLAN, Tunnel-Medium-Type=(tag=0) 802, Tunnel-Private-Group-ID=(tag=0) 3002, cisco-av-pair=audit-session-id=0ae4ffd50, Airespace-Wlan-Id=1, AcsSessionID=tis-n-i-dc-4/134439/2950,

Sep  4 00:02:23 t-na-e-dc-4 t-na-e-dc-4 CISE_Failed_Attempts 0002079848 3 1  SelectedAccessService=Default Network Access, FailureReason=12321 PEAP failed SSL/TLS handshake because the client rejected the ISE local-certificate, Step=11001, Step=11017, Step=15008, Step=15048, Step=15048, Step=15004, Step=11507, Step=12500, Step=11006, Step=11001, Step=11018, Step=12301, Step=12300, Step=11006, Step=11001, Step=11018, Step=12302, Step=12319, Step=12800, Step=12805, Step=12806, Step=12807, Step=12810, Step=12305, Step=11006, Step=11001, Step=11018, Step=12304, Step=12305, Step=11006, Step=11001, Step=11018, Step=12304, Step=12305, Step=11006, Step=11001, Step=11018, Step=12304, Step=12305, Step=11006, Step=11001, Step=11018, Step=12304, Step=12319, Step=12815, Step=12321, Step=12307, Step=11504, Step=11003, NetworkDeviceGroups=Device Type#All Device Types#Wireless Controllers, NetworkDeviceGroups=Location#All Locations#, ServiceSelectionMatchedRule=Dot1X_Wireless, EapTunnel=PEAP,

Sep  4 00:02:23 t-na-e-dc-4 t-na-e-dc-4 CISE_Failed_Attempts 0002079848 3 2  OpenSSLErrorMessage=SSL alert: code=0x230=560 \; source=remote \; type=fatal \; message="unknown CA", OpenSSLErrorStack=  120492:error:14418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:s3_pkt.c:1102:SSL alert number 48, CPMSessionID=0ae4fd50, EndPointMACAddress=00-00-00-00-00-00, Device Type=Device Type#All Device Types#Wireless Controllers, Location=Location#All Locations, Response={RadiusPacketType=AccessReject; },

Any help would be appreciated with this. 

Thanks,

Matt

------
Labels (2)
0 Likes
Reply
10 Replies
dharris@evercom Absent Member.
Absent Member.

Re: Question regarding regex syslog flexconnector

the generic_syslog parser will see part of the header. The timestamp and whatever that string is after the timestamp will be mapped to deviceHostname, (is it actually a hostname?) Create a submessage parser. You could use the CISE_<string> as a device Event class ID and use this to start your submessage. Is that number after CISE_<string> unique for every event?

So the regex would be something like:

Sep  4 00:02:23 t-na-e-dc-4 t-na-e-dc-4 CISE_Failed_Attempts


.*?[^ ]\\sCISE_(Failed_Attempts|Passed_Authentication)\\s(.*)

token[0].name=devEventClassId

token[1].name=startofMySubmessage

submessage.messageid.token=devEventClassId

submessage.token=startofMySubmessage

submessage[0].messageid=Failed_Attempts

submessage[0].pattern[0].count=1

submessage[0].pattern[0].regex=<blah>



0 Likes
Reply
mat053241 Super Contributor.
Super Contributor.

Re: Question regarding regex syslog flexconnector

Thanks for the response Dave - that definitely helps.  The number after CISE_<string> is different for each individual log record, but it's essentially cut into pieces when it's sent out.

Each piece contains a different set of information (section 1 might contain AD group information, section 4 could contain the device MAC address, etc.)   I guess i'm still a bit lost concerning how I might extract different items from the same Log ID when those parts aren't exactly within the same log item.

For example:

CISE_Failed_Attempts 0002079848 2 0 (Contains source IP, Username, Failure reason,

CISE_Failed_Attempts 0002079848 2 1 (Contains Location, MAC address and endpoint profile)

OR

CISE_Passed_Authentications 0002079848 4 0 (Contains source Ip, Username, destination port)

CISE_Passed_Authentications 0002079848 4 1 (Contains AD groups user belongs to)

CISE_Passed_Authentications 0002079848 4 2 (Contains source MAC address and endpoint profile)

CISE_Passed_Authentications 0002079848 4 3 (Contains indentity information)

Also, since there are many different failure Id's within the "FailureReason" item, how would I go about createing the regex string?  Would this be just another submessage pattern within the "Failed_Attempts" submessage.messageid?  Or would I need to create a new submessageid for each individual failure reason ID that I want to see?

For example:

Sep  3 00:02:14 tis-na-ise-dc-4 tis-na-ise-dc-4 CISE_Failed_Attempts 0002058856 3 1  SelectedAccessService=Default Network Access, FailureReason=12321 PEAP failed SSL/TLS handshake because the client rejected the ISE local-certificate, Step=11001, Step=11017, Step=15008,

Sep  3 00:04:11 tis-na-ise-dc-4 tis-na-ise-dc-4 CISE_Failed_Attempts 0002058856 3 1  SelectedAccessService=Default Network Access, FailureReason=24436 Machine Lookup in Active Directory failed, Step=11001, Step=11017, Step=15008,

Thanks again for the help!

Matt

------
0 Likes
Reply
dharris@evercom Absent Member.
Absent Member.

Re: Question regarding regex syslog flexconnector

there's a few things you could do here. I would use the ID after the CISE string as the submessage breakout

sorry dont have time right now but if you can wait a day i will get you a skeleton parser and you can see where i'm going with it.

0 Likes
Reply
mat053241 Super Contributor.
Super Contributor.

Re: Question regarding regex syslog flexconnector

I'd really appreciate that.  Thanks a lot for the help!

------
0 Likes
Reply
dharris@evercom Absent Member.
Absent Member.

Re: Question regarding regex syslog flexconnector

Matt

Here's the start of a submessage parser for your device

I split the submessage using the ID after the 'CISE_Failed_Attempts'

It's quick and dirty but hopefully this gives you an idea.

You also need to tweak agent.properties to include cise_syslog in the customsubagents list and put it right at the front.

and set usecustomsubagentlist to true

Change event field names where you see fit but make sure you use indexed fields wherever possible. Check the Field Sets under ArcSight System/Sortable Field sets if you are not sure what has been indexed.

create the categorisation file with deviceSeverity as the key.

comments.start.with=#

#do.unparsed.events=true

regex=[^\\s]+\\s(CISE_Failed_Attempts|CISE_Successful_Attempts)\\s(.*)

token.count=2

token[0].name=eventId

token[0].type=String

token[1].name=submsg_body

token[1].type=String

event.deviceSeverity=eventId

event.message=submsg_body

severity.map.medium.if.deviceSeverity=CISE_Failed_Attempts

severity.map.low.if.deviceSeverity=CISE_Successful_Attempts

event.deviceVendor=__stringConstant("Cisco")

event.deviceProduct=__stringConstant("CISE")

submessage.messageid.token=eventId

submessage.token=submsg_body

submessage.count=1

submessage[0].messageid=CISE_Failed_Attempts

submessage[0].pattern.count=2

submessage[0].pattern[0].regex=(0002058856)\\s(\\S\\s\\S)\\s+(SelectedAccessService)\\=([^,]+),\\s(FailureReason)\\=(\\S+)(.*)

submessage[0].pattern[0].fields=event.deviceEventClassId,event.deviceCustomString1,event.deviceCustomString2Label,event.deviceCustomString2,event.deviceCustomString3Label,event.deviceCustomString3,event.deviceCustomString4

submessage[0].pattern[1].regex=(0002079848)(.*)

submessage[0].pattern[1].fields=event.deviceEventClassId,event.deviceCustomString2


0 Likes
Reply
mat053241 Super Contributor.
Super Contributor.

Re: Question regarding regex syslog flexconnector

Thanks for that sample - it helps.  So what kind of connector would I need to set this up with?  Assuming just a basic Syslog damon? 

------
0 Likes
Reply
dharris@evercom Absent Member.
Absent Member.

Re: Question regarding regex syslog flexconnector

yep i used a syslog file reader but the syslog daemon will work.

let me know if you get stuck with anything

0 Likes
Reply
jgruwell Absent Member.
Absent Member.

Re: Question regarding regex syslog flexconnector

Did you ever get this working Matt? We are looking to integrate Cisco ISE and I was curious what solution you came up with.

0 Likes
Reply
mat053241 Super Contributor.
Super Contributor.

Re: Question regarding regex syslog flexconnector

Unfortunately I haven't had any time to work on it, but it's still in the back of my mind.  If/when i find a working solution, i'll report it back here.  In the meantime, if you figure it out, please feel free to post the results!

------
0 Likes
Reply
wrowe@neosecure Absent Member.
Absent Member.

Re: Question regarding regex syslog flexconnector

Hi Dave,

I am starting to develop a flex connector for Cisco ISE.  What I lack is a complete or partial Cisco ISE log so that I can see the format.  Obviously I cannot do this work without a sample log.  We do not have this device working yet since it needs to be installed into a working network and the engineers have not made that available to us yet.  So would it be possible that you could share a log format with me?  If you need to contact me offline I am werowe@walkerrowe.com.

regards,

Walker Rowe

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.