tj_delmarco
Established Member.
421 views

Custom parser for IIS issues

I have developed a flexconnector which parses a custom IIS log (using IIS advanced logging) into ArcSight. For the most part, it works fine.

However, I keep getting a parsing error like such:

[2017-07-05 20:03:36,634][ERROR][default.com.arcsight.agent.baseagents.k.a.a][processLine] [java.lang.RuntimeException: Parser : Unknown State !(a with state :2).  Line:2017-07-06|02:03:35.388|"REDACTED"|GET|/careers/|NONE|"https"|200|"REDACTED"|443|"android-app://com.google.android.googlequicksearchbox"|"NO COOKIE"|2017-07-06 02:03:35.388|2017-07-06 02:03:35.388|7093|509|"unknown_user"|"Mozilla/5.0 (Linux; Android 7.0; SM-N920V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"|"0.0.0.0"|0|"HTTP/1.1"|"REDACTED"|0.0.0.0|"no_proxy_used"

The only difference I can find with errors like this is the trailing slash in the /careers/ field. This is based on comparing this event to others that are parsing correctly. There are other errors like this which all have the trailing slash. I am not 100% sure that this is causing the parsing issue, and have no other way to troubleshoot it. Is there another level of logging which would provide more information on the parsing error?

This field is from cs-uri-stem (renamed to url for this instance) and is parsed into the RequestUrl field.

I have tried using __replaceAll(url,\/$,"") in the parser to replace the trailing slash with nothing, but cannot seem to get it to work properly.

Here is the parser I am currently using:


 delimiter=\

 text.qualifier="
 comments.start.with=\#
 token.count=24
 token[0].name=date
 token[0].type=Date
 token[0].format=yyyy-MM-dd
 token[1].name=time
 token[1].type=Time
 token[1].format=HH:mm:ss.SSS
 token[2].name=sitename
 token[2].type=String
 token[3].name=method
 token[3].type=String
 token[4].name=url
 token[4].type=String
 token[5].name=url_query_data
 token[5].type=String
 token[6].name=protocol
 token[6].type=String
 token[7].name=server_status_code
 token[7].type=String
 token[8].name=content_path
 token[8].type=String
 token[9].name=server_port
 token[9].type=Integer
 token[10].name=referer
 token[10].type=String
 token[11].name=cookie_data
 token[11].type=String
 token[12].name=request_start_time
 token[12].type=TimeStamp
 token[12].format=yyyy-MM-dd HH:mm:ss.SSS
 token[13].name=request_end_time
 token[13].type=TimeStamp
 token[13].format=yyyy-MM-dd HH:mm:ss.SSS
 token[14].name=bytes_sent
 token[14].type=Integer
 token[15].name=bytes_received
 token[15].type=Integer
 token[16].name=user_name
 token[16].type=String
 token[17].name=browser
 token[17].type=String
 token[18].name=client_ip
 token[18].type=IPAddress
 token[19].name=protocol_substatus
 token[19].type=String
 token[20].name=protocol_version
 token[20].type=String
 token[21].name=server_hostname
 token[21].type=String
 token[22].name=server_ip
 token[22].type=IPAddress
 token[23].name=proxy
 token[23].type=String
 event.deviceProduct=__stringConstant("REDACTED")
 event.deviceVendor=__stringConstant("Microsoft")
 event.name=__stringConstant("IIS Action")
 event.deviceCustomString1Label=__stringConstant("WebSite Name")
 event.deviceCustomString2Label=__stringConstant("URL Query Date")
 event.deviceCustomString3Label=__stringConstant("Referer")
 event.deviceCustomString4Label=__stringConstant("Protocol Substatus")
 event.deviceCustomString5Label=__stringConstant("Protocol Version")
 event.deviceCustomString6Label=__stringConstant("Proxy")
 event.deviceReceiptTime=__createTimeStamp(date,time)
 event.deviceCustomString1=sitename
 event.requestMethod=method
 #event.requestUrlurl
 event.deviceCustomString2=url_query_data
 event.applicationProtocol=protocol
 event.deviceAction=server_status_code
 event.filePath=content_path
 event.destinationPort=server_port
 event.deviceCustomString3=referer
 event.requestCookies=cookie_data
 event.startTime=request_start_time
 event.endTime=request_end_time
 event.bytesOut=bytes_sent
 event.bytesIn=bytes_received
 event.sourceUserName=user_name
 event.requestClientApplication=browser
 event.sourceAddress=client_ip
 event.deviceCustomString4=protocol_substatus
 event.deviceCustomString5=protocol_version
 event.deviceHostName=server_hostname
 event.deviceAddress=server_ip
 event.deviceCustomString6=proxy

IIS fields in the log file:

date time sitename method url url_query_data protocol server_status_code content_path server_port referer cookie_data request_start_time request_end_time bytes_sent bytes_received user_name browser cliient_ip protocol_substatus protocol_version server_hostname server_ip proxy

Parses correctly:

2017-07-05|00:00:00.671|"REDACTED"|GET|/about/about.html|NONE|"http"|403|"REDACTED"|80|"NO REFERER"|"NO COOKIE"|2017-07-05 00:00:00.671|2017-07-05 00:00:00.671|1445|76|"unknown_user"|"unknown_browser"|"0.0.0.0"|4|"HTTP/1.0"|"REDACTED"|0.0.0.0|"no_proxy_used"

Can anyone assist? I know I am close to getting this to work properly, but am missing something. It is critical that I get this working.

Thanks in advance!

Labels (2)
0 Likes
1 Reply
Highlighted
Outstanding Contributor.. LakeHealthInfoS Outstanding Contributor..
Outstanding Contributor..

Re: Custom parser for IIS issues

I see what your going for here but would also submit - if all you want is a specific field or group of fields -

 

the standard multi-server IIS log reader will work then you can create a filter to parse deeper content out of the parsed fields 

 

We do this for Active Synch access on the Exchange CAS servers - for corporate mobile mail monitoring - to unauthorized devices as each user needs to request mail acces on the phoes - we also use it to enforce VMware Boxer access as the USER agent - to ensure AirWatch compliance.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.