Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..
157 views

netskope logs parser support

Jump to solution

Dear's,

Am trying to integrate netskope with custom parser need help in the same. 

logs are forwarded in syslog format ,where i can see key value pairs in logs. can we integrate key value parser for syslog connector ,if so how please guide me.

I can see "syslog subagents use key-value parsers for secondary processing." in flex document  

have attached sample log syslog: raw logs  

1 Solution

Accepted Solutions
Highlighted
Knowledge Partner
Knowledge Partner

Hi

That looks like a nice Key Pair log format.

Because the Syslog message goes straight into keyvalues and there is no real string to extract that tells you it is NetSkope (from a quick look at least) then you will need to send the events to a dedicated Syslog SmartConnector instance that is only receiving NetSkope events. This is to stop any flexconnector / parser override from breaking any other devices sending logs in.

First you will need a Syslog flexconnector that then pushes the entire message to a Key Value parser via an extraprocessor (look in the flex guide for samples) - so something like:

/flexagent/syslog/netskope.subagent.sdkrfilereader.properties

regex=(.*)

token.count=1
token[0].name=MESSAGE
token[0].type=String

event.flexString1=MESSAGE

# with an extraprocessor 

extraprocessor.count=1
extraprocessor[0].type=keyvalue
extraprocessor[0].filename=netskope/netskope
extraprocessor[0].field=event.flexString1
extraprocessor[0].flexagent=true
extraprocessor[0].clearfieldafterparsing=false

this sends sends events to

flexagent/netskope/netskope.subagent.sdkkeyvaluefilereader.properties.

then in that parser you define where you want the tokens mapped to - some will be easy such as dst_ip to destinationAddress but others may need some thinking and planning.

 

View solution in original post

7 Replies
Highlighted
Knowledge Partner
Knowledge Partner

Hi

That looks like a nice Key Pair log format.

Because the Syslog message goes straight into keyvalues and there is no real string to extract that tells you it is NetSkope (from a quick look at least) then you will need to send the events to a dedicated Syslog SmartConnector instance that is only receiving NetSkope events. This is to stop any flexconnector / parser override from breaking any other devices sending logs in.

First you will need a Syslog flexconnector that then pushes the entire message to a Key Value parser via an extraprocessor (look in the flex guide for samples) - so something like:

/flexagent/syslog/netskope.subagent.sdkrfilereader.properties

regex=(.*)

token.count=1
token[0].name=MESSAGE
token[0].type=String

event.flexString1=MESSAGE

# with an extraprocessor 

extraprocessor.count=1
extraprocessor[0].type=keyvalue
extraprocessor[0].filename=netskope/netskope
extraprocessor[0].field=event.flexString1
extraprocessor[0].flexagent=true
extraprocessor[0].clearfieldafterparsing=false

this sends sends events to

flexagent/netskope/netskope.subagent.sdkkeyvaluefilereader.properties.

then in that parser you define where you want the tokens mapped to - some will be easy such as dst_ip to destinationAddress but others may need some thinking and planning.

 

View solution in original post

Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..
thank you for valuable solution. i will test it and get back.

should i place the key value parser in below location ?

flexagent/netskope/netskope.subagent.sdkkeyvaluefilereader.properties.
0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Hi

As long as the location of the parser matches the location mentioned in the extraprocessor section then you will be fine.

extraprocessor[0].type=keyvalue
extraprocessor[0].filename=netskope/netskope
extraprocessor[0].flexagent=true

If it doesnt work, then check the agent.log errors - they will tell you where the parser is looking for the files and confirm if there are errors or it cannot find the files.

Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

HI,

 

Am using keyvalue parser as below, how do i map the events to arcsight fields. is it same as normal regex parser ?

as this is 1st time am doing the key value parser. as the flex guide do not mention how to map in the document

key.delimiter=,
key.value.delimiter==
key.regexp=([^,=]+)

additionaldata.enabled=true
event.deviceVendor=__getVendor("Netskope")
event.deviceProduct=__stringConstant("Proxy")

0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Hi
Spend some time collecting all the possible tokens (check vendor guidance) and planning what field you want them to map to.

So looking at your log you attached - the first few tokens in the message are.....

src_time="N/A",os="Windows 10",user="**PERSONAL INFORMATION REMOVED**",browser="Native",sv="unknown",device="Windows Device",telemetry_app="",url="liveupdate.symantecliveupdate.com/minitri.flg"

So you will want something like:

token.count=<however many tokens you use>

token[0].name=src_time
token[1].name=os
token[2].name=user
token[3].name=browser
token[4].name=sv
token[5].name=device
token[6].name=telemetry_app
token[7].name=url
etc

etc

# Remember they all map as Strings by default - so if there are any timestamps / IP Addresses / Integers then map them as per the guide

 

Then you need to select the right event type and map them - such as:

event.sourceUserName=user

event.requestUrl=url

event.deviceCustomString1=os

event.deviceCustomString1Label=__stringConstant("Operating System")

etc

etc

 

 

Tags (1)
0 Likes
Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

Hi 

from the 2 sample logs shared below , if i create key value parser referring to 1st logs as below   

 i start with,

token[0].name=src_country

token[1].name=policy

etc....

 

will the second log get parsed ?.... as it contains 1st token as   file_size ,  browser_version

 

 

 

<134>src_country="IN",policy="MPHASIS_UNIVERSAL_SSLD_POLICY",src_time="N/A",src_timezone="N/A",bypass_traffic="yes",bypass_reason="SSL Bypass policy matched",transaction_id=0,userip="192.168.1.104",dst_region="Tamil Nadu",dst_timezone="Asia/Kolkata",ssl_decrypt_policy="yes",page="login.windows.net",src_zipcode="N/A",user="**PERSONAL INFORMATION REMOVED**",srcip="47.247.180.99",dst_country="IN",src_location="Mundi",dstip="20.190.145.143",src_region="Madhya Pradesh",domain="login.windows.net",dst_location="Chennai",dst_zipcode="600001",timestamp=1600090430,dstport=443,access_method="Client",traffic_type="CloudApp",app="Microsoft Office 365 Suite",type="page",url="login.windows.net",organization_unit="corp.mphasis.com/MphasiS Apps/Bangalore/ORR WTC/WTC 3/All Users",userkey="**PERSONAL INFORMATION REMOVED**",ur_normalized="**PERSONAL INFORMATION REMOVED**",site="Microsoft Office 365 Suite",ccl="high",user_generated="yes",count=1,category="Application Suite",_insertion_epoch_timestamp=1600090433,dst_longitude=80.2751,dst_latitude=13.086,src_longitude=76.5,src_latitude=22.066669464111,_id="4eb677d916345286e1205e9f",cci=89,page_id=0,appcategory="Application Suite"\n

 


<134>file_size=29,browser_version="85.0.4183.102",src_time="N/A",os="Windows 10",user="**PERSONAL INFORMATION REMOVED**",browser="Chrome",sv="unknown",device="Windows Device",telemetry_app="",url="www.google.com/async/newtab_promos",web_universal_connector="yes",app_session_id=8124001089256742843,site="google",alert="yes",activity="Download",src_timezone="N/A",srcip="136.185.164.44",userip="192.168.1.11",transaction_id=-9098472700816325919,managed_app="no",page="www.google.com/async/newtab_promos",dst_region="California",src_zipcode="N/A",object_type="File",object="f.txt",hostname="LTPBAN19... 10",src_country="IN",md5="6fed308183d5dfc421602548615204af",file_type="text/plain",src_location="Rishikesh",count=1,dstip="172.217.26.196",dst_location="Mountain View",dst_zipcode="N/A",dst_country="US",timestamp=1600090542,page_site="google",access_method="Client",traffic_type="Web",browser_session_id=2441360253740552250,device_classification="managed",tss_mode="inline",request_id=1871315543449796611,policy="WEB_MPHASIS_PRIVILEGED_USER_POLICY",action="alert",type="nspolicy",organization_unit="corp.mphasis.com/MphasiS Apps/Bangalore/ORR WTC/WTC 3/All Users",nsdeviceuid="053BC132-B102-960C-D364-782501AE4B1C",managementID="",userkey="**PERSONAL INFORMATION REMOVED**",ur_normalized="**PERSONAL INFORMATION REMOVED**",ccl="unknown",acked="false",alert_type="policy",alert_name="WEB_MPHASIS_PRIVILEGED_USER_POLICY",category="MPHASIS_PRIVILEGED_USER_WEB_URL_CATEGORY",_insertion_epoch_timestamp=1600090547,dst_longitude=-122.07851409912,dst_latitude=37.405990600586,src_longitude=78.316673278809,src_latitude=30.116670608521,_id="f8fdcf1791f352e3b434f9d9",other_categories=["MPHASIS_COVID19HELPDESK_WEB_URL", "MPHASIS_EXCO_USER_WEB_URL_CATEGORY", "MPHASIS_GENERAL_WEB_URL_CATEGORY", "MPHASIS_PRIVILEGED_USER_WEB_URL_CATEGORY", "MPHASIS_UNIVERSAL_ATP_WEB_URL_CATEGORY", "PREDEFINED_WEB_URL_CATEGORY", "Search Engines", "WEB_ORACLE_SERVER_GENERAL_INTERNET_POLICY"],page_id=0,appcategory="MPHASIS_PRIVILEGED_USER_WEB_URL_CATEGORY"\n

0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Yes, that is fine - it isnt in order and doesnt need to have 100% of the tokens

That is why i say to do some planning first - extract 24 hours of events and capture all the tokens seen, fiddle around in excel etc to list them all and map to an ArcSight field. Often a vendor will have a document that lists all the possible events / tokens that will help

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.