syslog flex connector

[nilkanthp]

Hello All,

I have created attached flex sdk file for huawei ids by qucky flex tool, this file is parsing all the lines while in quick flex tool. but when i keep this file in flex agent folder in syslog connector this file is not working at all.can any body help me here..?

ArcSight support is hopeless on this , when ever they see flex written any where with out reading anything they say professional service so , not hoping anything from them.

[Shaun]

First thing I would check is that your custom flexagent syslog parser is being picked up and successfully parsed by the SmartConnector. Look in agent.log for the filename of your flexagent syslog parser. I also don't see any attachments on your original post.

[nilkanthp]

HI thank u for your reply..

below is my sdk file.

and error i am getting in one of the agent.log file is

Error :-

[2017-12-04 01:41:29,142][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [137] message [137]
[2017-12-04 01:41:29,149][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [5355] message [5355]
[2017-12-04 01:41:29,149][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [5355] message [5355]

SDK file :-

#huawei ids Configuration File
replace.defaults=true
trim.tokens=true
comments.start.with=#

#<188>2017-11-29 21:29:58 DRIR3CNI01 %%01SEC/4/ATCKDF(l): AttackType="ICMP unreachable attack", slot="5", receive interface="GigabitEthernet5/0/0 GigabitEthernet5/0/1 GigabitEthernet5/0/0 ", proto="ICMP", src="10.110.2.5:0 10.100.2.5:0 10.110.15.20:0 ", dst="172.17.2.20:0 172.16.2.20:0 10.110.2.71:0 ", begin time="2017-11-29 21:29:30", end time="2017-11-29 21:29:57", total packets="201", max speed="0".\n\n

regex=<(\\d+)>(\\d+\\-\\d+\\-\\d+) (\\d\\d:\\d\\d:\\d\\d) (DRIR3CNI01) %%(\\d+)SEC\\/(\\d+)\\/ATCKDF\\(l\\): (AttackType=)"(\\D+)", (.*) (proto=)"(\\D+)", (src=)"(.*)", (dst=)"(.*)", (begin time=)".*

token.count=16

token[0].name=Token0
token[1].name=Token1
token[2].name=Token2
token[3].name=Token3
token[4].name=Token4
token[5].name=Token5
token[6].name=Token6
token[7].name=attackname
token[8].name=Token8
token[9].name=Token9
token[10].name=protocol
token[11].name=Token11
token[12].name=srcip
token[13].name=Token13
token[14].name=dstip
token[15].name=Token15

additionaldata.enabled=true

event.deviceVendor=__stringConstant(huawei)
event.deviceProduct=__stringConstant(ids)
event.applicationProtocol=protocol
event.deviceCustomDate1=srcip
event.deviceCustomDate2=dstip
event.deviceAction=attackname

submessage.messageid.token=dstip
submessage.token=dstip
submessage.count=4

#ICMP unreachable attack
submessage[0].messageid=1
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=((\\D+))
submessage[0].pattern[0].fields=event.deviceAction
submessage[0].pattern[0].mappings=$1

#ICMP
submessage[1].messageid=2
submessage[1].pattern.count=1
submessage[1].pattern[0].regex=((\\D+))
submessage[1].pattern[0].fields=event.applicationProtocol
submessage[1].pattern[0].mappings=$1

#10.110.2.5:0 10.100.2.5:0 10.110.15.20:0
submessage[2].messageid=3
submessage[2].pattern.count=1
submessage[2].pattern[0].regex=((.*))
submessage[2].pattern[0].fields=event.deviceCustomDate1
submessage[2].pattern[0].mappings=$1

#172.17.2.20:0 172.16.2.20:0 10.110.2.71:0
submessage[3].messageid=4
submessage[3].pattern.count=1
submessage[3].pattern[0].regex=((.*))
submessage[3].pattern[0].fields=event.deviceCustomDate2
submessage[3].pattern[0].mappings=$1

[Richie D]

The warnings you are getting are because you haven't defined a "Default" submessage descriptior..

Modify your sdk file:

submessage.count=5

And add the following lines at the bottom:

# Default submessage descriptor

submessage[4].pattern.count=1
submessage[4].pattern[0].regex=.*
submessage[4].pattern[0].fields=event.message
submessage[4].pattern[0].extramappings=event.name\=__stringConstant("Unparsed Event")

[nilkanthp]

Hi

i have change the sdk file to below one as per your suggestion .

getting same errors :-

[2017-12-04 02:59:56,707][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [161] message [161]
[2017-12-04 02:59:56,708][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [48002] message [48002]
[2017-12-04 02:59:56,716][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [137] message [137]
[2017-12-04 02:59:56,716][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [5355] message [5355]

SDK file :-

#huawei ids Configuration File
replace.defaults=true
trim.tokens=true
comments.start.with=#

#<188>2017-11-29 21:29:58 DRIR3CNI01 %%01SEC/4/ATCKDF(l): AttackType="ICMP unreachable attack", slot="5", receive interface="GigabitEthernet5/0/0 GigabitEthernet5/0/1 GigabitEthernet5/0/0 ", proto="ICMP", src="10.110.2.5:0 10.100.2.5:0 10.110.15.20:0 ", dst="172.17.2.20:0 172.16.2.20:0 10.110.2.71:0 ", begin time="2017-11-29 21:29:30", end time="2017-11-29 21:29:57", total packets="201", max speed="0".\n\n

regex=<(\\d+)>(\\d+\\-\\d+\\-\\d+) (\\d\\d:\\d\\d:\\d\\d) (DRIR3CNI01) %%(\\d+)SEC\\/(\\d+)\\/ATCKDF\\(l\\): (AttackType=)"(\\D+)", (.*) (proto=)"(\\D+)", (src=)"(.*)", (dst=)"(.*)", (begin time=)".*

token.count=16

token[0].name=Token0
token[1].name=Token1
token[2].name=Token2
token[3].name=Token3
token[4].name=Token4
token[5].name=Token5
token[6].name=Token6
token[7].name=attackname
token[8].name=Token8
token[9].name=Token9
token[10].name=protocol
token[11].name=Token11
token[12].name=srcip
token[13].name=Token13
token[14].name=dstip
token[15].name=Token15

additionaldata.enabled=true

event.deviceVendor=__stringConstant(huawei)
event.deviceProduct=__stringConstant(ids)
event.applicationProtocol=protocol
event.deviceCustomDate1=srcip
event.deviceCustomDate2=dstip
event.deviceAction=attackname

submessage.messageid.token=dstip
submessage.token=dstip
submessage.count=5

#ICMP unreachable attack
submessage[0].messageid=1
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=((\\D+))
submessage[0].pattern[0].fields=event.deviceAction
submessage[0].pattern[0].mappings=$1

#ICMP
submessage[1].messageid=2
submessage[1].pattern.count=1
submessage[1].pattern[0].regex=((\\D+))
submessage[1].pattern[0].fields=event.applicationProtocol
submessage[1].pattern[0].mappings=$1

#10.110.2.5:0 10.100.2.5:0 10.110.15.20:0
submessage[2].messageid=3
submessage[2].pattern.count=1
submessage[2].pattern[0].regex=((.*))
submessage[2].pattern[0].fields=event.deviceCustomDate1
submessage[2].pattern[0].mappings=$1

#172.17.2.20:0 172.16.2.20:0 10.110.2.71:0
submessage[3].messageid=4
submessage[3].pattern.count=1
submessage[3].pattern[0].regex=((.*))
submessage[3].pattern[0].fields=event.deviceCustomDate2
submessage[3].pattern[0].mappings=$1

# Default submessage descriptor

submessage[4].pattern.count=1
submessage[4].pattern[0].regex=.*
submessage[4].pattern[0].fields=event.message
submessage[4].pattern[0].extramappings=event.name\=__stringConstant("Unparsed Event")

 please tell me whats wronge ..

also note that the 1st sdk file i shared was working properly in qucky flex tool.

parsing all the required events.

[Richie D]

Whats the exact path and filename you are using for your parser?

i.e. <connector>/current/user/agent/flexagent/syslog/myparser.subagent.sdkrfilereader.properties

[Shaun]

The SYSLOG agent may remove parts of the syslog header (facility/loglevel, date and/or hostname).

In order to figure out exactly what you need to parse, I would recommend setting up a very simple parser and then look at the name field to see exactly what the SmartConnector stripped out of the original message:

regex=(.*?AttackType=.*?)

token.count=1

token[0].name=syslog

event.name=syslog

[nilkanthp]

HI,

first of all, Thanks you , you two for helping me ..

yes i have kept sdk file in syslog, flexagent folder only .

i am unable to create simple flex file from quck flex tool as you mentioned above i am trying that will update if i am sucessful .

is it like something ,  only 1 flex files will work per syslogs connector because i have  sdk file for huawei firewall which is working fine. but my ids file is not working.

here is sample logs can some one help ..

<188>2017-12-04 17:17:01 DRIR3CNI01 %%01SEC/4/ATCKDF(l): AttackType="ICMP unreachable attack", slot="5", receive interface="GigabitEthernet5/0/0 GigabitEthernet5/0/1 GigabitEthernet5/0/0 GigabitEthernet5/0/1 ", proto="ICMP", src="10.110.1.134:0 10.110.15.21:0 10.110.15.18:0 10.100.2.70:0 10.110.1.5:0 10.110.7.50:0 10.110.15.23:0 10.110.1.130:0 10.110.15.129:0 ", dst="10.110.2.71:0 ", begin time="2017-12-04 17:16:30", end time="2017-12-04 17:16:59", total packets="61", max speed="0".\n\n  

[nilkanthp]

Also i want to know can i write multple sdk files for single device's different type of logs.

how connector catch perticluar parser for perticular log type ..?

[Shaun]

Yes, you can have multiple *.subagent.sdkrfilereader.properties files in agent_dir/current/user/agent/flexagent/syslog. It will try to match logs based on the "regex=" line. Create a new file in that directory using the example I provided prior, then find the logs as they come into the ESM and look at the name field to see what your custom parser needs to be looking for.

[nilkanthp · Accepted solution]

Hello Thank you for helping me my issue is resolved now,

then issue was in event mapping  there are proper filed formats we have to check that before mapping the filed, i was mapping wrong filed, hence it was not parsing, now it is working thank you all again.

[marypriscilla24]

Hi, what was the issue with the data mapping? Did you not include the token[0].type=String in all the tokens? was that the only issue?