Big news! The community will be moving to a new platform April 21. Read more.
Big news! The community will be moving to a new platform April 21. Read more.
Commander Commander
Commander
3000 views

syslog flex connector

Jump to solution

Hello All,

I have created attached flex sdk file for huawei ids by qucky flex tool, this file is parsing all the lines while in quick flex tool. but when i keep this file in flex agent folder in syslog connector this file is not working at all.can any body help me here..?

ArcSight support is hopeless on this , when ever they see flex written any where with out reading anything they say professional service so , not hoping anything from them.

 

0 Likes
1 Solution

Accepted Solutions
Commander Commander
Commander

Hello Thank you for helping me my issue is resolved now,

then issue was in event mapping  there are proper filed formats we have to check that before mapping the filed, i was mapping wrong filed, hence it was not parsing, now it is working thank you all again.

View solution in original post

0 Likes
11 Replies
Fleet Admiral Fleet Admiral
Fleet Admiral
First thing I would check is that your custom flexagent syslog parser is being picked up and successfully parsed by the SmartConnector. Look in agent.log for the filename of your flexagent syslog parser. I also don't see any attachments on your original post.
0 Likes
Commander Commander
Commander

HI thank u for your reply..

below is my sdk file.

and error i am getting in one of the agent.log file is

Error :-

[2017-12-04 01:41:29,142][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [137] message [137]
[2017-12-04 01:41:29,149][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [5355] message [5355]
[2017-12-04 01:41:29,149][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [5355] message [5355]

 

SDK file :-

#huawei ids Configuration File
replace.defaults=true
trim.tokens=true
comments.start.with=#

#<188>2017-11-29 21:29:58 DRIR3CNI01 %%01SEC/4/ATCKDF(l): AttackType="ICMP unreachable attack", slot="5", receive interface="GigabitEthernet5/0/0 GigabitEthernet5/0/1 GigabitEthernet5/0/0 ", proto="ICMP", src="10.110.2.5:0 10.100.2.5:0 10.110.15.20:0 ", dst="172.17.2.20:0 172.16.2.20:0 10.110.2.71:0 ", begin time="2017-11-29 21:29:30", end time="2017-11-29 21:29:57", total packets="201", max speed="0".\n\n

regex=<(\\d+)>(\\d+\\-\\d+\\-\\d+) (\\d\\d:\\d\\d:\\d\\d) (DRIR3CNI01) %%(\\d+)SEC\\/(\\d+)\\/ATCKDF\\(l\\): (AttackType=)"(\\D+)", (.*) (proto=)"(\\D+)", (src=)"(.*)", (dst=)"(.*)", (begin time=)".*

token.count=16

token[0].name=Token0
token[1].name=Token1
token[2].name=Token2
token[3].name=Token3
token[4].name=Token4
token[5].name=Token5
token[6].name=Token6
token[7].name=attackname
token[8].name=Token8
token[9].name=Token9
token[10].name=protocol
token[11].name=Token11
token[12].name=srcip
token[13].name=Token13
token[14].name=dstip
token[15].name=Token15

additionaldata.enabled=true

event.deviceVendor=__stringConstant(huawei)
event.deviceProduct=__stringConstant(ids)
event.applicationProtocol=protocol
event.deviceCustomDate1=srcip
event.deviceCustomDate2=dstip
event.deviceAction=attackname

submessage.messageid.token=dstip
submessage.token=dstip
submessage.count=4

#ICMP unreachable attack
submessage[0].messageid=1
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=((\\D+))
submessage[0].pattern[0].fields=event.deviceAction
submessage[0].pattern[0].mappings=$1

#ICMP
submessage[1].messageid=2
submessage[1].pattern.count=1
submessage[1].pattern[0].regex=((\\D+))
submessage[1].pattern[0].fields=event.applicationProtocol
submessage[1].pattern[0].mappings=$1

#10.110.2.5:0 10.100.2.5:0 10.110.15.20:0
submessage[2].messageid=3
submessage[2].pattern.count=1
submessage[2].pattern[0].regex=((.*))
submessage[2].pattern[0].fields=event.deviceCustomDate1
submessage[2].pattern[0].mappings=$1

#172.17.2.20:0 172.16.2.20:0 10.110.2.71:0
submessage[3].messageid=4
submessage[3].pattern.count=1
submessage[3].pattern[0].regex=((.*))
submessage[3].pattern[0].fields=event.deviceCustomDate2
submessage[3].pattern[0].mappings=$1

0 Likes
Vice Admiral Vice Admiral
Vice Admiral

The warnings you are getting are because you haven't defined a "Default" submessage descriptior..

Modify your sdk file:

submessage.count=5

And add the following lines at the bottom:

# Default submessage descriptor

submessage[4].pattern.count=1
submessage[4].pattern[0].regex=.*
submessage[4].pattern[0].fields=event.message
submessage[4].pattern[0].extramappings=event.name\=__stringConstant("Unparsed Event")

 

 

0 Likes
Commander Commander
Commander

Hi

i have change the sdk file to below one as per your suggestion .

 

getting same errors :-

[2017-12-04 02:59:56,707][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [161] message [161]
[2017-12-04 02:59:56,708][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [48002] message [48002]
[2017-12-04 02:59:56,716][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [137] message [137]
[2017-12-04 02:59:56,716][WARN ][default.com.arcsight.agent.sdk.d.r][parseTokensNow] No empty message id submessage defined and no submessage description found for messageid [5355] message [5355]

SDK file :-

#huawei ids Configuration File
replace.defaults=true
trim.tokens=true
comments.start.with=#

#<188>2017-11-29 21:29:58 DRIR3CNI01 %%01SEC/4/ATCKDF(l): AttackType="ICMP unreachable attack", slot="5", receive interface="GigabitEthernet5/0/0 GigabitEthernet5/0/1 GigabitEthernet5/0/0 ", proto="ICMP", src="10.110.2.5:0 10.100.2.5:0 10.110.15.20:0 ", dst="172.17.2.20:0 172.16.2.20:0 10.110.2.71:0 ", begin time="2017-11-29 21:29:30", end time="2017-11-29 21:29:57", total packets="201", max speed="0".\n\n

regex=<(\\d+)>(\\d+\\-\\d+\\-\\d+) (\\d\\d:\\d\\d:\\d\\d) (DRIR3CNI01) %%(\\d+)SEC\\/(\\d+)\\/ATCKDF\\(l\\): (AttackType=)"(\\D+)", (.*) (proto=)"(\\D+)", (src=)"(.*)", (dst=)"(.*)", (begin time=)".*

token.count=16

token[0].name=Token0
token[1].name=Token1
token[2].name=Token2
token[3].name=Token3
token[4].name=Token4
token[5].name=Token5
token[6].name=Token6
token[7].name=attackname
token[8].name=Token8
token[9].name=Token9
token[10].name=protocol
token[11].name=Token11
token[12].name=srcip
token[13].name=Token13
token[14].name=dstip
token[15].name=Token15

additionaldata.enabled=true

event.deviceVendor=__stringConstant(huawei)
event.deviceProduct=__stringConstant(ids)
event.applicationProtocol=protocol
event.deviceCustomDate1=srcip
event.deviceCustomDate2=dstip
event.deviceAction=attackname

submessage.messageid.token=dstip
submessage.token=dstip
submessage.count=5

#ICMP unreachable attack
submessage[0].messageid=1
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=((\\D+))
submessage[0].pattern[0].fields=event.deviceAction
submessage[0].pattern[0].mappings=$1

#ICMP
submessage[1].messageid=2
submessage[1].pattern.count=1
submessage[1].pattern[0].regex=((\\D+))
submessage[1].pattern[0].fields=event.applicationProtocol
submessage[1].pattern[0].mappings=$1

#10.110.2.5:0 10.100.2.5:0 10.110.15.20:0
submessage[2].messageid=3
submessage[2].pattern.count=1
submessage[2].pattern[0].regex=((.*))
submessage[2].pattern[0].fields=event.deviceCustomDate1
submessage[2].pattern[0].mappings=$1

#172.17.2.20:0 172.16.2.20:0 10.110.2.71:0
submessage[3].messageid=4
submessage[3].pattern.count=1
submessage[3].pattern[0].regex=((.*))
submessage[3].pattern[0].fields=event.deviceCustomDate2
submessage[3].pattern[0].mappings=$1

# Default submessage descriptor

submessage[4].pattern.count=1
submessage[4].pattern[0].regex=.*
submessage[4].pattern[0].fields=event.message
submessage[4].pattern[0].extramappings=event.name\=__stringConstant("Unparsed Event")

 please tell me whats wronge ..

also note that the 1st sdk file i shared was working properly in qucky flex tool.

parsing all the required events.

 

 

 

0 Likes
Vice Admiral Vice Admiral
Vice Admiral

Whats the exact path and filename you are using for your parser?

i.e. <connector>/current/user/agent/flexagent/syslog/myparser.subagent.sdkrfilereader.properties

 

0 Likes
Fleet Admiral Fleet Admiral
Fleet Admiral

The SYSLOG agent may remove parts of the syslog header (facility/loglevel, date and/or hostname).

In order to figure out exactly what you need to parse, I would recommend setting up a very simple parser and then look at the name field to see exactly what the SmartConnector stripped out of the original message:

regex=(.*?AttackType=.*?)

token.count=1

token[0].name=syslog

event.name=syslog

0 Likes
Commander Commander
Commander

HI,

first of all, Thanks you , you two for helping me ..

yes i have kept sdk file in syslog, flexagent folder only .

i am unable to create simple flex file from quck flex tool as you mentioned above i am trying that will update if i am sucessful .

is it like something ,  only 1 flex files will work per syslogs connector because i have  sdk file for huawei firewall which is working fine. but my ids file is not working.

here is sample logs can some one help ..

<188>2017-12-04 17:17:01 DRIR3CNI01 %%01SEC/4/ATCKDF(l): AttackType="ICMP unreachable attack", slot="5", receive interface="GigabitEthernet5/0/0 GigabitEthernet5/0/1 GigabitEthernet5/0/0 GigabitEthernet5/0/1 ", proto="ICMP", src="10.110.1.134:0 10.110.15.21:0 10.110.15.18:0 10.100.2.70:0 10.110.1.5:0 10.110.7.50:0 10.110.15.23:0 10.110.1.130:0 10.110.15.129:0 ", dst="10.110.2.71:0 ", begin time="2017-12-04 17:16:30", end time="2017-12-04 17:16:59", total packets="61", max speed="0".\n\n  

0 Likes
Commander Commander
Commander

Also i want to know can i write multple sdk files for single device's different type of logs.

how connector catch perticluar parser for perticular log type ..?

0 Likes
Fleet Admiral Fleet Admiral
Fleet Admiral
Yes, you can have multiple *.subagent.sdkrfilereader.properties files in agent_dir/current/user/agent/flexagent/syslog. It will try to match logs based on the "regex=" line. Create a new file in that directory using the example I provided prior, then find the logs as they come into the ESM and look at the name field to see what your custom parser needs to be looking for.
0 Likes
Commander Commander
Commander

Hello Thank you for helping me my issue is resolved now,

then issue was in event mapping  there are proper filed formats we have to check that before mapping the filed, i was mapping wrong filed, hence it was not parsing, now it is working thank you all again.

View solution in original post

0 Likes
Hi, what was the issue with the data mapping? Did you not include the token[0].type=String in all the tokens? was that the only issue?
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.