Highlighted
saurabh.rohra Absent Member.
Absent Member.
502 views

How to parse further, Timebased DB FlexConnectors field using extraprocessor regex and using multiline?

Hello Everyone!

I have create a TimeBased DB FlexConnector in which I need to parse one column further. To do this job  I am using extraprocessor method and pointed it to a new regex properties file under the flexagent folder. I'm using multiline parsing in regex. My issue is when I run the connector I am not able to see any errors but I'm not able to get any events from the database rather than the native arcsight events.

Please find below my both properties file.

flexagent/MyInx/A.sdktbdatabase.properties

version.order=1

#version.query=select sysdate from Dual

query=SELECT A.AUDIT_ID,B.AUDIT_ITEM_ID,A.ACTION_CODE,A.DATE_TIME,A.USER_ID,C.FIRST_NAME||' '|| C.LAST_NAME AS USER_NAME, C.COMPANY_ABBV_NAME,B.CONTEXT FROM COL.GTP_AUDIT A INNER JOIN COL.GTP_AUDIT_ITEM B ON A.AUDIT_ID = B.AUDIT_ID INNER JOIN COL.GTP_USER C ON A.USER_ID = C.USER_ID WHERE ACTION_CODE IN ('LOGIN','LOGOUT') AND B.TYPE  IN ('02','03') AND DATE_TIME >= CAST(? AS TIMESTAMP)

timestamp.field=DATE_TIME

uniqueid.fields=AUDIT_ID,AUDIT_ITEM_ID,DATE_TIME,USER_ID

#Event Mapping

event.endTime=DATE_TIME

event.deviceCustomString1Label=__stringConstant("USERID")

event.deviceCustomString1=USER_ID

event.deviceCustomString2Label=__stringConstant("USERNAME")

event.deviceCustomString2=USER_NAME

event.deviceCustomString3Label=__stringConstant("AUDIT ID")

event.deviceCustomString3=AUDIT_ID

event.deviceCustomString4Label=__stringConstant("ACTION")

event.deviceCustomString4=ACTION_CODE

event.deviceCustomString5Label=__stringConstant("COMPANY_ABBV_NAME")

event.deviceCustomString5=COMPANY_ABBV_NAME

event.deviceCustomString6Label=__stringConstant("AUDIT_ITEM_ID")

event.deviceCustomString6=AUDIT_ITEM_ID

event.message=CONTEXT

event.deviceProduct=__stringConstant("A")

event.deviceVendor=__stringConstant("A")

#Extra-processing for extraction of IP address from CONTEXT field.

extraprocessor.count=1

extraprocessor[0].type=regex

extraprocessor[0].filename=MyInx/A_ad

extraprocessor[0].field=event.message

extraprocessor[0].flexagent=true

extraprocessor[0].clearfieldafterparsing=false

flexagent/MyInx/A_ad.sdktbdatabase.properties

# FlexAgent Regex Configuration File

do.unparsed.events=true

source.field=event.message


multiline.delimiter=|

multiline.start.regex=LOGIN_ID: .*

regex=(.*?|)\\sIP_ADDRESS:\\s(\\d+\\.\\d+\\.\\d+\\.\\d+)

token.count=2

token[0].name=Message

token[0].type=String

token[1].name=msg

token[1].type=IPAddress

event.name=Message

event.sourceAddress=msg

Labels (2)
0 Likes
Reply
1 Reply
maystrovichva Super Contributor.
Super Contributor.

Re: How to parse further, Timebased DB FlexConnectors field using extraprocessor regex and using multiline?

Maybe it is not actual yet, but though. I saw some problems with your sdktbdatabase.properties file.

    1. lastdate.query is absent. sdktbdatabase.properties file must have lastdate.query property. For example,

lastdate.query=SELECT cast(max(DATE_TIME) as date) from COL.GTP_AUDIT

.     Cast is needed if your DATE_TIME is not a date type in database.

    2. Query is wrong. You do not need cast placeholder ?.

query=SELECT \

A.AUDIT_ID AS AUDIT_ID, \

B.AUDIT_ITEM_ID AS AUDIT_ITEM_ID, \

A.ACTION_CODE AS ACTION_CODE, \

A.DATE_TIME AS DATE_TIME, \

A.USER_ID AS USER_ID, \

C.FIRST_NAME||' '|| C.LAST_NAME AS USER_NAME, \

C.COMPANY_ABBV_NAME AS COMPANY_ABBV_NAME, \

B.CONTEXT AS CONTEXT \

FROM COL.GTP_AUDIT A INNER JOIN COL.GTP_AUDIT_ITEM B ON A.AUDIT_ID = B.AUDIT_ID INNER JOIN COL.GTP_USER C ON A.USER_ID = C.USER_ID \

WHERE A.ACTION_CODE IN ('LOGIN','LOGOUT') AND B.TYPE  IN ('02','03') AND CAST(DATE_TIME AS DATE) >= ?

    

If you have casted DATE_TIME in lastdate.query, you must cast DATE_TIME in query too.

    3. Your A_ad property file have wrong name. As extraprocessor[0].type=regex, so A_ad must have a A_ad.sdkrfilereader.properties name.

    4. I don't know if your event.message is really multiline. If it is and you really need to use multiline feature then you must define not only multiline.start.regex option, but multiline.end.regex too.

  5. Try to get events from database without using extraprocessor. If success, try to use extraprocessor.

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.