Highlighted
P1shta
Visitor.
91 views

ExtraProcessors with multiplicated strings in span of one second

Hello all,

I've got realtime File flex connector.

It is regex type of parser, and I use 6 regex extraprocessor sub-parsers.

The issue is (or at least it looks like this) that every time there come more than one log line in interval of one second, the extraprocessor string in output (the part of input after GROUPADD/GROUPRECALL/USERDEACTIVATE/...) stays somehow the very same despite the fact, that these extraprocessed submessages don't even have the same structure and fields. It should be clearly visible on the screen capture I attached here. 

Comment to that picture: Source on the left side, Output copied from ESM on the right side. Yellow rectangle in source shows logs that happened in span of one second. You can see that they are of different type of extraprocessors, but if you look on the output, there is "dst_priv_A" in all lines instead of all "dst_priv_A" to "dst_priv_J" as it should be according to source file on the left side.

Similarly the red rectangle where they should be "dst_priv_D" to "dst_priv_O" and there are only "dst_priv_D" on all lines.

Even worse on the last two lines where are USERDEACTIVATE and USERDELETE extraprocessor tokens, which both do not have Destination User Privileges (dst_priv_X) field/parameter (and in the source they really don't) they suddenly have it in output in ESM.

I am totally lost and I am kindly asking for help here.

INPUT EXAMPLE:

src1|2019-11-04 14:06:55.522|user_anonym|GROUPADD|dst_user05261|dst_priv_A|
src1|2019-11-04 14:06:55.522|user_anonym|GROUPADD|dst_user05261|dst_priv_B|
src1|2019-11-04 14:06:55.522|user_anonym|GROUPADD|dst_user05261|dst_priv_C|

src2|2019-11-04 14:10:44.616|user_anonym|GROUPRECALL|dst_user01866|dst_priv_D|
src2|2019-11-04 14:10:44.814|user_anonym|GROUPRECALL|dst_user01866|dst_priv_O|
src2|2019-11-04 14:10:44.844|user_anonym|USERDEACTIVATE|dst_user01866|04.11.2019|
src2|2019-11-04 14:10:44.866|user_anonym|USERDELETE|dst_user01866|04.11.2019|

OUTPUT EXAMPLE:

Source Service Name,End Time,Source User Name,Name,Destination User ID,Destination User Privileges,Device Custom Date1,Device Custom Date1 Label,Device Custom Date2,Device Custom Date2 Label,Device Custom String1,Device Custom String1 Label,Device Custom String2,Device Custom String2 Label,Message
src1,Nov 04 2019 14:06:55,user_anonym,GROUPADD,dst_user05261,dst_priv_A,,,,,,,,,Access privileges added
src1,Nov 04 2019 14:06:55,user_anonym,GROUPADD,dst_user05261,dst_priv_A,,,,,,,,,Access privileges added
src1,Nov 04 2019 14:06:55,user_anonym,GROUPADD,dst_user05261,dst_priv_A,,,,,,,,,Access privileges added

src2,Nov 04 2019 14:10:44,user_anonym,GROUPRECALL,dst_user01866,dst_priv_D,,,,,,,,,Access privileges recalled
src2,Nov 04 2019 14:10:44,user_anonym,GROUPRECALL,dst_user01866,dst_priv_D,,,,,,,,,Access privileges recalled
src2,Nov 04 2019 14:10:44,user_anonym,USERDEACTIVATE,dst_user01866,dst_priv_D,Nov 04 2019 00:00:00,Date of user deactivation,,,,,,,User deactivated
src2,Nov 04 2019 14:10:44,user_anonym,USERDELETE,dst_user01866,dst_priv_D,Nov 04 2019 00:00:00,Date of user deletion,,,,,,,User deleted

SDKFILEREADER.PROPERTIES:

do.unparsed.events=false
token.count=5

token[0].name=SRC_Service_Name
token[0].type=Stringtoken[1].name=End_Time

token[1].type=TimeStamp
token[1].format=yyyy-MM-dd HH:mm:ss.SSS

token[2].name=SRC_User_Name
token[2].type=String

token[3].name=Name
token[3].type=String

token[4].name=Others
token[4].type=String

event.sourceServiceName=SRC_Service_Name
event.endTime=End_Time
event.sourceUserName=SRC_User_Name
event.name=Name
event.rawEvent=Others
event.deviceVendor=__stringConstant("ABC")
event.deviceProduct=__stringConstant("MM ABC DEFG")

extraprocessor.count=6

extraprocessor[0].type=regex
extraprocessor[0].filename=extraUSERACTIVATE
extraprocessor[0].field=event.rawEvent
extraprocessor[0].conditionfield=event.name
extraprocessor[0].conditiontype=regex
extraprocessor[0].conditionvalues=USERACTIVATE
extraprocessor[0].clearfieldafterparsing=true
extraprocessor[0].flexagent=true

and similarily with other 5 extraprocessors....

EXTRAPROCESSOR OF GROUPADD:

do.unparsed.events=false
token.count=2
regex=(\\S+)[|](\\S+)
token[0].name=DST_User_ID
token[0].type=String
token[1].name=Group_ID
token[1].type=String
event.destinationUserId=DST_User_ID
event.destinationUserPrivileges=Group_ID

EXTRAPROCESSOR OF GROUPRECALL:

do.unparsed.events=false
token.count=2
regex=(\\S+)[|](\\S+)
token[0].name=DST_User_ID
token[0].type=String
token[1].name=Group_ID
token[1].type=String
event.destinationUserId=DST_User_ID
event.destinationUserPrivileges=Group_ID

EXTRAPROCESSOR OF USERDEACTIVATE:

do.unparsed.events=false
token.count=2
regex=(\\S+)[|]([0-9.]+)
token[0].name=DST_User_ID
token[0].type=String
token[1].name=Date_Of_Deactivation
token[1].type=TimeStamp
token[1].format=dd.MM.yyyy
event.destinationUserId=DST_User_ID
event.deviceCustomDate1=Date_Of_Deactivation
event.deviceCustomDate1Label=__stringConstant("Date of user deactivation")

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.