Valued Contributor.. jonathan@datasi Valued Contributor..
Valued Contributor..
612 views

All event in flex deployed are populated only in the "Name Field". How can I extract values from it?

Hello Guys, 

Please guys, I have build a flex for an application log. After it was deployed, all events are mapped to "Name Field" only. But i want to have the values in the event mapped to other fields.

Please how can you use variables to extract the below strings into other fields. Vakues such as  (Failure, User, transaction type and Logging time)

"ID:unknown/SERVICE","RealUser":"ID:unknown/SERVICE","LoggingTime":"2018-02-15 16:31:39.955","SessionId":1500482383489,"TransactionId":1692558440,"TransactionType":"ActivateUser","Status":"FAILURE","RecordVersion":1,"SupplementaryData":{"Profile":" Subscriber Incomplete KYC Profile","ErrorMessage":"TOO_MANY_CONSECUTIVE_DIGITS_IN_PINCODE"}}

0 Likes
10 Replies
Honored Contributor.. jorgeoa Honored Contributor..
Honored Contributor..

Re: All event in flex deployed are populated only in the "Name Field". How can I extract v

Hello Jonathan,

I would use a Regex flexconnector. The log seems to be well structured so it should be easy to create de regex.

You can use the flexconnector regex tool:

%flexconnector_home%/current/bin/arcsight regex

0 Likes
Valued Contributor.. jonathan@datasi Valued Contributor..
Valued Contributor..

Re: All event in flex deployed are populated only in the "Name Field". How can I extract v

Yes, the flex has been developed and deployed. But the issue is that the line of events only populates in the "Name Field" in the Active Channels.

How do I populate it to be mapped to different fields?

0 Likes
Honored Contributor.. jorgeoa Honored Contributor..
Honored Contributor..

Re: All event in flex deployed are populated only in the "Name Field". How can I extract v

I'm sorry but without more detailed information I can't help you. Could you share the flexparser and any example event?

0 Likes
Valued Contributor.. jonathan@datasi Valued Contributor..
Valued Contributor..

Re: All event in flex deployed are populated only in the "Name Field". How can I extract v

see below the sample log file

Jan 18 15:20:06 asss00 {""InitiatingUser"":""ID:5207140/MM"",""RealUser"":""ID:5207140/MM"",""LoggingTime"":""2018-01-18 16:15:15.741"",""SessionId"":1500443854901,""TransactionId"":1572537673,""TransactionType"":""InitiateTransfer"",""Status"":""SUCCESS"",""RecordVersion"":1,""SupplementaryData"":{""UserAccountHolderIdentity"":""ID:22996405856/MSISDN"",""InitiatingAccountHolderIdentity"":{""Id"":""22967379796"",""IdentityType"":""MSISDN""}}}
Jan 18 15:20:06 asss00 {""InitiatingUser"":""ID:namuser/ADMIN"",""RealUser"":""ID:namuser/ADMIN"",""LoggingTime"":""2018-01-18 16:15:15.736"",""SessionId"":1500441800312,""TransactionId"":1572537672,""TransactionType"":""GetAccountHolderInfo"",""Status"":""SUCCESS"",""RecordVersion"":1,""SupplementaryData"":{""AccountHolderID"":""ID:2997255823/MSISDN""}}
Jan 18 15:20:06 asss00 {""InitiatingUser"":""ID:4542109/MM"",""RealUser"":""ID:4542109/MM"",""LoggingTime"":""2018-01-18 16:15:15.501"",""SessionId"":1500443854894,""TransactionId"":1572537659,""TransactionType"":""ExternalBalanceAdjust"",""Status"":""SUCCESS"",""RecordVersion"":1,""SupplementaryData"":{""Msisdn"":""2962604130"",""AccountType"":""MOBILE_MONEY"",""BalanceBefore"":""20100"",""BalanceAfter"":""100""}}
Jan 18 15:20:06 asss00 {""InitiatingUser"":""ID:namuser/ADMIN"",""RealUser"":""ID:namuser/ADMIN"",""LoggingTime"":""2018-01-18 16:15:15.699"",""SessionId"":1500441800312,""TransactionId"":1572537670,""TransactionType"":""GetAccountHolderInfo"",""Status"":""SUCCESS"",""RecordVersion"":1,""SupplementaryData"":{""AccountHolderID"":""ID:22962808402/MSISDN""}}
Jan 18 15:20:06 asss00 {""InitiatingUser"":""ID:namuser/ADMIN"",""RealUser"":""ID:namuser/ADMIN"",""LoggingTime"":""2018-01-18 16:15:14.471"",""SessionId"":1500441800312,""TransactionId"":1572537612,""TransactionType"":""GetAccountHolderInfo"",""Status"":""SUCCESS"",""RecordVersion"":1,""SupplementaryData"":{""AccountHolderID"":""ID:22997754018/MSISDN""}}
Jan 18 15:20:06 asss00 {""InitiatingUser"":""ID:namuser/ADMIN"",""RealUser"":""ID:namuser/ADMIN"",""LoggingTime"":""2018-01-18 16:15:14.477"",""SessionId"":1500441800312,""TransactionId"":1572537613,""TransactionType"":""GetAccountHolderInfo"",""Status"":""SUCCESS"",""RecordVersion"":1,""SupplementaryData"":{""AccountHolderID"":""ID:22961140292/MSISDN""}}
Jan 18 15:20:06 asss00 {""InitiatingUser"":""ID:5946850/MM"",""RealUser"":""ID:5946850/MM"",""LoggingTime"":""2018-01-18 16:15:14.859"",""SessionId"":1500443854882,""TransactionId"":1572537632,""TransactionType"":""CashIn"",""Status"":""SUCCESS"",""RecordVersion"":1,""SupplementaryData"":{}}
Jan 18 15:20:06 asss00 {""InitiatingUser"":""ID:5395670/MM"",""RealUser"":""ID:5395670/MM"",""LoggingTime"":""2018-01-18 16:15:14.044"",""SessionId"":1500443854872,""TransactionId"":1572537595,""TransactionType"":""Transfer"",""Status"":""SUCCESS"",""RecordVersion"":1,""SupplementaryData"":{""Amount"":{""Amount"":20000,""Currency"":""XOF""},""SendingFri"":{""Fri"":""FRI:22962846402/MSISDN""},""SenderNote"":""0"",""ReceivingFri"":{""Fri"":""FRI:22967990153/MSISDN""},""ReceiverMessage"":""0""}}
Jan 18 15:20:06 asss00 {""InitiatingUser"":""ID:namuser/ADMIN"",""RealUser"":""ID:namuser/ADMIN"",""LoggingTime"":""2018-01-18 16:15:14.396"",""SessionId"":1500441800312,""TransactionId"":1572537610,""TransactionType"":""GetAccountHolderInfo"",""Status"":""SUCCESS"",""RecordVersion"":1,""SupplementaryData"":{""AccountHolderID"":""ID:22997005477/MSISDN""}}

Regex: regex=\\{\\""[^\\""]+\\""\\\:\\""([^\\""]+)\\""\\,\\""[^\\""]+\\""\\\:\\""([^\\""]+)\\""\\,\\""[^\\""]+\\""\\\:\\""([^\\""]+)\\""\\,\\""[^\\,]+\\""\\\:([^\\,]+)\\,\\""[^\\""]+\\""\\\:([^\\,]+)\\,\\""[^\\""]+\\""\\\:\\""([^\\""]+)\\""\\,\\""[^\\,]+\\""\\\:\\""([^\\""]+)\\""\\,\\""[^\\""]+\\""\\\:(\\d+),\\""[^\\""]+\\""\\\:(.*)

I used the above regex for the sample logs. I hope this helps.

Thanks

0 Likes
Honored Contributor.. jorgeoa Honored Contributor..
Honored Contributor..

Re: All event in flex deployed are populated only in the "Name Field". How can I extract v

Hi Jonathan,

Your regex doesn't match the entire log line (the text at the beginning "Jan 18 15:20:06 asss00 " is missing in the regex) and uses a lot of unnecessary escape characters.

The log you pasted now has two double quote characters to delimit the fields, but in your first message it uses only one.

I attach a file with a sample log as I think is the right syntax (without those two double quote) and the regex parser created with the Quick Flex tool.

Sample log line I've used:

Jan 18 15:20:06 asss00 {"InitiatingUser":"ID:5395670/MM","RealUser":"ID:5395670/MM","LoggingTime":"2018-01-18 16:15:14.044","SessionId":1500443854872,"TransactionId":1572537595,"TransactionType":"Transfer","Status":"SUCCESS","RecordVersion":1,"SupplementaryData":{"Amount":{"Amount":20000,"Currency":"XOF"},"SendingFri":{"Fri":"FRI:22962846402/MSISDN"},"SenderNote":"0","ReceivingFri":{"Fri":"FRI:22967990153/MSISDN"},"ReceiverMessage":"0"}}

I hope this helps you

Regards

0 Likes
Valued Contributor.. jonathan@datasi Valued Contributor..
Valued Contributor..

Re: All event in flex deployed are populated only in the "Name Field". How can I extract v

Many thanks Jorgeoa.

I have simulated the regex and the sample logs you sent in my VM to see how it populates in arcsight console. the production logs are streamed via syslog to the connector server.

So, I'm using a syslog connector and I used a syslog simulator to stream the logs to the console. But for some reasons, it still maps all the events to the "Name"field in the ArcSight console.  

Thanks Once again.

0 Likes
Highlighted
Honored Contributor.. jorgeoa Honored Contributor..
Honored Contributor..

Re: All event in flex deployed are populated only in the "Name Field". How can I extract v

Hi,

If your are using a syslog server you need to define the parser as a override. I haven't done it, check out the FlexConnector developer guide (https://community.softwaregrp.com/t5/ArcSight-Connectors/HPE-ArcSight-FlexConnector-Developer-s-Guide/ta-p/1584874?nm) and other posts in the forum (https://community.softwaregrp.com/t5/forums/searchpage/tab/message?filter=location&q=syslog%20override&location=category:arcsight&collapse_discussion=true)

Maybe you have to modify the regular expression and remove the first part ([^{]+)

Update:

Here is a very useful document about syslog parser override: https://community.softwaregrp.com/t5/ArcSight-Discussions/Reparsing-Data-in-ArcSight-Fields/m-p/1587587/thread-id/1593

Regards

0 Likes
Valued Contributor.. jonathan@datasi Valued Contributor..
Valued Contributor..

Re: All event in flex deployed are populated only in the "Name Field".

Hi Jorgeoa

 

Many thanks for the update. I will test it with this guide and get back to you about the outcome.

 

Regards

0 Likes
Acclaimed Contributor.. Shaun Acclaimed Contributor..
Acclaimed Contributor..

Re: All event in flex deployed are populated only in the "Name Field". How can I extract v

Looks like JSON. The flexagent or flexconnector should have the field mappings in it. If you are getting all the data into the name field, then your parser isn't working correctly.
0 Likes
Valued Contributor.. jonathan@datasi Valued Contributor..
Valued Contributor..

Re: All event in flex deployed are populated only in the "Name Field". How can I extract v

Yes.  The Fields were mapped accordingly. I would not know what or where I missed anything.

 

Thanks

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.