Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE
GBing Valued Contributor.
Valued Contributor.
1291 views

Nested JSON Parsing for Symantec ATP

I am writing a flex connector for Symantec ATP. I have written most of the parser however I am a little confused on how to tokenize the nested parts of the log file. 

A single log is broken into two top level elements "emailinfo" and "incidents", my assumtion is that emailinfo is my trigger node. Is that right?

Also the next question is surounding the "filesAndLinks", given that a email can have many files and links in it how does the parser know to parse each dictionary in the array? Currently i am trying to tokenize these elements as the following but they are just coming up as blank in ArcSight ESM, what am I doing wrong here?

# nodeType
token[16].name=filesAndLinks
token[16].type=String
token[16].location=filesAndLinks/nodeType

# fileNameOrURL
token[17].name=fileNameOrURL
token[17].type=String
token[17].location=filesAndLinks/fileNameOrURL

# fileSize
token[18].name=fileSize
token[18].type=String
token[18].location=filesAndLinks/fileSize

# fileType
token[19].name=fileType
token[19].type=String
token[19].location=filesAndLinks/fileType

# md5
token[20].name=md5
token[20].type=String
token[20].location=filesAndLinks/md5

# sha256
token[21].name=sha256
token[21].type=String
token[21].location=filesAndLinks/sha256

# index
token[22].name=index
token[22].type=Integer
token[22].location=filesAndLinks/index

# parentIndex
token[23].name=parentIndex
token[23].type=Integer
token[23].location=filesAndLinks/parentIndex

# linkSource
token[24].name=linkSource
token[24].type=String
token[24].location=filesAndLinks/linkSource

Below is a sanitised example of a single log.

{
    "emailInfo":{
    "xMsgRef":"DATA",
    "longMsgRef":"DATA",
    "messageId":"DATA",
    "isOutbound":false,
    "messageSize":NUMBER,
    "mailProcessingStartTime":NUMBER,
    "subject":"DATA",
    "envFrom":"DATA",
    "headerFrom":"DATA",
    "headerReplyTo":"",
    "envTo":[
        "DATA"
    ],
    "headerTo":[
        "DATA"
    ],
    "senderIp":"IP",
    "country":"",
    "HELOString":"DATA",
    "avQuarantinePenId":"DATA",
    "filesAndLinks":[
        {
            "nodeType":"DATA",
            "fileNameOrURL":"DATA",
            "fileSize":NUMBER,
            "fileType":"DATA",
            "md5":"DATA",
            "sha256":"DATA",
            "index":NUMBER,
            "parentIndex":NUMBER,
            "linkSource":""
        },
        {
            "nodeType":"DATA",
            "fileNameOrURL":"DATA",
            "fileSize":NUMBER,
            "fileType":"DATA",
            "md5":"DATA",
            "sha256":"DATA",
            "index":NUMBER,
            "parentIndex":NUMBER,
            "linkSource":"DATA"
        }
    ]
},
"incidents":[
    {
        "xMsgRef":"DATA",
        "addressContexts":[
            {
                "name":"DATA",
                "domain":"DATA",
                "isSender":false
            }
    ],
    "severity":"DATA",
    "securityService":"DATA",
    "detectionMethod":"DATA",
    "verdict":"DATA",
    "action":"DATA",
    "reason":"DATA",
    "filesAndLinks":[
        {
             "nodeType":"DATA",
             "fileNameOrURL":"DATA",
             "fileSize":NUMBER,
             "fileType":"DATA",
             "md5":"DATA",
             "sha256":"DATA",
             "malwareName":"DATA",
             "malwareCategory":"DATA",
             "index":NUMBER,
             "parentIndex":NUMBER,
             "xMsgRef":"DATA",
             "linkSource":"DATA"

        }
    ],
    "dmasInfo":[

    ],
    "dmasDelivered":false
    }
]
}

 

Labels (1)
0 Likes
4 Replies
Highlighted
stefan.oancea Outstanding Contributor.
Outstanding Contributor.

Re: Nested JSON Parsing for Symantec ATP

Hello GBing,

I looked into your JSON and tried a few things. There are two points you need to take into account:

1. The way that you move from the "emailInfo" part into the "incidents" part during parsing

2. Addressing the JSON Array parts. For example "emailinfo->filesAndLinks" and the entire "incidents" node, both look like JSON Arrays.

I have tried out the following easy parser to see that I can retrieve the information from your JSON, and it works; I have also slightly changed some of the values of the fields just to make sure I am picking up from the right keys:

trigger.node.location = /emailInfo

token.count = 3

token[0].name = xMsgRef
token[0].type = String
token[0].location = xMsgRef

token[1].name = filesAndLinks
token[1].type = String
token[1].format = __uri(nodeType)
token[1].location = filesAndLinks

token[2].location = ../incidents
token[2].type = String
token[1].format = __uri(xMsgRef)
token[2].name = incidents

event.name = xMsgRef
event.message = filesAndLinks
event.deviceCustomString1 = incidents
event.deviceProduct = __stringConstant("JSON")
event.deviceVendor = __stringConstant("JSON")

The key take aways from my point of view are:

-> If you define "emailinfo" as the trigger node, you can still navigate into the contents of "incidents" by using the following path: "../incidents" as location. So practically you navigate back one level.

-> In the example above, the JSON Arrays will be written into the fields as a sub token map in itself. For example what I get in the message field is:

[{"nodeType":"DATA1","fileNameOrURL":"DATA","fileSize":1,"fileType":"DATA","md5":"DATA","sha256":"DATA","index":1,"parentIndex":1,"linkSource":""},{"nodeType":"DATA2","fileNameOrURL":"DATA","fileSize":1,"fileType":"DATA","md5":"DATA","sha256":"DATA","index":1,"parentIndex":1,"linkSource":"DATA"}]

protect.jpg

 

If you look into the "JSON Parsers for Complex Event Schemas" chapter from the Flex Connector Developer's Guide you will see where the above comes from :) - Flex Guide.  There is a better option of using __collection() instead of __uri() format, and for collection you can also add extra mapping files and sub-parsers. You can find something about this in the guide, but not a lot; personally I haven't tested using a sub-parser yet.

Hope this at least gives you a good starting point. If I will have more time perhaps I will also look more into this, it would be interesting to achieve in a better parsed way.

All the best,

Stefan

GBing Valued Contributor.
Valued Contributor.

Re: Nested JSON Parsing for Symantec ATP

Thank you Stefan for the in depth reply!

I didn't realise you could still acess "incidents" like that so that was a lot of help.

Funnily enough you mentioned using collections and that was the next thing I was going to give a try for "filesAnLinks".

For simplicity sake I am going to try and avoid using a sub parser unless I have too, but if I do I'll be sure to post back here and share what I learnt.

Regards,

Graeme

0 Likes
jsl@actinet.cz Respected Contributor.
Respected Contributor.

Re: Nested JSON Parsing for Symantec ATP

Hello, any news on using __collection?

I was browsing connector doc, but documentation for JSON arrays there is very poor.

Also no mention how to address collection keys/values.

Regards

Jan

0 Likes
Trusted Contributor.. migueldacruz1 Trusted Contributor..
Trusted Contributor..

Re: Nested JSON Parsing for Symantec ATP

Hello GBing,

 

could you share your ATP FlexConnector? Thank you in advance,

Miguel.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.