Big news! The community will be moving to a new platform April 21. Read more.
Big news! The community will be moving to a new platform April 21. Read more.
Cadet 3rd Class
Cadet 3rd Class
791 views

How to parse variable JSON fields for the same event type?

I am trying to parse Azure Resource Manager logs using the JSON FlexFolder Connector and am having trouble with how to configure the parsing file when there are fields that do not appear in all events.  There are two event examples below, jobs/write and parsexml/action.  

The jobs/write event has additional fields inside the claims field like "ipaddr" and "name"  and has the properties field, none of which exist in the parsexml/action event.  The documentation for developing a custom parser only shows how to do a 1-for-1 parsing of tokens for JSON and doesn't discuss how to handle fields that do not exist in every event.  Any help would be most appreciated.

 

jobs/write event below

{
"authorization": {
"action": "Microsoft.Scheduler/jobCollections/jobs/write",
"scope": "/subscriptions/9446f748-d132-4d91-8376-ab3f4b0f9r04/resourceGroups/faketaco-testing/providers/Microsoft.Scheduler/jobCollections/rr-testing/jobs/alerting-processor-run"
},
"caller": "fake@fake.com",
"channels": "Operation",
"claims": {
"aud": "https://management.core.windows.net/",
"iss": "https://sts.windows.net/89accea6-2b47-43a4-b381-b0410b3778dd/",
"iat": "1519848719",
"nbf": "1519848719",
"exp": "1519852619",
"http://schemas.microsoft.com/claims/authnclassreference": "1",
"aio": "ATQAy/8GAAAA65F2vbMfyOZ1vIj/0H2DSYBJUV/9gUC+9c24aSXqwwiU+yX0DmsOr/5KPoMC7WdP",
"http://schemas.microsoft.com/claims/authnmethodsreferences": "pwd,mfa",
"appid": "0fe60f94-01c6-4300-ac64-2badb735d457",
"appidacr": "1",
"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname": "Fake",
"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname": "Fake",
"groups": "e12c410f-86fb-463f-ae57-1eb9554a21f7,fe80c04c-2fb7-46ac-be15-254c36194a4c,daaae8e0-e512-43f5-9840-af8011b5d1f7,48526705-8db3-4b05-9af2-a2a8d95d5ceb,58f92203-626a-4b66-8329-2ea4a9da567d",
"ipaddr": "53.153.83.212",
"name": "Fake Fake",
"http://schemas.microsoft.com/identity/claims/objectidentifier": "1ac647bc-a4c2-4026-853b-af4049127046",
"onprem_sid": "S-1-5-21-1900692997-427066686-65291307-2231",
"puid": "100300008A77D6C4",
"http://schemas.microsoft.com/identity/claims/scope": "user_impersonation",
"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "MAsIw1mF-mXxJ4cc4DOyburlKxfzLJBLIQBPGocyMCI",
"http://schemas.microsoft.com/identity/claims/tenantid": "89accea6-2b47-43a4-b381-b0410b3778dd",
"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name": "fake@fake.com",
"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "fake@fake.com",
"uti": "PuFHA-Gy90KTKb2q06QjAA",
"ver": "1.0"
},
"correlationId": "97503595-4cf9-4598-b397-ea1ed9e816fc",
"description": "",
"eventDataId": "47d7a7e8-3c38-45dc-9990-31b994426a57",
"eventName": {
"value": "EndRequest",
"localizedValue": "End request"
},
"category": {
"value": "Administrative",
"localizedValue": "Administrative"
},
"httpRequest": {
"clientRequestId": "",
"clientIpAddress": "53.153.83.212",
"method": "PUT"
},
"id": "/subscriptions/9446f748-d132-4d91-8376-ab3f4b0f9r04/resourceGroups/faketaco-testing/providers/Microsoft.Scheduler/jobCollections/rr-testing/jobs/alerting-processor-run/events/47d7a7e8-3c38-45dc-9990-31b994426a57/ticks/636554459533674137",
"level": "Informational",
"resourceGroupName": "faketaco-testing",
"resourceProviderName": {
"value": "Microsoft.Scheduler",
"localizedValue": "Microsoft.Scheduler"
},
"resourceId": "/subscriptions/9446f748-d132-4d91-8376-ab3f4b0f9r04/resourceGroups/faketaco-testing/providers/Microsoft.Scheduler/jobCollections/rr-testing/jobs/alerting-processor-run",
"resourceType": {
"value": "Microsoft.Scheduler/jobCollections/jobs",
"localizedValue": "Microsoft.Scheduler/jobCollections/jobs"
},
"operationId": "97503595-4cf9-4598-b397-ea1ed9e816fc",
"operationName": {
"value": "Microsoft.Scheduler/jobCollections/jobs/write",
"localizedValue": "Microsoft.Scheduler/jobCollections/jobs/write"
},
"properties": {
"statusCode": "Created",
"serviceRequestId": "51f58b29-89bb-4ff9-9c7d-f6c14c140623"
},
"status": {
"value": "Succeeded",
"localizedValue": "Succeeded"
},
"subStatus": {
"value": "Created",
"localizedValue": "Created (HTTP Status Code: 201)"
},
"eventTimestamp": "2018-02-28T20:19:13.3674137Z",
"submissionTimestamp": "2018-02-28T20:19:30.3103612Z",
"subscriptionId": "9446f748-d132-4d91-8376-ab3f4b0f9r04",
"tenantId": "89accea6-2b47-43a4-b381-b0410b3778dd"
}

 

publishxml/action action below:

{
"authorization": {
"action": "Microsoft.Web/sites/slots/publishxml/action",
"scope": "/subscriptions/9446f748-d132-4d91-8376-ab3f4b0f9r04/resourceGroups/faketaco-dev/providers/Microsoft.Web/sites/faketaco-developers-dev/slots/staging"
},
"caller": "3647a659-df46-411d-b91e-911563add5d5",
"channels": "Operation",
"claims": {
"aud": "https://management.azure.com/",
"iss": "https://sts.windows.net/89accea6-2b47-43a4-b381-b0410b3778dd/",
"iat": "1519848864",
"nbf": "1519848864",
"exp": "1519852764",
"aio": "Y2NgYHC7GzV54fflG4+InFRZIhKWBQA=",
"appid": "e409ee81-c932-4d5a-96a2-ebfb406b1f49",
"appidacr": "1",
"http://schemas.microsoft.com/identity/claims/identityprovider": "https://sts.windows.net/89accea6-2b47-43a4-b381-b0410b3778dd/",
"http://schemas.microsoft.com/identity/claims/objectidentifier": "3647a659-df46-411d-b91e-911563add5d5",
"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "3647a659-df46-411d-b91e-911563add5d5",
"http://schemas.microsoft.com/identity/claims/tenantid": "89accea6-2b47-43a4-b381-b0410b3778dd",
"uti": "dVPS2SFVlkicYM5ZmTcTAA",
"ver": "1.0"
},
"correlationId": "c395afb6-a699-48e6-972d-9939c3a8a3ef",
"description": "",
"eventDataId": "053a6d35-29a5-4aa9-b1af-3c66b59081dc",
"eventName": {
"value": "BeginRequest",
"localizedValue": "Begin request"
},
"category": {
"value": "Administrative",
"localizedValue": "Administrative"
},
"httpRequest": {
"clientRequestId": "",
"clientIpAddress": "53.153.83.212",
"method": "POST"
},
"id": "/subscriptions/9446f748-d132-4d91-8376-ab3f4b0f9r04/resourceGroups/faketaco-dev/providers/Microsoft.Web/sites/faketaco-developers-dev/slots/staging/events/053a6d35-29a5-4aa9-b1af-3c66b59081dc/ticks/636554459648256927",
"level": "Informational",
"resourceGroupName": "faketaco-dev",
"resourceProviderName": {
"value": "Microsoft.Web",
"localizedValue": "Azure Web Sites"
},
"resourceId": "/subscriptions/9446f748-d132-4d91-8376-ab3f4b0f9r04/resourceGroups/faketaco-dev/providers/Microsoft.Web/sites/faketaco-developers-dev/slots/staging",
"resourceType": {
"value": "Microsoft.Web/sites/slots",
"localizedValue": "Microsoft.Web/sites/slots"
},
"operationId": "c395afb6-a699-48e6-972d-9939c3a8a3ef",
"operationName": {
"value": "Microsoft.Web/sites/slots/publishxml/action",
"localizedValue": "Microsoft.Web/sites/slots/publishxml/action"
},
"status": {
"value": "Started",
"localizedValue": "Started"
},
"subStatus": {
"value": "",
"localizedValue": ""
},
"eventTimestamp": "2018-02-28T20:19:24.8256927Z",
"submissionTimestamp": "2018-02-28T20:19:40.1798612Z",
"subscriptionId": "9446f748-d132-4d91-8376-ab3f4b0f9r04",
"tenantId": "89accea6-2b47-43a4-b381-b0410b3778dd"
}

Labels (3)
Tags (1)
0 Likes
1 Reply
Lieutenant Commander Lieutenant Commander
Lieutenant Commander

JSON parsers are very flexible, you don't need a match on every Token. Simply assign it and map it. When the field is there, it will be mapped, if not, it won't.

Hope that helps

Thorsten
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.