Highlighted
Respected Contributor.. Respected Contributor..
Respected Contributor..
1421 views

FlexConnector JSON parser - Could not retrieve trigger nodes

Jump to solution

Hello,

My FlexConnector (FlexConnector REST) has problem with parsing JSON, I cannot find mistake.

Agent.log writes: 

[2019-08-16 15:55:45,220][WARN ][default.com.arcsight.agent.sdk.b.h][getTriggerNodes] Could not retrieve trigger nodes for the trigger node location [null] specified in the parser. Cannot parse Json : [JSON content]
[2019-08-16 15:55:45,220][WARN ][default.com.arcsight.agent.sdk.b.h][parseJsonTokens] Couldnt find trigger nodes in the Json [JSON content]
[2019-08-16 15:55:45,220][ERROR][default.com.arcsight.agent.sdk.b.h][isJSONEventUnparsed] Cannot parse JSON events [null]
[2019-08-16 15:55:45,220][ERROR][default.com.arcsight.common.log.j][logUnparsedEvent] Cannot parse raw event [Unparsed JSON event] with ArcSight SmartConnector [class com.arcsight.agent.loadable.agent._FlexRestApiAgent], and Parser [class com.arcsight.agent.sdk.b.h]. Parser Result: []. Parsing Exception: [].

JSON content looks:

{
"took": 6,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1144,
"max_score": 1.0,
"hits": [
{
"_index": "logs_v1",
"_type": "auditlog",
"_id": "81a2547d-3bd6-4d22-b6ad-10ab7ea87c77",
"_score": 1.0,
"_source": {
"id": "81a2547d-3bd6-4d22-b6ad-10ab7ea87c77",
"caseId": "172476",
"description": "Dokument D001 byl předán do Elektronické služby.",
"documentId": "187226_900bbc64874642879875f1d748b09ee1",
"ended": "2019-08-15T14:48:24.5754724+02:00",
"started": "2019-08-15T14:48:24.5754724+02:00",
"type": "EleSend",
"status": "Success",
"userId": "5"
}
},
... list of hits continues

My parser looks:

trigger.node.location=/hits/hits

token.count=11

token[0].name=eventId
token[0].type=String
token[0].location=_source/id

token[1].name=caseId
token[1].type=String
token[1].location=_source/caseId

token[2].name=description
token[2].type=String
token[2].location=_source/description

token[3].name=ended
token[3].type=String
token[3].location=_source/ended

token[4].name=started
token[4].type=String
token[4].location=_source/started

token[5].name=eventType
token[5].type=String
token[5].location=_source/type

token[6].name=status
token[6].type=String
token[6].location=_source/status

token[7].name=userId
token[7].type=String
token[7].location=_source/userId

token[8].name=documentId
token[8].type=String
token[8].location=_source/documentId

token[9].name=logType
token[9].type=String
token[9].location=_type

token[10].name=score
token[10].type=String
token[10].location=_score

 

Any idea where is the problem?

Regars,
Jan Sevela

Labels (1)
Tags (2)
0 Likes
1 Solution

Accepted Solutions
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

The JSON is missing a closing square bracket it seems? Or is there more to this event?

What is the name of your parser file? jsonparser.properties?

I wonder if it complains about the names being the same (hits and hits), that would be more an issue with our jsonparser than the JSON content though.

What happens if you set the trigger.node.location to only root "/", and then try to access a few values outside of hits and inside hits, for example:

trigger.node.location=/

token.count=3

#Event that is on root
token[0].name=took
token[0].type=String
token[0].location=took

# Event one array down
token[1].name=total
token[1].type=String
token[1].location=_shards/total

# Event array inside another array using [] instead of {}
token[2].name=status
token[2].type=String
token[2].location=hits/hits/_source/status
-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius

View solution in original post

0 Likes
4 Replies
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

The JSON is missing a closing square bracket it seems? Or is there more to this event?

What is the name of your parser file? jsonparser.properties?

I wonder if it complains about the names being the same (hits and hits), that would be more an issue with our jsonparser than the JSON content though.

What happens if you set the trigger.node.location to only root "/", and then try to access a few values outside of hits and inside hits, for example:

trigger.node.location=/

token.count=3

#Event that is on root
token[0].name=took
token[0].type=String
token[0].location=took

# Event one array down
token[1].name=total
token[1].type=String
token[1].location=_shards/total

# Event array inside another array using [] instead of {}
token[2].name=status
token[2].type=String
token[2].location=hits/hits/_source/status
-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius

View solution in original post

0 Likes
Highlighted
Respected Contributor.. Respected Contributor..
Respected Contributor..

Hello,

Hello, there is 1144 events (/hits/total). I wrote only first event. If I copy complete JSON content from log, brackets are OK, everyone closed.

Name is eessi.jsonparser.properties, in agent.properties is...nothing...I dont understand I thought there was value: 
agents[0].parserfile=eessi

OK, my mistake. It was the problem! Thank you.

Now Connector works better, without these ERRORs, but problem with $START_AT_TIME variable, if I set at agent.properties:
agents[0].eventsurl=http\://10.10.10.10\:9222/logs/auditlog/_search?q\=ended\:[$START_AT_TIME TO *]&size\=10000
agents[0].startattime=2019-08-08T00:00:00.000

Connector makes first query with this timestamp, but next query converts to value something like this: 
1566474788719

in agent.log this execute URL is:
[2019-08-22 13:25:59,112][INFO ][default.com.arcsight.agent.loadable.agent._FlexRestApiAgent][getHttpResponse] Executing the Events URL : [http://10.10.10.10\:9222/logs/auditlog/_search?q=ended:[1566474788719 TO *]&size=10000]

If I had problems with ERRORs, URL execute was the same:
[2019-08-16 15:29:01,543][INFO ][default.com.arcsight.agent.loadable.agent._FlexRestApiAgent][getHttpResponse] Executing the Events URL : [http://10.10.10.10\:9222/logs/auditlog/_search?q=ended:[2019-08-08T00:00:00.000 TO *]]

but now, when parser works, conector changing it to number value after first quering.

Endpoint cannot accept connector format number value. I tried this URL in browser and 0 total events in JSON.

 

 Any idea please?

 

Regards,
Jan Sevela

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

It's not often enough that I do these REST flex connectors compared to normal flex, but i'm 99% sure it is based on deviceReceiptTime.

After the initial run, it will store the newest deviceReceiptTime in memory, and when doing the next URL request it will use that as the start time.

Now since you don't have that set in your mapping file it uses the default, and since ESM default is Epoch/Unix time, you get these timestamps instead.

So what you need to do is:

Create a token for the time, hopefully it is somewhere in the JSON, and set it like this:

#Time of the event
token[0].name=time
token[0].type=String
token[0].location=/some/location/for/time

event.deviceReceiptTime=__createOptionalTimeStampFromString(time,"yyyy-MM-ddTHH:mm:ss.SSS")

 

Hopefully that resolves it 🙂

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Highlighted
Respected Contributor.. Respected Contributor..
Respected Contributor..

Hello,

Thank you for you answer, but resolve of this problem is parameter from last documentation of FlexConnector REST developers guide, agent.properties:

timestamp_format_of_api_vendor

If it is set correct for REST API endpoint, it works!

Regards,

Jan Sevela

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.