populating deviceExternalid

[habex379]

Hi,

it looks like I cannot reply on 
https://community.microfocus.com/t5/Archive-Discussion-Board/Arcsight-not-picking-up-quot-deviceExternalid-quot-field-at/td-p/1513739

hence my comment here:

"it looks like it is not about turbo mode or similar, I just wrote a flex and experiencing the same:
despite properly tokenized, only got deviceExternalId populated in a few cases.
I can see the raw, can check with regex tester, the value is there, but doesn't appear in the active channel, except a few cases.

I didn't see such exceptions mentioned in ESM 101 nor ESM Admin Guide nor Console User's Guide nor Flex Dev Guide. Environment: connector framework v7.12.0.8149.0, ESM v6.9.1, "

KR,
H.

[mschleich]

Hi habex379,

It is very strange because I use this field a lot to differentiate device category having the same device Vendor.

By example EX for Exchange and AD for Domain Controller, and I index this field in Loggers.

With one simple query on that field, I can simplify a search for specific devices in place of using OR or IN and the IPs.

Do not forget that [deviceExternalId] is a string field limited to 255 chars!

Most of my connectors are in Complete Turbo Mode thus  I cannot verify if it is the issue but could you please retry with Complete Turbo Mode.
Regarding CEF, this field exists and i remember that this field is used to detect devices attached to a manager by example, this field is populated with SourceFire Events to identify the sensor.

It should work.

Could you please firstly check the format type and the data length then could you please check into the agent.log or agent.wrapper.out.log if you can see ERROR or WARN events about this field?

I hope this will help you to solve your issue.

Thanks
Regards

Michael

[habex379]

Hi Michael,

many thanks for your comment.
In my case it is about a token matching \\d+, but having its type set as string. The field length never exceeds 5. Yeah, the connector logs were the first to check, but no such field related warnings, errors in there.

As to turbo mode.
Despite "You can configure connectors to send more or less event data on a per-connector basis, and the ESM Manager can be set to read and maintain more or less event data, independent of the connector setting" (taken from SmartConnectorUserGuide), it is unlike aggregation (see preserve common fields yes/no), tubo mode means either always available or always missing, not "sometimes set". Regardless of this, I have checked it, turbo mode is unchanged, i.e. still set to complete.

I have changed the parser, now putting the mentioned token into externalId instead of deviceExternalId and there are no more issues since that. Note: externalId is even shorter (field / type / length):
- deviceExternalId / String / 255
- externalId / String / 40

I fully agree, it _should_ work, however it doesn't. :S

I had a chat with some colleagues re this, sometimes they have experienced similar issues. Their opinion was, that this behaviour looks to vary depending on connector version too. Maybe a knowledgeable MF engineer is reading this topic and will comment it (or file a bug)?...
Environment: connector framework v7.12.0.8149.0, ESM v6.9.1,

Kindest regards,
H.

[mschleich]

Hi habex379,

I think the issue is because you have parsed as \\d+ which is for digit thus it won't work.

You have 2 solutions:

your token is configured as an integer in that case, you have to use a string conversion function to convert digit in string and then you can map to deviceExternalId

your token is directly configured as a string, in that case you can directly save it as deviceExternalId

I don't think it is a SmartConnector version issue, I am using this too and it is working as expected but you have to take into account that this is not the best choice to identify events ID, in that case, it is better to use externalId or deviceEventClassId.

This will have an impact in the DB with device descriptor but I am not sure if it still the case with ESM v6.9.1 and above because this was specific to Oracle.

To summarize, use deviceExternalId if the number of unique value is very low on the other case use another fields like the one mentioned above.

Thanks
Regards

Michael

[habex379]

Hi Michael,

thanks for your message. Yes, \\d+ has been tokenized as a string, it definitely works. The issue was about the side table protection. Maybe I won't map into deviceExternalId, however it was good to get this challenge solved. 🙂

Best regards,
H.

[vitz1 · Accepted solution]

deviceExternalId is still protected by the side table protector

https://community.microfocus.com/t5/ArcSight-User-Discussions/Too-many-devices-being-created-possible-parsing-problem/td-p/1551545 

post oracle times a lot of people say, you could go easily and disable it, but i dont belive so.

Not sure about your issue in detail. however if you have one deviceAddress and more then 10 different deviceExternalIds for the same IP, you will have an empty field for the rest of the deviceExternalId s. And this is "expected". 

I assume there might be a parsing issue, as deviceExternalid should be used to identify a device, and not an alert from a device. Use event.externalId instead.

Cheers

A

[habex379]

Nope, fortunately there is no parsing issue, but the side table protector you have mentioned. I have planned to put process ID into deviceExternalId. Out of curiosity I have disabled the side table protector as per your suggestion and now getting both, extrenalId _and_ deviceExternalId filled properly. Thanks for calling my attention to this, might make sense to change your nick to "Figjam"....
....no, not because of this: https://www.stonewallkitchen.com/dw/image/v2/AAYB_PRD/on/demandware.static/-/Sites-swk-catalog/default/dw24b02b8b/images/100825-New-Lid.jpg
Not even because of this: https://www.stonewallkitchen.com/dw/image/v2/AAYB_PRD/on/demandware.static/-/Sites-swk-recipes/default/dwa779c2d9/images/rec_bakedcamembert.jpg
But because of this: https://en.wikipedia.org/wiki/FIGJAM 🙂

Have a nice evening,
H.

[vitz1]

@habex379 

Thanks for the Flowers 😉 and the JAM.

"Parsing issue" refers to: somebody puts something in the wrong field 

Have good day.

A