Can the connectors be configured for D-Duplication?

[jlf23_231]

Can the connectors be configured for D-Duplication?

[john.petropoulo1]

No they can’t.

To de-dup events the connector would have to have a very accurate understanding of how a device behaves under specific conditions. Today, the connector only categorizes events in very general terms, and doesn’t provide any logic on pulling out the single event that is interesting while suppressing the rest.

This is further complicated if you think about the event mappings and the required fields for proper reporting and correlation. Some products provide a single event that is very useful, however many split the information over several events and require upstream correlation/tracking.

/J

[MaryCordova]

Aggregation could achieve some similar results to de-duplication couldn't it?

[andrew.dalbor]

Agree with Mary on that one.  Aggregation would achieve pretty similar results depending on the aggregation conditions.

[prentice@hpe.co]

That depends on what you consider duplication.

If you are talking about a log entry from a single device being collected more than once, that shouldn't happen (ideally, but it can...).

If you are talking about a log entry from a single device that the device keeps repeating, the connector can't do much about that.

If you are talking about a group of devices creating multiple copies of the same log entry, you've got other problems (and no, the connector isn't going to help).

Aggregation will not dedupe events. It will collect very similar log entries over a small time period and send a single event with an aggregation count based on the number of log entries, but that is not deduplication.

Apologies for being pedantic, but it seems like it was needed.

[jdickinson@hpe.]

Event duplication in connectors usually happens because there are multiple destinations in one connector registered and forwarding events.  1 destination is likely forwarding to a logger that in turn is forwarding to the ESM.  The other destination is registered to the ESM and is also forwarding events.  Look at the event in question there should be 2 different event id's and 2 different original agent id's. if the same event has different event id then it is actually 2 entries in the database.  If it also has 2 original agent id's then it is coming from one of the other destinations in the connector.

Alternatively event duplication could be confused with event aggregation. If the log has many similar events that seem to be the same.  these can be aggregated so that 1 event is sent to the ESM with a aggregated event count of 1 or more.