This is an old thread, so it's possible that this no longer matters, but I stumbled across it and wanted to contribute.
ID based flex connectors rarely run into this kind of issue, since every row should have a completely unique ID. However, with time-based flex connectors, it is possible to have multiple entries with the exact same timestamp and the same values in all the other fields. For instance, in a kind of batch job, there may be two legitimate events that hit for the same values at the same second, thus presenting the appearance of duplicates.
The way I get around this is by adding a count field to the query and then mapping that count to event.aggregatedEventCount.
For example, instead of this:
query =SELECT USERNAME, USER_ID, CREATED FROM all_users WHERE CREATED >= ? ORDER BY CREATED
I might do this:
query =SELECT USERNAME, USER_ID, CREATED, COUNT(CREATED) as EVENT_COUNT FROM all_users WHERE CREATED >= ? GROUP BY USERNAME, USER_ID, CREATED ORDER BY CREATED
In the example above, I created an additional value called "EVENT_COUNT" by simply doing a "count" on the "CREATED" field. Then I grouped by the fields you were trying to select so they would be grouped together. For every unique combination of USERNAME, USER_ID, and CREATED there will now be only a single row, and if there were duplicates in the database then EVENT_COUNT will return with a value equal to the number of occurrences in the database.
If you do it this way you will have more accurate data and should no longer see the duplicate event errors.
Cheers!
--dbw
