Commodore Commodore
Commodore
2006 views

Oralce ID/Time based Flexconnector - [ERROR] Event with duplicate ID

Dear all,

I am trying to collect event from an application throught Oracle DB with FlexConnector. But when I ran FlexConnector with configuration file attached here, it could not collect anything. When I checked agent.log, it showed st like below:

[2012-04-06 15:11:14,640][ERROR][default.com.arcsight.agent.sdk.b.g.ab][processQuery] Event with duplicate ID [|] for [jdbc:oracle:thin:@Oracle10gR2:1521:orcl], ignoring
[2012-04-06 15:11:14,640][ERROR][default.com.arcsight.agent.sdk.b.g.ab][processQuery] Event with duplicate ID [|] for [jdbc:oracle:thin:@Oracle10gR2:1521:orcl], ignoring
[2012-04-06 15:11:14,640][ERROR][default.com.arcsight.agent.sdk.b.g.ab][processQuery] Event with duplicate ID [|] for [jdbc:oracle:thin:@Oracle10gR2:1521:orcl], ignoring

Could anyone help me to explain this?

Thank you very much.

Best regards,

Linh.

Labels (2)
0 Likes
7 Replies
Commander
Commander

Linh Vu, have you fixed the problem?

HAKAN
0 Likes
Absent Member.
Absent Member.

Hello Linh,

Have u fixed the problem ?

I'm facing same issues!

regards

Santhosh I

0 Likes
Vice Admiral
Vice Admiral

Hey,

I guess you already figured this out but your uniqueid.fields needs to be a comma separated list of fields to use should your DB come across a non-unique 'ID_NUM'. combine ID_NUM with another column to make it a unique event.

Tom

0 Likes
Absent Member.
Absent Member.

Also remember that:


"If you use the > ? setting in the query then no events prior to the last completed query will be returned.  If you have the < ? setting in the query than the first time that the SmartConnector runs it will pull in all of the old events in the database, generating the high EPS.  As it moves forward to the next query it will still retrieve a lot records, but recognize and ignore the duplicates."

Regards

Michael

0 Likes

This is an old thread, so it's possible that this no longer matters, but I stumbled across it and wanted to contribute.

ID based flex connectors rarely run into this kind of issue, since every row should have a completely unique ID. However, with time-based flex connectors, it is possible to have multiple entries with the exact same timestamp and the same values in all the other fields. For instance, in a kind of batch job, there may be two legitimate events that hit for the same values at the same second, thus presenting the appearance of duplicates.

The way I get around this is by adding a count field to the query and then mapping that count to event.aggregatedEventCount.

For example, instead of this:

query =SELECT USERNAME, USER_ID, CREATED FROM all_users WHERE CREATED >= ? ORDER BY CREATED

I might do this:

query =SELECT USERNAME, USER_ID, CREATED, COUNT(CREATED) as EVENT_COUNT FROM all_users WHERE CREATED >= ? GROUP BY USERNAME, USER_ID, CREATED ORDER BY CREATED

In the example above, I created an additional value called "EVENT_COUNT" by simply doing a "count" on the "CREATED" field. Then I grouped by the fields you were trying to select so they would be grouped together. For every unique combination of USERNAME, USER_ID, and CREATED there will now be only a single row, and if there were duplicates in the database then EVENT_COUNT will return with a value equal to the number of occurrences in the database.

If you do it this way you will have more accurate data and should no longer see the duplicate event errors.

Cheers!

--dbw

0 Likes
Commodore Commodore
Commodore

I had similar issue with "duplicate ID" and was able to resolve the issue based on another article.  this is how i fixed the duplicate iD issue.

select AU.USER_NAME AS USERNAME,\
       to_timestamp(to_char(AL.ATTEMPT_TIME, 'DD-MON-YYYY HH24:MI:SS'), 'DD-MON-YYYY HH24:MI:SS') AS EVENTDATETIME\
FROM LOGINS AL, USER AU \
WHERE al.user_id = au.user_id \
AND to_timestamp(to_char(AL.ATTEMPT_TIME, 'DD-MON-YYYY HH24:MI:SS'), 'DD-MON-YYYY HH24:MI:SS') > ? \
ORDER BY AL.attempt_time

lastdate.query=select MAX(to_timestamp(to_char(ATTEMPT_TIME, 'DD-MON-YYYY HH24:MI:SS'), 'DD-MON-YYYY HH24:MI:SS')) from LOGINS

timestamp.field=EVENTDATETIME
uniqueid.fields=eventdatetime

Thank you

Murali

0 Likes
Absent Member.
Absent Member.

Hey

I had the same problem,

I resolved it by giving alias for each columns in the select (Time AS Timestamp etc.), as well I changed the mapping of from endTime to flexDate1.

Hope it will help.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.