Highlighted
Super Contributor.. linhvm Super Contributor..
Super Contributor..
1664 views

Oralce ID/Time based Flexconnector - [ERROR] Event with duplicate ID

Dear all,

I am trying to collect event from an application throught Oracle DB with FlexConnector. But when I ran FlexConnector with configuration file attached here, it could not collect anything. When I checked agent.log, it showed st like below:

[2012-04-06 15:11:14,640][ERROR][default.com.arcsight.agent.sdk.b.g.ab][processQuery] Event with duplicate ID [|] for [jdbc:oracle:thin:@Oracle10gR2:1521:orcl], ignoring
[2012-04-06 15:11:14,640][ERROR][default.com.arcsight.agent.sdk.b.g.ab][processQuery] Event with duplicate ID [|] for [jdbc:oracle:thin:@Oracle10gR2:1521:orcl], ignoring
[2012-04-06 15:11:14,640][ERROR][default.com.arcsight.agent.sdk.b.g.ab][processQuery] Event with duplicate ID [|] for [jdbc:oracle:thin:@Oracle10gR2:1521:orcl], ignoring

Could anyone help me to explain this?

Thank you very much.

Best regards,

Linh.

Labels (2)
0 Likes
Reply
7 Replies
Hakan Trusted Contributor.
Trusted Contributor.

Re: Oralce ID/Time based Flexconnector - [ERROR] Event with duplicate ID

Linh Vu, have you fixed the problem?

HAKAN
0 Likes
Reply
santhosh.i Absent Member.
Absent Member.

Re: Oralce ID/Time based Flexconnector - [ERROR] Event with duplicate ID

Hello Linh,

Have u fixed the problem ?

I'm facing same issues!

regards

Santhosh I

0 Likes
Reply
AS_User Honored Contributor.
Honored Contributor.

Re: Oralce ID/Time based Flexconnector - [ERROR] Event with duplicate ID

Hey,

I guess you already figured this out but your uniqueid.fields needs to be a comma separated list of fields to use should your DB come across a non-unique 'ID_NUM'. combine ID_NUM with another column to make it a unique event.

Tom

0 Likes
Reply
katango1 Absent Member.
Absent Member.

Re: Oralce ID/Time based Flexconnector - [ERROR] Event with duplicate ID

Also remember that:


"If you use the > ? setting in the query then no events prior to the last completed query will be returned.  If you have the < ? setting in the query than the first time that the SmartConnector runs it will pull in all of the old events in the database, generating the high EPS.  As it moves forward to the next query it will still retrieve a lot records, but recognize and ignore the duplicates."

Regards

Michael

0 Likes
Reply
deathbywedgie1 Frequent Contributor.
Frequent Contributor.

Re: Oralce ID/Time based Flexconnector - [ERROR] Event with duplicate ID

This is an old thread, so it's possible that this no longer matters, but I stumbled across it and wanted to contribute.

ID based flex connectors rarely run into this kind of issue, since every row should have a completely unique ID. However, with time-based flex connectors, it is possible to have multiple entries with the exact same timestamp and the same values in all the other fields. For instance, in a kind of batch job, there may be two legitimate events that hit for the same values at the same second, thus presenting the appearance of duplicates.

The way I get around this is by adding a count field to the query and then mapping that count to event.aggregatedEventCount.

For example, instead of this:

query =SELECT USERNAME, USER_ID, CREATED FROM all_users WHERE CREATED >= ? ORDER BY CREATED

I might do this:

query =SELECT USERNAME, USER_ID, CREATED, COUNT(CREATED) as EVENT_COUNT FROM all_users WHERE CREATED >= ? GROUP BY USERNAME, USER_ID, CREATED ORDER BY CREATED

In the example above, I created an additional value called "EVENT_COUNT" by simply doing a "count" on the "CREATED" field. Then I grouped by the fields you were trying to select so they would be grouped together. For every unique combination of USERNAME, USER_ID, and CREATED there will now be only a single row, and if there were duplicates in the database then EVENT_COUNT will return with a value equal to the number of occurrences in the database.

If you do it this way you will have more accurate data and should no longer see the duplicate event errors.

Cheers!

--dbw

0 Likes
Reply
Super Contributor.. MuraliKV Super Contributor..
Super Contributor..

Re: Oralce ID/Time based Flexconnector - [ERROR] Event with duplicate ID

I had similar issue with "duplicate ID" and was able to resolve the issue based on another article.  this is how i fixed the duplicate iD issue.

select AU.USER_NAME AS USERNAME,\
       to_timestamp(to_char(AL.ATTEMPT_TIME, 'DD-MON-YYYY HH24:MI:SS'), 'DD-MON-YYYY HH24:MI:SS') AS EVENTDATETIME\
FROM LOGINS AL, USER AU \
WHERE al.user_id = au.user_id \
AND to_timestamp(to_char(AL.ATTEMPT_TIME, 'DD-MON-YYYY HH24:MI:SS'), 'DD-MON-YYYY HH24:MI:SS') > ? \
ORDER BY AL.attempt_time

lastdate.query=select MAX(to_timestamp(to_char(ATTEMPT_TIME, 'DD-MON-YYYY HH24:MI:SS'), 'DD-MON-YYYY HH24:MI:SS')) from LOGINS

timestamp.field=EVENTDATETIME
uniqueid.fields=eventdatetime

Thank you

Murali

0 Likes
Reply
naamam Absent Member.
Absent Member.

Re: Oralce ID/Time based Flexconnector - [ERROR] Event with duplicate ID

Hey

I had the same problem,

I resolved it by giving alias for each columns in the select (Time AS Timestamp etc.), as well I changed the mapping of from endTime to flexDate1.

Hope it will help.

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.