Highlighted
Absent Member.
Absent Member.
1259 views

How to read historical events from time based flex db

Jump to solution

Hi Guys,

Can someone please help me to configure time based flex db properties so that it reads all or some historical events from the database everytime connector is restarted. I have tried this,

Sample Query:

query = select a,b,c from d where time_stamp < ? (It only fetches all the historical events first time connector is installed)

TimeStamp.field= time_stamp

uniqueid.fields=time_stamp

in agent.properties,

agent[0].startatdate = 01/07/2015 00:00:00 (This also does not work and I keep on getting the error "Duplicate records with unique id were found. Ignoring")

I need it for testing as DB admin can not generate events everytime I need to see the output after applying various functions. I believe, there is some file where records of received logs are maintained with their unique id and based on that connector decides, if the record is duplicate or not. However not sure about it.

Can someone please help me.

Regards,

Amit Gupta

Tags (2)
0 Likes
1 Solution

Accepted Solutions
Highlighted
Absent Member.
Absent Member.

After hours of hit and trial, I found the way.

Where audit_time < ? will work first time but not after restart as event ids are saved and duplicate events will be ignored.

These files are under /agent/agentdata. So just delete these files and restart. All the old events will be read in every restart in the Time Based Flex Connector.

Much required while building to map the data properly and test multiple times.

Regards,

Amit

View solution in original post

0 Likes
3 Replies
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

References:

https://protect724.hp.com/message/39963#39963

https://protect724.hp.com/message/38456#38456

Any guess Bro. Reading old Row entries from the Database using Time based flex

0 Likes
Highlighted
Absent Member.
Absent Member.

After hours of hit and trial, I found the way.

Where audit_time < ? will work first time but not after restart as event ids are saved and duplicate events will be ignored.

These files are under /agent/agentdata. So just delete these files and restart. All the old events will be read in every restart in the Time Based Flex Connector.

Much required while building to map the data properly and test multiple times.

Regards,

Amit

View solution in original post

0 Likes
Highlighted
Respected Contributor.
Respected Contributor.

Hi All..I did come across a similar item (error) today, this is a time based Flex-connector and did try almost all the possible solutions discussed on this community however had no luck. I did replace my original query with a very straight forward one-line query which din't work though, below is what written in to the parser file

version.order=1

version.id=1

version.query=SELECT USER_NAME from FND_USER

query = SELECT B.USER_NAME, B.USER_ID, B.CREATION_DATE FROM FND_USER B WHERE B.CREATION_DATE <= ? Order by B.CREATION_DATE

timestamp.field= B.CREATION_DATE

uniqueid.fields= B.CREATION_DATE, B.USER_ID, B.USER_NAME

event.name=__stringConstant("XX system logs")

event.sourceUserName=B.USER_NAME

event.sourceUserId=B.USER_ID

event.deviceReceiptTime=B.CREATION_DATE.

I did attempt almost all suggestions from various posts in the community including

replace the < with >, ? with a specific date, create and assign a new field by modifying the query as below, etc.,

SELECT B.USER_NAME, B.USER_ID, B.CREATION_DATE, COUNT(B.CREATION_DATE) as EVENT_COUNT FROM FND_USER B WHERE B.CREATION_DATE <= ? group by B.USER_ID, B.USER_NAME B.CREATION_DATE order by B.CREATION_DATE

though these changesdin't get me the results. All I faced is the same error below

"[ERROR][default.com.arcsight.agent.sdk.d.b.k][processQuery] Event with duplicate ID [|||] for [jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=xxx.xxx.xxx.xxx)(PORT=1530))(CONNECT_DATA=(SERVICE_NAME=XX)(INSTANCE_NAME=YYYY))], ignoring"

Looking forward for your suggestions, thanks. in advance

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.