Folder Regex Reader Current Event

[renjithrjames]

Good Day!

In our environment , We have a requirement to integrate a application which is recording the data to flat file. In this same file it already having 3 year data but we don't want read older data (complete logs in the file), which the requirements say activities only from the current date.Is there any possibility to say the folder agent to take log starting from current day.

Appreciate quick help!

Thanks

Renjith R James

[renjithrjames]

Expecting Support...

[A-Aron]

That depends on a few things.

- how often is the file updated? If there is one file that has events from more than 3 years, this must be a very low event rate. Most solutions have a file rotation that will rotate the file out after it was read by the connector, maybe using date in the file name. Then the connector reads only the current file.

If you are only doing this once, can you manually change the files so the current file has the current events?

There are other things you could use too. For example the regex file reader lets you specify

line.ignore.regex=put here the regex that matches dates before the current date

For example if your dates are 02-Jan-2014 and 03-Mar-2015 then you could do

line.ignore.regex=(\d{1,2}-\S{3}-2014 | \d{1,2}-Jan-2015| and so on)

[renjithrjames]

Hi Aaron,

Thanks for your sharing. I am afraid if this parameter utilize the process highly hence it is one of the mission critical system. However, as you mentioned, we are on the discussion to rotate file on daily basis.

Thanks & Regards

Renjith R James

[Pushpendra_Rathi]

Hi Renjith

Try the following:

1) You can use/configure the "agents[x].startatend = true" parameter in agent.properties file...

agents[x].startatend

The default is true. Useful when log files to be processed already exist and contain data at connector startup or when the log file rotation takes place. Setting this value to false will cause the entire file to be read at every startup, which could lead to duplicate events. So you can set this value to true so that the connector will start reading the file from the END (after getting newer events) and not from beginning...

2) And you can also use the parameter "agents[x].preservestate=true"

agents[x].preservestate=true

it marks the file in the location last read at shutdown and begins to read the file from that location at restart means This will tell the SmartConnector to remember the last record that was processed and then when new records are added it will grab them.

Regards

Pushpendra

[renjithrjames]

Hi Pushpendra,

Thanks for your help, in addition to above parameters. The same folder is having archiving events also, every hour the log file is archieving in .gz extension and so I modified the wildcard entry as *.log (because the current log file is .log format), and after the parameter change it is not picking the log. Is there any parameters to track file modification ?(I tried parameter "agents[x].foldertable[x].followexternalrotation=true" which doesn't work).

Thanks & Regards

Renjith