Mutliple Folder Follower - Conditional Filtering?
Current situation as such:
1) I have a network device which upload logs hourly to a file server.
2) As observed from the logs uploaded on the file server, each file uploaded hourly is a new file of its own with file name format; log.0, log.1,etc
3) Multiple Folder Follower connector is being used to read and parse the file contents, batch mode.
4) The troubling issue is that the uploaded files contain overlap of/duplicate events. Eg. For log.0, it contains events from 0000 - 0100 Hrs and the subsequent log.1 contains events from 0000 - 0200 Hrs.
I was wondering if any form of conditional filtering can be done. Eg. If the time registered in the event is earlier than the current NTP time, it will filtered and not sent to the Logger.
Any alternatives for suggestion is much appreciated.