Multifolder Flex path filter
I am trying to process data from a cifs repository, but I need to access some specific directories only:
But \\access_to_share\2016\08\02\ is full of different directories/files, where my user doesn't even have permissions to access. So I cannot use agents[x].processfoldersrecursively (It will have tons of access errors, and the repo is huge so it will take a long time to scan for new files). And I don't want to process the whole repo, I just need to process the new files.
I have tried to use "date based rotation" feature enabled by agents[x].wildcard = \'yyyy'\'MM'\'dd'\specific_path\file-'yyyyMM'.log, but I can not get it to work, It crash with a java.lang.null exception and a "is not correct for date based rotation"
Re: Multifolder Flex path filter
Right, so i changed it to:
Then, this is what I can see on the logs:
... In dir [null] verifying [FILEWITHDATA.CONF.201501] with regex [\\16\\08\\SPECIFIC_FILE\\.*\\FILE-1608.*\.LOG]
[2016-08-02 11:28:36,217][DEBUG][default.com.arcsight.agent.ee.b][WildcardFileFilter] Initializing wildcard filter [\16\08\specific_file\*\file-1608*.LOG]. Corresponding regex [\\16\\08\\SPECIFIC_FILE\\.*\\FILE-1608.*\.LOG]
If I change the "folder" to include the dynamic part -'\\'yy'\\'MM'\\specific_file\\'- the logs will show that the path "\\path_to\repo\\yy\MM\specific_file" doesn't exist, so its clear that the path is static, and I can not even use a '*'
And the wildcard regex is only matched with the actual file, not with the full path.
Any ideas? The problem is that every time that the connector tries to read a subfolder where the user does not have read access, the scan begins again, so the connector is not able to locate any files to process.