Issue Maximum number of cache files reached in Smart Connector Regex File Reader
I found this WARN in my Flex connector Regex file reader:
[2020-03-23 20:45:20,549][WARN ][default.com.arcsight.agent.s.h][shiftDown] Maximum number of cache files reached  for cache [3LqGPN28BABCcr0jaxKkpsc==.m1]
[2020-03-23 20:45:20,549][WARN ][default.com.arcsight.agent.s.h][dropReader] Dropped  alerts in  ms. for cache [3LqGPN28BABCcr0jaxKkpsc==.m1 (total dropped = 47025)
can anyone tell me how I can solve this problem, or if it's already happened to someone?
What you can see is the results of caching issues, which happens because of the network issues or destination cannot cope with incoming EPS.
I would recommend checking what's happening and why the connector is caching towards the destination.
At the moment, you can move cache files to some other location and leave the agentdata folder empty for the next connector restart.
Check the destination ID (in agent.properties) and compare it to the files under user/agent/userdata.
If you have changed the destination or re-register it, the ID will change so the old cache files will remain there (in that case you can delete them).
What is the EPS in rate compared to EPS out? Is it similar or the connector is not processing the events fast enough?
You can try increasing the batch size as well as the number of threads which might improve the situation.
The next document is explaining the same and will tell you which commands to use:
You can also check your destination if it's overloaded or if there are any other issues.
It would be good to check below articles to get more information:
I hope this helps.
thank you for the answer, there are two destinations and send to ESM, I set the batching to 400 and the cache size to 10 GB for each destination, but the problem persists.
The Connector processes one file per day of 2/2.5 GB size.
could you tell me some parameters to modify to solve the problem?
If you have ESM as destinations, then setting the batch size to 400 is the maximum value to go.
Here are the settings related to ESM transport, which will enable multi threads and queue size. You can try with different values here:
Other than that, check if events are parsed, do you have any fatal or error messages in logs.
If this doesn't help, then check if ESM can handle the incoming EPS and if there are any issues there.
thank you for your reply, I applied what you advised but I still receive the WARN message, fortunately there are no ERROR or FATAL messages.
I also set the cache size to 10 GB.
how do you say to proceed now ?
Hello @UbissIctAppSic1 ,
Well, if you did everything, there is not much to do from the connector's side. Just make sure you restart the connector after adding those parameters to agent.properties.
Check and follow my recommendation from the first update.
What is the EPS in? What is the transport value to that destination?
Do you have any older cache files (compare the destination ID as described)?
Check ESM side as well.
Increasing the cache size is not a solution. If your cache is constantly growing due to the fact that EPS out is lower compared to incoming EPS, then no matter what value you set it will fill at some point.
Try to put ESM closer to connector (not sure what is the RTT in your case, check this as well).