SmartConnector - Does post-processing cache affect pre-processing cache?
Before my question, here is what i know about Log flow in a smart connector (E.g. Syslog connector). Please correct me if i am wrong about any details
- File extension for Pre-processing cache files: .queue Reference link
- This cache starts building up if the incoming EPS is very high and Connector cannot process the logs at that rate. Are there any other reasons for occurrence of pre-processing cache?
- File extension for Post-processing cache files: .dlft
- This cache starts building up if connector cannot send data to Logger. Reasons might be - Logger is down, Network issue and connector cannot send data to logger. Are there any other reasons for occurrence of post-processing cache?
* Pre-processing and Post-processing cache files are stored in location - \ArcSight Connectors\Connector\current\user\agent\agentdata
* Default size of agentdata folder is set to 1GB
* New logs will be dropped if agentdata folder reaches the size limit of 1GB
Once logs are processed, these post-processed logs are forwarded to destination (ESM/Logger) in 70:30 ratio. 70% live logs and 30% cached logs are forwarded.....Is this correct?
If the agentdata folder is still within size limit (1 GB), does post-processing cache affect smart connector processing capability?
i.e. Both Post-processing and Pre-processing cache are building up but both are within 1 GB limit. Does post-processing file size impact smart connector performance and then pre-processing cache further increases?
Is there any relation between post and pre-processing cache? If not, are they completely independent? No matter what the pre and post caching size is, smart connector won't be affected unless file size limit of 1 GB is exceeded.
NO it is shared for all Destination if this has not changed with the last connector version.
If you have setup 50GB, it is for ESM and Logger, if you sent events to both destinations.
But you have to take inconsideration that it is compressed 10x (Avg). Sometimes more but sometimes less also, it depends of the raw logs.
Now to answer to Sujay, I need to search more information to confirm my expectation but I think there is only one cache
done by the connector this is why these files are all saved in agentdata. Because this morning my ESM was down for few hours, I have found only DFLT files on cache. Regarding the information I have found, Queue files can be used by Syslog connector to process logs but it is not mandatory! They are not considered as connector event caching files.
If I found more information about this, I will come to you.