Highlighted
Absent Member.
Absent Member.
835 views

SmartConnector - Does post-processing cache affect pre-processing cache?

Dear All,

Before my question, here is what i know about Log flow in a smart connector (E.g. Syslog connector). Please correct me if i am wrong about any details

Smart Connector Caching.jpg


Cache Details:


Pre-processing cache

  • File extension for Pre-processing cache files: .queue Reference link
  • This cache starts building up if the incoming EPS is very high and Connector cannot process the logs at that rate. Are there any other reasons for occurrence of pre-processing cache?

Post-processing cache

  • File extension for Post-processing cache files: .dlft
  • This cache starts building up if connector cannot send data to Logger. Reasons might be - Logger is down, Network issue and connector cannot send data to logger. Are there any other reasons for occurrence of post-processing cache?


* Pre-processing and Post-processing cache files are stored in location - \ArcSight Connectors\Connector\current\user\agent\agentdata

* Default size of agentdata folder is set to 1GB

* New logs will be dropped if agentdata folder reaches the size limit of 1GB

Once logs are processed, these post-processed logs are forwarded to destination (ESM/Logger) in 70:30 ratio. 70% live logs and 30% cached logs are forwarded.....Is this correct?


Question:


If the agentdata folder is still within size limit (1 GB), does post-processing cache affect smart connector processing capability?

i.e. Both Post-processing and Pre-processing cache are building up but both are within 1 GB limit. Does post-processing file size impact smart connector performance and then pre-processing cache further increases?


Is there any relation between post and pre-processing cache? If not, are they completely independent? No matter what the pre and post caching size is, smart connector won't be affected unless file size limit of 1 GB is exceeded.


Regards,

Sujay

Labels (3)
0 Likes
Reply
3 Replies
Highlighted
Acclaimed Contributor.. Acclaimed Contributor..
Acclaimed Contributor..

I believe the cache size limit (1GB) is per-destination, not the whole of userdata directory.

0 Likes
Reply
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Hi Shaun,

NO it is shared for all Destination if this has not changed with the last connector version.

If you have setup 50GB, it is for ESM and Logger, if you sent events to both destinations.

But you have to take inconsideration that it is compressed 10x (Avg). Sometimes more but sometimes less also, it depends of the raw logs.

Now to answer to Sujay, I need to search more information to confirm my expectation but I think there is only one cache

done by the connector this is why these files are all saved in agentdata. Because this morning my ESM was down for few hours, I have found only DFLT files on cache. Regarding the information I have found, Queue files can be used by Syslog connector to process logs but it is not mandatory! They are not considered as connector event caching files.

If I found more information about this, I will come to you.

Thanks

Kind Regards

Michael

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Thanks Michael.

Even i am trying to get info from HP and other sources. Let me know if you find anything.

Regards,

Sujay

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.