What is "m1.cache.dflt.0" cache files ?
I got the the following definition for M1 cache but it does not make sense to me that why connector has to remember the correlated events as correlation is done on ESM.
I have seen some connectors which has almost empty cache (queue.syslogd.0 or .cache.dflt.0) but still lots of "m1.cache.dflt.0" files which is really strange. Does someone have a better explanation for this M1 cache and how connector can have lots of M1 cache files when there is no actual cache?
The connectorid.m1.cache.dflt.f (data files), connectorid.m1.size.dflt (size file), andconnectorid.tmp.dflt files make up another, generally smaller cache (by default up to 10 files of up to 1MB each). This cache is used to remember correlated events (which are also known as M1s) until all of the events that they refer to have been accepted by the Manager and assigned numeric IDs. (see ID Map). This cached is managed by the M1Processor component.
Here's what I remember for connectors:
The m1 cache contains the internal connector events to send to ESM, it does not contain the parsed/processed events. Here is a doc that briefly mentions this:
queue.syslogd.N cache is the inbound cache to the connector itself - in a healthy connector, it should be low in size but the file index number may increment over time (for instance, when the connector receives a spike in traffic and has to cache all that incoming data for the connector to parse). If this cache is growing rapidly, additional connectors may be needed to support the event load.
<destinationId>.cache.dflt.N cache is the normalized event cache - the connector has completed all parsing, mapping, categorization, etc. and creates cache files to send to ESM (or logger, or other destination). Too many of these either means the manager (or other destination) is overloaded (or it has asked the connector to pause sending), the connector is processing events faster than can be sent but is still able to parse and process them, or, the destination event transport is down.
Hope that helps.