ArcSight Connector Cache Management
In ArcSight architecture, ArcSight connectors cache events locally, to prevent event loss, whenever these events cannot be immediately sent to ESM or logger destinations.
Log flow between the connectors and ESM or Logger can be disrupted because of several reasons, including bandwidth limitation (slow speed links, especially in case of MSS) or inability of ESM or Logger to process incoming event flow (large setups with high variation in incoming events).
On such occasions, connectors store these events on the local disk, called event cache, till they can retransmit those events to ESM or logger.
While caching prevents events from getting lost, it is important to identify instances of caching and the extent to which caching occurs. The default cache size configured in arcsight connectors, though sufficient, is not unlimited, and hence uncontrolled/unmonitored caching over longer periods can lead to prolonged delays in events reaching ESM/Logger or events getting dropped/lost forever.
In the article below, we present some useful mechanisms to detect whether connectors in your arcsight setup are caching events, to what extent, and whether you are losing any events at all.
ArcSight Internal Events
ArcSight connectors generate various internal events that can be used to detect caching. Below is a listing of event codes that are present in the deviceEventClassId field.
|1||agent:019||Connector is caching events because they could not be immediately transmitted to the Manager.|
|2||agent:020||Connector has emptied its cache of events.|
|3||agent:029||Connector was forced to drop its cached data.|
ArcSight Internal Events with deviceEventClassId=agent:019, indicate that the connector has cached events since they could not be sent to ESM or Logger. This event also contains the count of events that are cached. On a daily basis, monitor the count of occurrence of this event on all connectors and also the maximum cache size on these connectors.
Events with deviceEventClassId=agent:020 indicate that the connector has emptied its cache. This does not mean that the connector has emptied its cache by dropping the cached events. This means that the connector has emptied its cache by successfully sending all cached logs to ESM or logger. On a daily basis, monitor that for each connector that has generated one or more agent:019 events, you have at least one agent:020 event. If not, it means that your connectors’ cache has not been emptied and you may see events older than a day on ESM/Logger.
Events with deviceEventClassId=agent:029 indicate that your connector’s cache was full and it had to drop some or all of the cached logs to make space in the cache for newer events. This event means that you have lost some logs. This also means that your connector is receiving too many events and is not able to process and send them to ESM or logger. You should consider dividing the load among two or more connectors and also measure the overall EPS received at your ESM or Loggers.
Connector Cache Settings
ArcSight connectors have a default setting to use 1 GB on the local disk for caching events. On average, 1 GB of cache space is sufficient to hold around 15 Million events (~175 EPS averaged over a day). Depending on the type of logs (firewall/OS/IPS) and some other connector settings (e.g. Turbo Mode), your event cache may get full anywhere between 10-15 Million events. Once the cache is full, ArcSight connectors will drop events on FIFO basis.
Track event cache size on all connectors using ArcSight internal events mentioned above (deviceEventClassId=agent:019). If your setup has connectors that frequently touch 8-10 Million cache size mark, you should consider increasing their cache size from the default 1 GB.
However, increasing cache size does not ensure that events will not be dropped. Increasing cache size only helps you retain the events temporarily. You will have to fine tune or upgrade the setup so that all connectors in your setup cache minimal events.
Re: ArcSight Connector Cache Management
Need every one help ,
Suppose I have 1GB of cache , and if my cache is filled up to 70% , I should get an alert that my cache is 70% filled, is there any way we can monitor this.