Highlighted
jng1 Absent Member.
Absent Member.
2551 views

"Connector dropping events" - How does this work?

Jump to solution

hi,

recently discovered that my connectors are dropping events when i did a search in the device and also in the ArcSight Console.

what actually puzzled me was that, why is the rule for detecting dropped/dropping events not firing?

opening up the rule conditions i see the following:

rule-droppingEvents.png

which doesnt really tells me how it really work, other than the trigger "Device Event category = /Agent/Cache/Dropped". this tells me that the event is actually being sent by the connector.

my question is, how exactly does the connector knows that it is dropping events and the logic/algorithm involved? i do not need to know them in code, just a summary overview would be good, so that i can analyze or deduce what are the limitations involved.

Labels (1)
0 Likes
1 Solution

Accepted Solutions
Absent Member.. Dean Farrington Absent Member..
Absent Member..

Re: "Connector dropping events" - How does this work?

Jump to solution

Each connector has a cache setting that is specific to the destination (connectors with 2 destinations will have a separate cache for each destination). If the connector cannot move the events fast enough then they start going into cache, this is a normal thing and can help your infrastructure absorb spikes in event rate. The problems can come if the cache is not able to clear itself.

Once the maximum size of the cache is reached the oldest events will start to be purged to make room for more incoming events. When this happens you get the message that indicates that the connector is dropping events.

If it is truly a connector having the issue you need to look at the throughput on the connector and the connector configuration to see if you can cause the events to come through quicker or split the data across more than one connector depending on the issue.

If you have SuperConnectors then ESM also maintains a 1 Gb cache file per SuperConnector that buffers events on the source ESM side (before the events get into the SuperConnector). If that cache gets full you can also see "Dropping Events" messages. If you have removed a SuperConnector the cache file is not automatically removed and it will continue to buffer events. This leads to the Dropped Events messages as the data has nowhere to go.

If you have run into this you need to manually remove the cache file and restart ESM. If the SuperConnector does not exist the cache file will not be recreated and the unnecessary caching will stop.

0 Likes
4 Replies
Absent Member.. Dean Farrington Absent Member..
Absent Member..

Re: "Connector dropping events" - How does this work?

Jump to solution

Each connector has a cache setting that is specific to the destination (connectors with 2 destinations will have a separate cache for each destination). If the connector cannot move the events fast enough then they start going into cache, this is a normal thing and can help your infrastructure absorb spikes in event rate. The problems can come if the cache is not able to clear itself.

Once the maximum size of the cache is reached the oldest events will start to be purged to make room for more incoming events. When this happens you get the message that indicates that the connector is dropping events.

If it is truly a connector having the issue you need to look at the throughput on the connector and the connector configuration to see if you can cause the events to come through quicker or split the data across more than one connector depending on the issue.

If you have SuperConnectors then ESM also maintains a 1 Gb cache file per SuperConnector that buffers events on the source ESM side (before the events get into the SuperConnector). If that cache gets full you can also see "Dropping Events" messages. If you have removed a SuperConnector the cache file is not automatically removed and it will continue to buffer events. This leads to the Dropped Events messages as the data has nowhere to go.

If you have run into this you need to manually remove the cache file and restart ESM. If the SuperConnector does not exist the cache file will not be recreated and the unnecessary caching will stop.

0 Likes
jdolan Frequent Contributor.
Frequent Contributor.

Re: "Connector dropping events" - How does this work?

Jump to solution

If a connector is dropping events and producing the agent cache dropped events should thier also be entry in the "Queue Drop Count" in the connectors status? I am getting agent cache dropped on a connector but am not seeing anything in the "Queue Drop Count".

0 Likes
alexandros_n Honored Contributor.
Honored Contributor.

Re: "Connector dropping events" - How does this work?

Jump to solution

The term 'queue' is used for incoming traffic (usually of a syslog daemon connector). 'Cache' is for the outgoing to the destinations.

0 Likes
jng1 Absent Member.
Absent Member.

Re: "Connector dropping events" - How does this work?

Jump to solution

Hi farridem,

thanks for the insightful and detailed explanation, i am much more clearer in this aspect of dropping events.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.