We've been using this internally for a while now, and I wanted to share in case this is valuable to anyone.
Before Use - PLEASE READ the documentation provided about filtering out from your destination, and pointing the flex to a temporary invalid location prior to the filter out being applied so you do not end up with your entire agent.log streaming into your destination.
We use it for some pretty specific use cases (specifically looking for the inbound syslog queue size being exceeded and dropping events on the floor, memory yellow and red zone events) so it may not be for everyone.
This will work on both a connector appliance (directions for this is specifically provided in the attached doc) and on a software connector (a few steps will vary based on your OS and where you have them installed).
I consider this to be a beta version and am more than willing to incorporate feedback. There are alternate/better ways to do many of the things being accomplished here - please contribute to the discussion and let me know what those are. If we don't need this watchdog to get at some of this info, that would be great. Also please feel free to correct my chosen nomenclature being used in these events, even if it's just to increase the accuracy in describing what is going on.
Here's an extract from the doc that shows what agent.log data is being extracted and sent, and in what fields:
All Events
Sample Event:
[2013-11-19 09:44:56,329][WARN ][default.com.arcsight.agent.lg.g][dropReader] Dropped [1000] alerts in [52] ms. for cache [373pMIz4BABCABn6QLl7dOw== (total dropped = 14507)
ArcSight ESM Field | - agent.log field
|
endTime | Time_stamp |
deviceReceiptTime | Time_stamp |
deviceVendor | ‘ArcSight’ |
deviceProduct | ‘ConnectorAgentLog’ |
filePath | Connector Folder Name |
deviceProcessName | class |
deviceAction | method |
Specific Event Types
dropReader
Sample Event:
[2013-11-19 09:44:56,329][WARN ][default.com.arcsight.agent.lg.g][dropReader] Dropped [1000] alerts in [52] ms. for cache [373pMIz4BABCABn6QLl7dOw== (total dropped = 14507)
ArcSight ESM Field | - agent.log field
|
deviceEventClassId | dropReader1 |
name | ‘Forwarder Cache Dropping Events’ |
deviceCustomNumber1 | Dropped events |
deviceCustomNumber2 | Number of milliseconds it took to drop the events |
deviceCustomString1 | Cache ID |
deviceCustomNumber1Label | ‘Dropped Events’ |
deviceCustomNumber2Label | ‘Milliseconds’ |
deviceCustomString1Label | ‘cache ID’ |
addBatch1
Sample Event:
[2013-11-19 09:39:21,201][INFO ][default.com.arcsight.agent.of.i$c_][addBatch] Number of waiting events is too large [610], so stalling incoming sender for [false|0||] (stalled [1] times)
ArcSight ESM Field | - agent.log field
|
deviceEventClassId | addBatch1 |
name | ‘Receiver Caching’ |
deviceCustomNumber1 | Waiting Events |
deviceCustomNumber2 | Number of times the receiver has stalled |
deviceCustomString1 | Stalling for |
deviceCustomNumber1Label | ‘Receiver Cache’ |
deviceCustomNumber2Label | ‘Times Stalled’ |
deviceCustomString1Label | ‘Stalling for’ |
addBatch2
Sample Event:
[2013-11-19 09:39:21,379][INFO ][default.com.arcsight.agent.of.i$c_][addBatch] List of waiting events was too large for [false|0||], but now is back within bounds [210]
ArcSight ESM Field | - agent.log field
|
deviceEventClassId | addBatch2 |
name | ‘Receiver Clearing Cache’ |
deviceCustomNumber1 | Current cache |
deviceCustomString1 | PLACEHOLDER |
deviceCustomNumber1Label | ‘Receiver Cache’ |
deviceCustomString1Label | ‘PLACEHOLDER’ |
enableDropMode
Sample Event:
[2013-12-05 11:17:41,166][WARN ][default.com.arcsight.agent.util.c.c][enableDropMode] File queue now dropping events (101 files)
ArcSight ESM Field | - agent.log field
|
deviceEventClassId | enableDropMode1 |
name | Receiver Cache Dropping Events |
deviceCustomNumber1 | Number of files present in receiver cache |
deviceCustomNumber1Label | ‘Current Cache’ |
disableDropMode
Sample Event:
[2013-12-05 11:17:41,322][INFO ][default.com.arcsight.agent.util.c.c][disableDropMode] File queue no longer dropping events (100 files)
ArcSight ESM Field | - agent.log field
|
deviceEventClassId | disableDropMode1 |
name | Receiver Cache Returned To Max |
deviceCustomNumber1 | Number of files present in receiver cache |
deviceCustomNumber1Label | Current Cache |
start
Sample Event:
[2013-12-05 10:45:22,933][INFO ][default.com.arcsight.agent.baseagents.i.e][start] Successfully started the file reader for the file[/opt/arcsight/connector_1/current/user/agent/agentdata/lXcxDi8BABCAEizyuOrv7w==.syslogd.191413] with startatend[false]
ArcSight ESM Field | - agent.log field
|
deviceEventClassId | Start1 |
name | ‘File Successfully Read By Connector’ |
flexString1 | Name of the file that was read |
flexString2 | Is the connector reading starting at the end of the file? |
flexString1Label | ‘File Name’ |
flexString2Label | ‘startatend’ |
onMemoryShortage
Sample Event:
[2013-11-19 10:15:38,114][INFO ][default.com.arcsight.agent.Agent$8][onMemoryShortage] Memory warning detected: Memory has reached yellow zone
ArcSight ESM Field | - agent.log field
|
deviceEventClassId | onMemoryShortage1 |
name | Memory Yellow Zone |
message | Event Message |
memoryShortageOccurring
Sample Event:
[2013-11-19 11:20:47,733][FATAL][default.com.arcsight.agent.Agent$1$0][memoryShortageOccurring] Memory usage in red zone after garbage collection.
ArcSight ESM Field | - agent.log field
|
deviceEventClassId | memoryShortageOccurring1 |
name | Memory Red Zone |
message | Event Message |
DEFAULT
ArcSight ESM Field | - agent.log field
|
deviceEventClassId | DefaultParser |
Message | Event Message |
