Highlighted
hhogden24
New Member.
2569 views

ArcSight Connector watchdog FlexConnector

We've been using this internally for a while now, and I wanted to share in case this is valuable to anyone.

Before Use - PLEASE READ the documentation provided about filtering out from your destination, and pointing the flex to a temporary invalid location prior to the filter out being applied so you do not end up with your entire agent.log streaming into your destination.

We use it for some pretty specific use cases (specifically looking for the inbound syslog queue size being exceeded and dropping events on the floor, memory yellow and red zone events) so it may not be for everyone.

This will work on both a connector appliance (directions for this is specifically provided in the attached doc) and on a software connector (a few steps will vary based on your OS and where you have them installed).

I consider this to be a beta version and am more than willing to incorporate feedback. There are alternate/better ways to do many of the things being accomplished here - please contribute to the discussion and let me know what those are. If we don't need this watchdog to get at some of this info, that would be great. Also please feel free to correct my chosen nomenclature being used in these events, even if it's just to increase the accuracy in describing what is going on.

Here's an extract from the doc that shows what agent.log data is being extracted and sent, and in what fields:

All Events

Sample Event:
[2013-11-19 09:44:56,329][WARN ][default.com.arcsight.agent.lg.g][dropReader] Dropped [1000] alerts in [52] ms. for cache [373pMIz4BABCABn6QLl7dOw== (total dropped = 14507)

ArcSight ESM Field

  1. agent.log field

endTime

Time_stamp

deviceReceiptTime

Time_stamp

deviceVendor

‘ArcSight’

deviceProduct

‘ConnectorAgentLog’

filePath

Connector Folder Name

deviceProcessName

class

deviceAction

method

Specific Event Types

dropReader

Sample Event:
[2013-11-19 09:44:56,329][WARN ][default.com.arcsight.agent.lg.g][dropReader] Dropped [1000] alerts in [52] ms. for cache [373pMIz4BABCABn6QLl7dOw== (total dropped = 14507)

ArcSight ESM Field

  1. agent.log field

deviceEventClassId

dropReader1

name

‘Forwarder Cache Dropping Events’

deviceCustomNumber1

Dropped events

deviceCustomNumber2

Number of milliseconds it took to drop the events

deviceCustomString1

Cache ID

deviceCustomNumber1Label

‘Dropped Events’

deviceCustomNumber2Label

‘Milliseconds’

deviceCustomString1Label

‘cache ID’

addBatch1

Sample Event:
[2013-11-19 09:39:21,201][INFO ][default.com.arcsight.agent.of.i$c_][addBatch] Number of waiting events is too large [610], so stalling incoming sender for [false|0||] (stalled [1] times)

ArcSight ESM Field

  1. agent.log field

deviceEventClassId

addBatch1

name

‘Receiver Caching’

deviceCustomNumber1

Waiting Events

deviceCustomNumber2

Number of times the receiver has stalled

deviceCustomString1

Stalling for

deviceCustomNumber1Label

‘Receiver Cache’

deviceCustomNumber2Label

‘Times Stalled’

deviceCustomString1Label

‘Stalling for’

addBatch2

Sample Event:
[2013-11-19 09:39:21,379][INFO ][default.com.arcsight.agent.of.i$c_][addBatch] List of waiting events was too large for [false|0||], but now is back within bounds [210]

ArcSight ESM Field

  1. agent.log field

deviceEventClassId

addBatch2

name

‘Receiver Clearing Cache’

deviceCustomNumber1

Current cache

deviceCustomString1

PLACEHOLDER

deviceCustomNumber1Label

‘Receiver Cache’

deviceCustomString1Label

‘PLACEHOLDER’

enableDropMode

Sample Event:
[2013-12-05 11:17:41,166][WARN ][default.com.arcsight.agent.util.c.c][enableDropMode] File queue now dropping events (101 files)

ArcSight ESM Field

  1. agent.log field

deviceEventClassId

enableDropMode1

name

Receiver Cache Dropping Events

deviceCustomNumber1

Number of files present in receiver cache

deviceCustomNumber1Label

‘Current Cache’

disableDropMode

Sample Event:
[2013-12-05 11:17:41,322][INFO ][default.com.arcsight.agent.util.c.c][disableDropMode] File queue no longer dropping events (100 files)

ArcSight ESM Field

  1. agent.log field

deviceEventClassId

disableDropMode1

name

Receiver Cache Returned To Max

deviceCustomNumber1

Number of files present in receiver cache

deviceCustomNumber1Label

Current Cache

start

Sample Event:
[2013-12-05 10:45:22,933][INFO ][default.com.arcsight.agent.baseagents.i.e][start] Successfully started the file reader for the file[/opt/arcsight/connector_1/current/user/agent/agentdata/lXcxDi8BABCAEizyuOrv7w==.syslogd.191413] with startatend[false]

ArcSight ESM Field

  1. agent.log field

deviceEventClassId

Start1

name

‘File Successfully Read By Connector’

flexString1

Name of the file that was read

flexString2

Is the connector reading starting at the end of the file?

flexString1Label

‘File Name’

flexString2Label

‘startatend’

onMemoryShortage

Sample Event:
[2013-11-19 10:15:38,114][INFO ][default.com.arcsight.agent.Agent$8][onMemoryShortage] Memory warning detected: Memory has reached yellow zone

ArcSight ESM Field

  1. agent.log field

deviceEventClassId

onMemoryShortage1

name

Memory Yellow Zone

message

Event Message

memoryShortageOccurring

Sample Event:
[2013-11-19 11:20:47,733][FATAL][default.com.arcsight.agent.Agent$1$0][memoryShortageOccurring]  Memory usage in red zone after garbage collection.

ArcSight ESM Field

  1. agent.log field

deviceEventClassId

memoryShortageOccurring1

name

Memory Red Zone

message

Event Message

DEFAULT

ArcSight ESM Field

  1. agent.log field

deviceEventClassId

DefaultParser

Message

Event Message

5 Replies
Acclaimed Contributor.. lless Acclaimed Contributor..
Acclaimed Contributor..

Re: ArcSight Connector watchdog FlexConnector

This looks interesting!

You see We! Analyze By Or Cohen | SourceForge.net tools?

0 Likes
hhogden24
New Member.

Re: ArcSight Connector watchdog FlexConnector

I had not seen that - thanks for pointing it out!

0 Likes
drbeanz Absent Member.
Absent Member.

Re: ArcSight Connector watchdog FlexConnector

Harold,

Very interesting, thanks for sharing this! I created a Python script to configure the agent.properties file, speeding the process and minimizing human errors. The script should work on ConApps as well (since I used older Python syntax), but requires SSH access.

Additionally, I changed my Filter Out string to the following:

deviceEventClassId EQ "DefaultParser" Or deviceEventClassId Is "NULL"

Jordan

Outstanding Contributor.. andrew.dalbor Outstanding Contributor..
Outstanding Contributor..

Re: ArcSight Connector watchdog FlexConnector

Awesome post and awesome work Harold.  Thanks a ton for sharing!

Going to give this a go hopefully this week.

Seems like there should either a) already be a connector that does this OR b) some way for those events to make it to a destination or a notification

0 Likes
Trusted Contributor.. emilian.darie1 Trusted Contributor..
Trusted Contributor..

Re: ArcSight Connector watchdog FlexConnector

Hello, looks nice.I found it as I implemented some impstats and I saw some issues on some pipe.

I am in the moment when I need to see if the connector can not process so many events and it reached the limit, which makes the pipe in front of him to reach its limits.

This looks usefull-I ll share some afterthoughts later, but nice one again.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.