ESM Annotation Event Generator

over 5 years ago

Hi all,

Internal events generated upon event annotation is something that I've been wanting to see in ESM for some time. Although there's a feature request for this (NGS-11310), I decided to developed a work-around as part of a SIEM gamification project.

Annotation Event Generator (AEG) is a script written in python. It uses a mysql module to facilitate the database connection. When executed, the script carries out the following operations:

  1. Query the annotation table for the max modification_time (this is the same technique used by time-based DB SmartConnectors)
  2. Store the current max timestamp in a local file for next execution
  3. Retrieve all rows in the annotation table with a modification_time newer than last recorded max timestamp
  4. Generate syslog events with annotation details for every retrieved row (event_id).

With this method, annotation events can be generated without any modification to the database, however, since the script relies on a query (not a DB-trigger) it is critical that the script is executed frequently. An event that is annotated multiple times between script executions only generates a single event with the stage that is current. Although the version field can be used to detect when an event is annotated multiple times between script executions (version increment higher than one), it is not guaranteed that the annotation details are maintained in the audit_trail field. The audit_trail field has a size limit of 2048 bytes in the ESM DB while the current_comment field has a size limit of 1024 bytes - this means that information in the audit_trail field could easily be lost/overwritten through multiple edits.

Annotation events allows a user to:

  1. Automate real-time response based on analyst workflow
  2. Send notifications based on event annotation stages
  3. Perform QA/review of analyst annotations
  4. Track annotation metrics
    • Time to review (per analyst): Time diff between MRT and event marked as reviewed
    • Time to respond (per analyst): Time diff between MRT and initial stage
    • Time to close (per analyst): Time diff between event marked as reviewed and final stage

Example scenario

A high priority event comes in to the main channel indicating that a host has been infected with a highly disruptive malware (e.g. cryptolocker). When an analyst sees this, the event is assigned the "re-image" stage (annotation). An annotation event is generated, matching a rule condition, causing a "re-image" rule to trigger. When triggered, the rule executes a script that makes API calls to the endpoint infrastructure (McAfee ePO) and assigns a "re-image" tag to the asset. Once the infected asset has been tagged, a policy is used to limit connectivity (block all access other than re-image instructions and helpdesk sites). Note: The existing methods to do this without annotation events include integration command (requiring analysts to take action before/after annotation) and scheduled rules (not real-time)

The attached script can be used to generate annotation events. Latest version of the python mysql module is available here:

Script usage

aeg_<ver>.py <file_path> <db_hostname> <db_username> <db_password> <db_port> <db_name> <syslog_host> <syslog_port>

ex. python aeg_<ver>.py /tmp/ arcsight arcsightpw 3306 arcsight 10514

Annotation Event CEF mappings

Name: "Annotation Event"

Vendor: "HP"

Product: "AEG"

SourceHost: server fqdn

DestinationUser: modified_by

CustomString1: audit_trail

CustomString2: current_comment

CustomString3: stage

CustomString4: flag_labels (comma separated list of annotation flags)

CustomDate1: modification_time

CustomDate2: event_mrt

CustomNumber1: version

CustomNumber2: event_id

CustomNumber3: flags (int)

Let me know if you have any questions or suggestions.



Update: ESM content written by attached (AEG.arb)

Comment List
Related Discussions