ESM Event ID and Event Forwarding
ESM event ID structure
In ESM an eventID is a 64 bit long. These 64 bits are divided into two sections:
- The upper 16 bits identify the path that the event took as it was forwarded between ESM managers
- The lower 48 bits uniquely identify the event in the source ESM.
Note that since Java has only a signed 64 bit value, in multi-tier deployment that use the higher 16 bit, event IDs may be presented as negative.
How does the ID change when the event is forwarded?
When forwarding an event, the upper 16 bits serve the purpose of identifying the exact path that the event went thru in with each ESM manager traversed being a "hop". By default:
- a "hop" is represented by 3 bits, which allow 7 managers in a tier (but see caveat below).
- 16 bits allow allows 5 layers tiers (3 bits x 5 hops = 15 bits with 1 to spare).
A manager will reject events sent from forwarders beyond the maximum managers per tier (7 by default). A Java exception is written to server.log from the class EventPathInfoManager. See attachment for an example.
The originating ESMs are the 1st hop and store their ID in the first hop field. Therefore in the common scenario of several ESMs forwarding to a single master ESM one "hop" field is used.
One can change the default using the server.properties file attribute "server.pathinfo.single.bits". By changing the value from 3 to a different number, the number of managers in a tier can be increased to 2^n-1. For example, 4 would yield 15 managers. Conversely, the number of layers will be reduced so that they fit in 16 bits, so 4 would allow 4 layers.
One result of this mechanism is that the event id for an event changes when moving between managers:
- The lower 48 bits are preserved.
- The upper 16 bits change to reflects the path of managers that the event traversed as described above.
An important note:
- The same value have to be set for all managers in the hierarchy and cannot be changed after events have already been collected. The results are unexpected.
Forwarding information about base events
ESM uses the field "base_event_ids" to store a list of the base events related to a correlated events. This field is available as a regular field in the ArcSight Command Center search pane and is not directly accessible in the ArcSight console and is used internally.
The ESM forwarding connector forwards the "base_event_ids" and modified the event ids in the same manner described above as the event id is modified. However the field is not forwarded by other connectors. So for example in a scenario in which a forwarding connector forwards to a syslog connector which in turn forwards to another ESM system, this field would not be preserved.
As a side not, to view "base_event_ids" in the console create a variable using the EvaluateVelocityTemplate function with the template "$baseEventIds".
Working the with the event ID on the master ESM?
As a result, the original event id at the originating manager would include only the lower 48 bits (with the upper 16 bits set to zero). This means that to manually locate an event in the originating manager, one would have to extract the lower 48 bits from the event id using a bit mask or a modulo operator.
Another options would be to assign the original event ID to a custom field on the lower tier so that a clean 48 bit event ID is available on subsequent layers. Fortunately, all you need is a single pre-persistence rule on the lower tier ESM to copy the event ID of every correlated event to a custom field.
Ivan Malinjak from the ArcSight support team provided the following detailed instructions on how to do that. Thanks Ivan!
On the lower tier ESM:
1. Create “Pre-persistence rule”.
2. To limit this rule only for Correlation Events, set in “Conditions” tab: “Type = Correlation”
3. In “Actions” tab set – On Every Event -> Add -> Set Event Field
4. In the field “Flex number 1” type: $eventId -> OK -> Apply
On the TOP ESM:
1. In the Viewer panel Add column: “Flex Number 1”
2. Field “Flex Number 1” now shows Event ID of the source ESM manager.
3. Additionally this “field set” can be saved for future use in another resources.