Idea ID: 2821916

ArcSight ESM: Direct Kafka Consumption

Status : New Idea
9 months ago

[Brief Description]

This is a key request for large environments where kafka is being used as the message bus, it aggregates all the audit trails and then the different security solutions, are jut subscribed to the desired kafka clusters and topics.

 

In our scenario, the security solution (ArcSight ESM) should be able to consume messages from kafka directly without deploying a middleware component - such as the arcsight agents.

This is the target scenario:

 

ArcSight Agents (Producer) --> KAFKA Clusters {Topics} <-- ArcSight ESM.

*Note that the arcsight agents collect the logs, process them and publish messages into a kafka topic(s).

*Such messages are stored in plain text following the CEF.

 

Without this feature, we have to deploy 2 agents per each data feed .. which ends with double efforts:

 

ArcSight Agents (Producer) --> Kafka Cluster {Topics} <-- ArcSight Agents (Consumer) --> ArcSight ESM.

 

Have in mind that for large deployments, this is a mess, for instance, I have + 200 producer agents, should I deploy another 200 ones just to consume data already processed ? (which is already normalized, filtered and aggregated)

 

I believe that this is a key feature to have, in fact, many other SIEMs has already implemented (i.e. Splunk, Qradar, Sentinel ..)

 

[Benefits / Value]

 

Not only it's an enhancement for the data pipeline architecture and saves lots of unnecessary components but also opens the solution to a well known and reliable message bus such as kafka - which has become "de facto data streaming ". 

 

[Design details]

As said previously, this is the target scenario:

 

ArcSight Agents (Producer) --> KAFKA Clusters {Topics} <-- ArcSight ESM.

 

*Note that the arcsight agents collect the logs, process them and publish messages into a kafka topic(s).

*Such messages are stored in plain text following the CEF.

*ArcSight ESM will act as a data consumer for cef messages configured on kafka topics.

*ArcSight ESM should be able to consume data from different kafka clusters and its desired topics.

 

Hope that makes sense,

 

Regards,

 

Karl Alfaro.

  • Hi,

    I have to highlight that the feature is more or less present on 7.2p1 but it's not fully working and with some constraints (just 1 kafka cluster and up to 25 topics).

     

    https://community.microfocus.com/t5/ArcSight-User-Discussions/esm-to-kafka/m-p/2760400#

     

    Regards,

    Karl.