ArcSight ESM: Direct Kafka Consumption

Idea ID 2821916

ArcSight ESM: Direct Kafka Consumption

[Brief Description]

This is a key request for large environments where kafka is being used as the message bus, it aggregates all the audit trails and then the different security solutions, are jut subscribed to the desired kafka clusters and topics.

 

In our scenario, the security solution (ArcSight ESM) should be able to consume messages from kafka directly without deploying a middleware component - such as the arcsight agents.

This is the target scenario:

 

ArcSight Agents (Producer) --> KAFKA Clusters {Topics} <-- ArcSight ESM.

*Note that the arcsight agents collect the logs, process them and publish messages into a kafka topic(s).

*Such messages are stored in plain text following the CEF.

 

Without this feature, we have to deploy 2 agents per each data feed .. which ends with double efforts:

 

ArcSight Agents (Producer) --> Kafka Cluster {Topics} <-- ArcSight Agents (Consumer) --> ArcSight ESM.

 

Have in mind that for large deployments, this is a mess, for instance, I have + 200 producer agents, should I deploy another 200 ones just to consume data already processed ? (which is already normalized, filtered and aggregated)

 

I believe that this is a key feature to have, in fact, many other SIEMs has already implemented (i.e. Splunk, Qradar, Sentinel ..)

 

[Benefits / Value]

 

Not only it's an enhancement for the data pipeline architecture and saves lots of unnecessary components but also opens the solution to a well known and reliable message bus such as kafka - which has become "de facto data streaming ". 

 

[Design details]

As said previously, this is the target scenario:

 

ArcSight Agents (Producer) --> KAFKA Clusters {Topics} <-- ArcSight ESM.

 

*Note that the arcsight agents collect the logs, process them and publish messages into a kafka topic(s).

*Such messages are stored in plain text following the CEF.

*ArcSight ESM will act as a data consumer for cef messages configured on kafka topics.

*ArcSight ESM should be able to consume data from different kafka clusters and its desired topics.

 

Hope that makes sense,

 

Regards,

 

Karl Alfaro.

1 Comment
Honored Contributor.
Honored Contributor.

Hi,

I have to highlight that the feature is more or less present on 7.2p1 but it's not fully working and with some constraints (just 1 kafka cluster and up to 25 topics).

 

https://community.microfocus.com/t5/ArcSight-User-Discussions/esm-to-kafka/m-p/2760400#

 

Regards,

Karl.

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.