New Ranks & Badges For The Community!
Notice something different? The ranks and associated badges have gone "Star Fleet". See what they all mean HERE
Highlighted
Lieutenant Commander
Lieutenant Commander
318 views

esm to kafka

Hi All,

I want to know if there is a process to get ESM 7.0 to consume events from a normal Kafka topic? This us not eventbroker, or TH. But a normal Kafka topic.

If possible, how would one go about to achieve this?

Kind regards.

0 Likes
3 Replies
Highlighted
Fleet Admiral
Fleet Admiral

Hello,

yes it is via "ArcSight FlexConnector for Kafka" and for that please take a look on the pdf document from this link :

https://community.microfocus.com/t5/ArcSight-Connectors/ArcSight-FlexConnector-for-Kafka/ta-p/2689807

 

Best Regards,

 

Daniel

0 Likes
Highlighted
Vice Admiral
Vice Admiral

Hi All,

 

This is a valid request: the security solution (ArcSight ESM) should be able to consume messages from kafka directly without deploying a middleware component - such as the arcsight agents.

 

Have in mind the following scenario:

 

ArcSight Agents (Producer) --> KAFKA Cluster {Topics} <-- ArcSight ESM.

*Note that the arcsight agents collect the logs, process them and publish messages into a kafka topic(s).

*Such messages are stored in plain text following the CEF.

 

Without this feature, we have to deploy 2 agents per each data feed .. which ends with double efforts:

 

ArcSight Agents (Producer) --> Kafka Cluster {Topics} <-- ArcSight Agent Consumer --> ArcSight ESM.

 

Have in mind that for large deployments, this is a mess, for instance, I have + 200 producer agents, should I deploy another 200 ones just to consume data already processed ? (normalized, filtered and aggregated)

 

I believe that this is a key feature to have, in fact, many other SIEMs has already implemented (i.e. Splunk, Qradar, Sentinel ..)

 

Hope that makes sense,

 

Regards,

 

Karl.

0 Likes
Vice Admiral
Vice Admiral

BTW, it seems that the feature is more or less available on 7.2p1, we've proceed to configure it "successfully", but when starting the manager, the data consumption crashed. Furthermore this also revealed another limitations:

 

* Data consumption is limited to just 1 kafka cluster AND up to 25 topics.

 

If the feature is ready to work, it makes no sense to put such constraints, ArcSight ESM should be able to consume data from many kafka clusters and several topics. The only requirement is that such data should be properly parsed on CEF.

 

To finish, when configuring the feature we miss the consumer group parameter, is there a way to apply it?

I'm afraid that without this, arcsight esm consumes from the "default" cg, which is not a good practice and may lead to data loss.

 

Regards,

 

Karl.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.