esm to kafka

Hi All,

I want to know if there is a process to get ESM 7.0 to consume events from a normal Kafka topic? This us not eventbroker, or TH. But a normal Kafka topic.

If possible, how would one go about to achieve this?

Kind regards.

  • Hello,

    yes it is via "ArcSight FlexConnector for Kafka" and for that please take a look on the pdf document from this link :

    https://community.microfocus.com/t5/ArcSight-Connectors/ArcSight-FlexConnector-for-Kafka/ta-p/2689807

     

    Best Regards,

     

    Daniel

  • Hi All,

     

    This is a valid request: the security solution (ArcSight ESM) should be able to consume messages from kafka directly without deploying a middleware component - such as the arcsight agents.

     

    Have in mind the following scenario:

     

    ArcSight Agents (Producer) --> KAFKA Cluster {Topics} <-- ArcSight ESM.

    *Note that the arcsight agents collect the logs, process them and publish messages into a kafka topic(s).

    *Such messages are stored in plain text following the CEF.

     

    Without this feature, we have to deploy 2 agents per each data feed .. which ends with double efforts:

     

    ArcSight Agents (Producer) --> Kafka Cluster {Topics} <-- ArcSight Agent Consumer --> ArcSight ESM.

     

    Have in mind that for large deployments, this is a mess, for instance, I have 200 producer agents, should I deploy another 200 ones just to consume data already processed ? (normalized, filtered and aggregated)

     

    I believe that this is a key feature to have, in fact, many other SIEMs has already implemented (i.e. Splunk, Qradar, Sentinel ..)

     

    Hope that makes sense,

     

    Regards,

     

    Karl.

  • BTW, it seems that the feature is more or less available on 7.2p1, we've proceed to configure it "successfully", but when starting the manager, the data consumption crashed. Furthermore this also revealed another limitations:

     

    * Data consumption is limited to just 1 kafka cluster AND up to 25 topics.

     

    If the feature is ready to work, it makes no sense to put such constraints, ArcSight ESM should be able to consume data from many kafka clusters and several topics. The only requirement is that such data should be properly parsed on CEF.

     

    To finish, when configuring the feature we miss the consumer group parameter, is there a way to apply it?

    I'm afraid that without this, arcsight esm consumes from the "default" cg, which is not a good practice and may lead to data loss.

     

    Regards,

     

    Karl.