Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE

Massively scalable centralized event processing with ease

Security_Guest Frequent Contributor.
Frequent Contributor.
1 0 1,717

Guest post by Viktor Doundakov, Product Manager 

SmartConnectors in Micro Focus ArcSight Data Platform (ADP) simplify onboarding of security data from multiple sources; collecting, normalizing, categorizing, enriching, aggregating, and filtering that data at ingestion for ease and speed of consumption by downstream applications. These connectors can handle heavy flows of data compared to competing solutions and have advanced caching capabilities to help with data spikes. Even so, as those flows increase it can become necessary to add additional connectors in order to keep up with the high flow.

Massively scalable centralized event processing with ease.pngWhile you can attempt to address increases of raw data flows by having administrators deploy extra hardware resources at your data ingestion points, trying to handle data spikes or data storms with traditional standalone connectors involves significant over-provisioning of the connectors’ hardware. From a processes and personnel perspective that would mean that you would need to have IT personnel available 24x7 with the expertise to deploy and configure those additional connectors and manage the distribution of their workload. To help reduce the need to have expert personnel available on standby, Micro Focus has simplified the ability to scale out in the recent 2.30 release of ArcSight Data Platform

The new Syslog Collector and Connector in Event Broker (CEB) features in ADP give you the ability to move the computationally intensive data normalization and enrichment processes from standalone connectors into the massively scalable Event Broker cluster infrastructure. It’s in this cluster infrastructure where ArcSight Management Center (ArcMC) takes advantage of the fast resource allocation and management provided by the ArcSight Event Broker Kubernetes foundation, enabling you to achieve better hardware utilization, while reducing the computational load on the raw syslog data collection layer. 

These new features work together in two stages to complement, augment, or even take over work performed by the SmartConnectors deployed at your data sources. In the first stage, ADP uses a lightweight Syslog Collector to collect the raw event data from multiple sources and post those data streams as Kafka topics in the Event Broker queue for immediate processing. This reduces computing requirements needed at collection and in certain cases even doubles throughput of a single collector compared to a Syslog Collector. Additionally, it decreases your network traffic going into Event Broker by maintaining a single data feed. 

In the second stage, the containerized CEB processors deployed in the Kubernetes-driven Event Broker cluster perform the normalization, categorization, enrichment, aggregation, and filtering tasks on the data sets and distribute them to the proper downstream destinations. This not only offloads these processor-heavy tasks to the highly available and scalable ADP Event Broker cluster environment, but it makes it easy within the ArcMC interface to stand up additional CEB instances with a click of a button. This simplicity is thanks in large part to ArcSight pre-configuration capabilities working in concert with the embedded Kubernetes and Kafka to automate the configuration and deployment of the CEB processors, which in turn enables high levels of scalability, workload distribution, and high availability. 

The following is an example scenario of how these new features might work in your environment. You set up a lightweight Syslog Collector to collect data from some Linux servers in your datacenter. In a single data feed, the Syslog Collector sends the data to ADP Event Broker. You deploy one or more CEB processors in your Event Broker infrastructure to normalize, categorize, aggregate, and filter the data into security events, and distribute them to the proper Event Broker topics for consumption by your analytic tools, data lakes, and other targeted applications in your environment. Over time as your organization grows, those Linux servers start generating more and more data, increasing the processor utilization of the CEB. Using the ArcMC interface, you easily and quickly deploy additional CEB processors to handle the increased data flow. 

A month later, multiple data spikes occur as your users prepare for a big enterprise-wide event. In a manner of minutes, your administrators can deploy additional CEB processors from the ArcSight Management Center GUI. The underlying Kubernetes resource management of the Event Broker cluster automatically identifies the nodes in the Event Broker cluster where there are compute resources available and deploys the CEB processors there. The automated configuration of the deployed CEB processors balances the workload among the multiple CEB processors for optimal performance and availability. When the spike subsides, the processors can be taken down with a single click, resulting in freed compute resources that can be used for other data flows as needed. 

Even with all the benefits that the Syslog Collector and CEB features provide, it’s important to understand that they’re not intended to replace SmartConnectors. Rather, they give you a massively scalable option to centralize the normalization and enrichment process your SmartConnectors typically perform. It's likely that most organizations that choose to take advantage of these massively scalable event processing capabilities will continue to use SmartConnectors as well. This allows deployment models to be designed that better fit unique organizational needs based on infrastructure, location, size, processing, and a variety of other factors. 

No matter your exact implementation, you’ll be able to enjoy the following significant benefits as you take advantage of the Syslog Collector and CEB processors features available in ADP 2.30: 

  • Centralized and massively scalable data normalization with better compute resources utilization
  • Reduced syslog related network traffic into Event Broker by up to half with a single raw data feed from Syslog Collectors, depending on the number of syslog event consumers in your environment
  • Up to twice as much syslog data collection throughput when using the lightweight Syslog Collectors compared to Syslog SmartConnectors
  • Vastly simplified administration at any scale, with automated single click addition or removal of CEB processors
  • Higher availability through automated parallel execution of CEB processors and ability to handle failures

 To learn more about how the new Syslog Collector and Connector in Event Broker (CEB) features in ADP can deliver easy to administer massively scalable event processing, talk to your ArcSight solution architects or sales representatives, or contact us directly.

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.