In the era of big data technologies, organizations are looking for ways to adopt a new data collection, storage and processing model, in order to solve big data management challenges and increase operational efficiency.

Imagine the situation where you have multiple data sources each sending the data in its own format to multiple collection stores. Performing data processing, data analysis and various services on top of all this data would become an extremely challenging task. Therefore, there’s a high demand to simplify data collection and data storage from various endpoints and enable common data analysis and data visualization, as well as value-adding services on top of the collected data.

Therefore, industry experts have made recommendations about the need to define a central data lake, in order to simplify and empower capabilities such as advanced analytics.

Collect Once Store Once

To address this requirement, we at Micro Focus are continuously developing innovative solutions to help our customers achieve greater operational efficiency in their data management. Our IT Operations Management (ITOM) Collect Once Store Once (COSO) strategy is to bring together efficient only-once data collection and storage, instead of collecting and storing the data multiple times and in multiple places for different needs, and to provide various services on top of this multi-source data. (These services can be used across multiple ITOM solutions, or be restricted to a particular product or a solution.)

As part of our COSO strategy, we developed a unified big data platform consolidating multiple domains – the COSO Data Lake (previously known as the Intelligent Data Lake (IDL)). Based on Micro Focus Vertica and ITOM Container Deployment Foundation (CDF), it provides data storage and is able to receive and process high-volume and high-velocity data from a variety of independent data sources. The COSO Data Lake therefore acts as a common data ingestion platform, offering you the following key benefits:

• Optimized data ingestion providing a common platform to process and store data, with built-in machine learning and analytics
• Ability to handle terabytes of data at a high ingestion rate
• Reduced complexity: easier integrations, pre-integrated use cases, the ability to scale out horizontally as needed
• Easy installation and configuration, simple management
• Built-in analytical functions and SQL access to machine learning (Vertica capabilities)
• Cost-effectiveness: Lower maintenance costs

The COSO Data Lake is utilized by a set of our ITOM solutions. Containerized solutions are available as microservice components that include analytics, dashboarding and reporting and can be easily deployed anywhere in your environment. The shift to a microservices architecture enables us to integrate and manage our assets more easily, faster adapt to market changes, offer better scalability and reduce the overall complexity of your IT environment.

Use of the COSO Data Lake in Containerized Operations Bridge

Operations Bridge collects topology, events and metrics, and stores this data in the COSO Data Lake, from physical, virtual, on-premise and cloud infrastructure deployments.

Figure 1 – Operations Bridge: Automated AIOps for the Digital Enterprise

The COSO Data Lake was initially introduced with Operations Bridge 2018.02. With every quarterly release, we continually enhance it by supporting more data types that can be streamed into the Data Lake, bringing additional services on top of the collected data and introducing performance improvements.

Thus, initially, Operations Bridge was able to collect system metrics from Operations Agents streamed into the COSO Data Lake. With Operations Bridge 2018.05, metric streaming from SiteScope and BPM probes was added, followed by the data from Management Packs, Real User Monitor (RUM) and Diagnostics (Operations Bridge 2018.11). In addition, we support Virtualization Collector that collects the VMware topology, metrics and events and gives you visibility into the entire VMware Virtualization infrastructure. The virtualization data is streamed to the COSO Data Lake and can be visualized in Business Value Dashboards (BVD) and Performance Dashboards (PD). Last but not least, you can Bring Your Own Business Intelligence (BYOBI) tool to visualize the collected data.

You can build dashboards/reports on the collected data and do centralized thresholding on top of this data. In addition, automatic event analytics runs in the background providing automatic event grouping (using machine learning algorithms) thus improving operator efficiency since fewer events need to be processed.

Figure 2 – A Graphical Overview of the COSO Data Lake

As you can see in the diagram, we provide a single framework for managing data collection and storing the data into the COSO Data Lake (powered by Vertica). The COSO Data Lake possesses scalable interfaces to ingest and process data. Data Access APIs allow you to access the collected data out of the COSO Data Lake, in order to use it for other purposes than the ones presented in the diagram.

Note: Currently, we provide out-of-the-box metric streaming policies to the COSO Data Lake for the following Operations Bridge Manager (OBM) Management Packs:

• OBM Management Pack for Microsoft SQL Server
• OBM Management Pack for Infrastructure
• OBM Management Pack IBM WebSphere Application Server

Services on Top of the COSO Data Lake

As already outlined in the overview diagram, the following services provided on top of the COSO Data Lake are used by Operations Bridge:

Central Thresholding: Allows you to configure static thresholds for all data coming into the COSO Data Lake. This feature is especially useful in situations where thresholding is not available at the data source or is not under control of the central monitoring team (for example, Virtualization Data Collector and Streaming Agent). The thresholds are evaluated on the received data samples. A threshold violation generates a corresponding event in OBM.

For more information, see Performance Management on the Micro Focus Documentation Portal.

Event Analytics based on machine-learning algorithms: The first analytic capability on top of the COSO Data Lake, Event Analytics automatically finds patterns in the event stream and uses the information to group events together, which with a high probability belong to the same problem. This can consequently reduce the total number of events displayed in OBM and simplify event processing for the operators.

Figure 3 – Events Automatically Grouped by Event Analytics

For more information, see Event Analytics on the Micro Focus Documentation Portal.

Dashboarding and Reporting: Business Value Dashboards and OBM Performance Dashboards can be used to visualize the data stored in the COSO Data Lake. The data is accessed by the Performance Dashboard to graph metrics and the Business Value Dashboard to show reports on the collected data.

Figure 4 – Performance Dashboard on the COSO Data Lake Data

Note: Business Value Dashboard is a cross-domain service that can be consumed by several ITOM solutions. Performance Dashboards are accessible from Operations Bridge.

For more information, see Dashboards and virtualization on the Micro Focus Documentation Portal.

BYOBI: We also open up access to the data available in the COSO Data Lake based on well-defined APIs. This, for instance, allows you to make use of third-party Business Intelligence (BI) tools present in your environment (also known as BYOBI – Bring you own BI tool) to create reports and dashboards on the data available in the Data Lake and thus visualize the data you collect from ITOM products using the BI tool of your choice (e.g., Microsoft PowerBI, SAP Business Intelligence, Tableau, IBM Cognos, etc.).

See a separate blog BYOBI to visualize the data in COSO Data Lake for more information. In addition, check out the Guide to using an external BI tool on Operations Bridge data (this white paper will introduce you to the concepts of the BYOBI tool and provide instructions on how to build reports on top of the data available in the COSO Data Lake).

Operations Bridge End User Management (EUM): End User Management focuses on application performance and availability. An EUM Operator User Interface (UI) is available inside OBM. This new UI provides an overview of the application status and the status over time (based on the BPM data in the COSO Data Lake).

Detailed steps on how to install and configure the COSO Data Lake, how to configure metric streaming and how to set up access to the streamed data from PDs and BVDs are described on the Micro Focus Documentation Portal.

A Few Words about CDF and Vertica

Now that we have introduced our COSO strategy and the COSO Data Lake, let’s briefly talk about the processes and technologies behind them. The COSO Data Lake is based on the Container Deployment Foundation (CDF).

The CDF is a new delivery and deployment model that provides a container management and orchestration framework leveraging the Docker container architecture and using Kubernetes for container orchestration. A set of Micro Focus ITOM solutions are successfully running on the CDF enabling easier integrations of capabilities within the same solution. (In fact, capabilities come as pre-integrated containers, with a lot of pre-integration delivered for standard use cases.) Based on the open-source software – Docker containers and Kubernetes clusters – the CDF brings quick software installation and a consistent upgrade model with rolling updates. All this is done on top of an automated and modern orchestration layer, built natively to handle container clusters at scale.

The COSO Data Lake is powered by Vertica, a software-based analytics platform that provides high-performance advanced analytics and in-database machine learning, supporting all major cloud platforms and all popular data formats. Please, see the Vertica documentation to learn more.

Don’t wait! Start your COSO journey today! The container technology it is based on is an industry trend that is rapidly gaining maturity and offers tremendous benefits, so take the opportunity to learn about it and take advantage of container architecture features, such as microservices. With our rich portfolio, we are able to combine containerized and non-containerized capabilities and deliver you the innovative solution to support your business needs. If you are already using OBM, you do not have to replace it immediately – if need be, classic versions of OBM can be integrated with the COSO Data Lake on a container deployment. If you are interested and need more information, please contact your Micro Focus representative for assistance.

We encourage you to try out our new features and enhancements! For further information on our offerings, visit the Operations Bridge product page, explore our documentation resources and check out our videos and blogs.

Go with a leader!

Did you know that we have been awarded leadership status by EMA in their market research of Advanced IT Analytics?
“Its capabilities to learn and predict anomalies across multiple data sources is compelling. Its focus on automation, or what it calls “automated AIOps,” is both distinctive and highly relevant to growing trends and needs within the industry.” 
Dennis Drogseth, VP of Research, EMA.

Download their report here and see their white paper that includes customer-measured RoI!

If you have feedback or suggestions, don’t hesitate to comment on this article.

Explore full capabilities of Operations Bridge by taking a look at these pages on our practitioner portal: Operations Bridge Manager, Operations Bridge Analytics, Operations Connector (OpsCx), Business Value Dashboard (BVD) and Operations Orchestration (OO) documentation!

Events

To get more information on this release and how customers are using Operations Bridge, we are happy to announce the following events:

• Webinar replay –How Garanti Bank Delivers an Excellent End User Experience with Operations Bridge 
• Webinar replay – Hybrid IT is the new normal
• Webinar replay – Solving the complexity of Hybrid Cloud monitoring
• Webinar replay – 6 Ways to Simplify the Management of Your Multi-Cloud Environment
• See all the Micro Focus events Worldwide

Read all our news at the Operations Bridge blog.

Related Items

Explore all Operations Bridge capabilities and technology integrations by visiting these resources:

• Operations Bridge
• Operations Bridge Manager (formerly OMi)
• Operations Bridge Technology Integrations eBook
• Learn how Boeing shifted to focus on business value of IT using Operations Bridge
• Listen to what value Operations Bridge provides to Accenture
• Learn how Vodafone Germany has obtained key value from Operations Bridge
• See how Banco Sabadell manages public cloud resources from Operations Bridge
• Learn how key visibility allowed the IT team at Swiss Mobiliar to become a strategic business partner