Monitoring and Debugging the Vertica Data Sync Process in ZENworks

2 Likes
over 1 year ago

To improve the performance of the Dashboard feature in ZENworks, especially for larger customers, the ZENworks 2020 release supports an additional database called the Vertica database. When Vertica is configured in your zone, the existing RDBMS database will continue to be the primary database and data that is required to populate meaningful dashlets is obtained from the Vertica database. The data migration process from the RDBMS to Vertica can be divided into two parts; firstly, the initial bulk data migration process, which is followed by the Change Data Capture process. The Change Data Capture (CDC) process continuously syncs new or updated data from the existing RDBMS to Vertica.

This cool solution explains how to monitor the Change Data Capture (CDC) process to ensure that the syncing of data from RDBMS to Vertica functions effectively and the corrective actions to be taken if any one of the components that are part of the CDC process, fails.

1. Change Data Capture Process Workflow

The following diagram explains the CDC process from the RDBMS to Vertica:

clipboard_image_0.png

  1. The Kafka connectors (Kafka producer) identifies changes in the respective RDBMS database tables.

  2. These changes are published to a topic within a Kafka broker.

  3. The ZENworks loader stream processor receives the message and writes it to the Vertica database and the Kafka broker receives an acknowledgment of the processed message.

Kafka also uses a Schema Registry, which is a separate entity that producers and consumers talk to,

for sending and retrieving schemas that describe the data models for the messages.

2. Monitoring the Components of the Change Data Capture Process

To ensure that the Change Data Capture(CDC) process is working as expected, all of the components that are part of the CDC process, need to be monitored. The following components are used in the CDC process:
• ZooKeeper
• Kafka services
• Kafka
• Kafka-connect
• Schema-registry
• Vertica database
To know more about these components, see the Online Documentation site.
To view the status of the components, you need to navigate to the Diagnostics page in ZENworks Control Center (ZCC). After logging into ZCC, click the Diagnostics section on the left pane and view the status in the following three sections:
• Vertica Cluster
• Zookeeper Cluster
• Kafka Cluster

A snapshot of Diagnostics page is as follows:

clipboard_image_1.png

A. Vertica Cluster

Starting or Stopping the Vertica database or a few Vertica nodes that are part of the Vertica database

To start or stop the Vertica nodes or the Vertica database, it is recommended that you use the Admin Tools command.

  1. Obtain the Vertica database’s super user password by running the zman command zman srvgc.

  2. Login as the Vertica admin user by using the command su dbadmin.

 

Note: In this case dbadmin is the Vertica super user.

  1. Run the Admintools command by using the command admintools.

 

A dialog box is displayed where the options to start or stop the database and to start or stop the nodes are displayed. Execute any one of the options.

The ideal status of the Vertica cluster should be displayed as up and the status of all the Vertica servers should be displayed as Running. The other statuses displayed for Vertica cluster are warning and down.

  • If the status is up, then it indicates that all the Vertica nodes (servers) in the cluster are up and running.

  • If the status is displayed as warning, then it indicates that the Vertica cluster is functioning but certain nodes are down and the administrator needs to start these nodes.

If the status is displayed as down, then it indicates that the Vertica cluster is not functioning and the Vertica database needs to be restarted. However, before restarting the database, it is recommended to check the Vertica catalog to identify the root case of the database failure.

clipboard_image_3.png

B. ZooKeeper Cluster

ZooKeeper is a webapp within the ZENworks server and should be up and running irrespective of whether Vertica or Kafka is configured in the zone or not. The various statuses of the ZooKeeper cluster displayed in the Diagnostics page are up, warning and down.

  • If the status is up, then it indicates that all the servers in the ZooKeeper cluster are functioning as expected.

  • If the status is displayed as down, then it indicates that the ZENServer services of the server in which ZooKeeper is configured, is down.

  • If the status is displayed as warning, then it indicates that a few nodes in which ZooKeeper is configured, are down. The administrator needs to check the service-messages.log of these servers, fix the issue reported in the log, and then restart the ZENServer service in order to restart ZooKeeper. However, if ZooKeeper still does not come up, then the administrator needs to enable debug logging for ZooKeeper to identify the root cause of the failure.

Enabling ZooKeeper debug logging

On a Linux Server:

The administrator needs to uncomment the configurations section ‘zookeeper Linux logging’ within the file /opt/novell/zenworks/share/tomcat/conf/log4j.properties, and restart the ZENServer service. The log file will be generated at /var/opt/novell/log/zenworks/zookeeper-messages.log.

 

On a Windows Server:

The administrator needs to uncomment the configurations section ‘zookeeper Windows logging’ within the file C:\Program Files (x86)\Novell\ZENworks\share\tomcat\conf\log4j.properties, and restart the ZENServer service. The log file will be generated at C:\\Program Files (x86)\\Novell\\ZENworks\\logs\\zookeeper-messages.log.

C. Kafka Cluster

The Kafka cluster consists of three services that are:

  • Kafka

  • Kafka-connect

  • Schema-registry

The various Kafka cluster statuses that are displayed in the diagnostics page are up, warning or down.

If the Kafka Cluster status is displayed as warning or down, then the administrator should check the logs of the services that are down, before restarting the service. The logs are avilable at the following locations:

  • Kafka log file located at /var/opt/novell/log/zenworks/kafka-logs/server.log

  • Kafka-connect log file located at /var/opt/novell/log/zenworks/kafka-logs/kafka-connect.log

  • Schema-registry log file located at /var/opt/novell/log/zenworks/kafka-logs/schema-registry.log

Starting/Stopping Kafka Services

The commands to be executed to start or stop Kafka and its services are:

  • Kafka: systemctl start/stop zenworks-kafka.service

  • Kafka-connect: systemctl start/stop zenworks-connect.service

  • Schema-registry: systemctl start/stop zenworks-schema-registry.service

Data Sync Status

When all the essential components are functioning as expected, then check the status of the CDC process to verify whether the RDBMS to Vertica data sync pipeline is functioning or not. This can be viewed in the Data Sync Status section of the Diagnostics page.

clipboard_image_4.png

The Data Sync status section will display the status of the CDC process for several tables as shown in the following screenshot:

clipboard_image_5.png

The various columns displayed in this section represents the different components or the state of the CDC process. The columns are as follows:

  • Connector Name: Connector is the service that reads data from the RDBMS and publishes it to Kafka. Here Connector Name indicates the name of the connector for each database table.

  • Connector Status: Represents the status of each connector. The status will be displayed as either Running or Failed. In case of a failure, check the kafka connector logs available at /opt/novell/log/zenworks/kafka-logs/kafka-connect.log, fix the issue reported in the log and then reconfigure the connector. For more information, see Reconfiguring failed connectors section(last section).
  • Consumer Name: Consumer is a thread running within the ZENworks loader services that reads data from Kafka and publishes it to Vertica. Here Consumer Name indicates the name of the consumer for each table.

  • Loader Name: As explained previously, a consumer runs within the ZENworks loader services. Therefore, the Loader Name indicates the name of the ZENworks loader in which a particular consumer is running.

  • Last sync from Kafka: Indicates the most recent time when data fetched/polled from kafka.

  • Last Sync to Vertica: Indicates the most recent time when data was written to Vertica by the stream processor.

  • Consumer status: Similar to Connector Status, Consumer Status indicates the status of the ZENworks loader thread that is responsible for writing data to Vertica. This status will either be displayed as either Running or Failed. In case of a failure check the loader-messages.log available at var/opt/novell/log/zenworks/loader-messages.log to identify the root cause of the failure. After fixing the issue reported in loader-messages.log, restart the ZENworks loader services.

Reconfiguring failed connectors

To reconfigure a connector, run the following command:

zman server-role-kafka-reconfigure-connectors -c <name of the connector>

You can also retrieve the names of all the connectors in the command line utility by running the command zman server-role-kafka-list-connectors .

After reconfiguring the Kafka connector, restart the Kafka-connect service by running the command systemctl restart zenworks-connect.service .

 

Labels:

How To-Best Practice
Configuration Management
Comment List
Anonymous
Related Discussions
Recommended