Introducing LoadRunner Data Hub

by Micro Focus Employee in Application Delivery Management

Authors: Tedy Yu, Shira Finson

LoadRunner Professional (LRP) 2022 brings a long list of new features and enhanced functionalities, including the LoadRunner Data Hub. The Data Hub allows you to run load tests via a message broker without worrying too much about the network condition between Controller and the load generators, asking IT to open certain TCP ports, learning how to use MI Listener when crossing a firewall, etc. In other words, it greatly simplifies the architecture of the network. In this blog, we’ll guide you through it.

Prior Solutions

Figure 1. Controller and LG are on the same network

Figure 2. Controller and LG are separated by firewall

In the past, LoadRunner Controller had to connect either directly with load generators or indirectly through the MI Listener to cross a firewall. If one of those components failed or its network dropped intermittently, the whole test run was interrupted.

What is the Data Hub?

Figure 3. Data Hub’s role in LoadRunner

The LoadRunner Data Hub is built on top of Apache Kafka, a well-known open-source distributed data streaming platform. It can be a single Kafka broker instance or a cluster of Kafka broker instances with a predefined configuration, allowing communication between LoadRunner products only.

If you enable the Data Hub, both Controller and the load generators will send and receive messages to/from the broker without worrying about the status of the other side. In Kafka’s terminology, a sender is called a producer, and a receiver is called a consumer. Both Controller and the load generators are Kafka producers and consumers at the same time, to allow two-way communication. They’re also called Kafka clients, from the perspective of the Kafka broker acting as the server.

For enhanced security, the Data Hub enforces mutual-SSL validation and thus prevents arbitrary access without the correct certificates. For the same reason, you can’t use your own Kafka broker instance or cluster to replace the Data Hub unless the certificates are properly configured. 

Install the Data Hub

We recommend installing the LoadRunner Data Hub on an Ubuntu computer using the installer, which is available at Micro Focus Software Licenses and Downloads. You can find more detailed instructions in the LoadRunner Professional Help

The default installation folder is /opt/MF/MF_LoadRunner_Data_Hub. After the installation is complete, a Kafka service starts, and is listening on a TCP port (9443 by default) for clients to connect using LoadRunner default SSL certificates.

listeners=SSL://:9443

ssl.truststore.location=/opt/MF/MF_LoadRunner_Data_Hub/app/config/cert/kafka.server.truststore.jks

ssl.truststore.password=***

ssl.truststore.type=PKCS12

ssl.keystore.location=/opt/MF/MF_LoadRunner_Data_Hub/app/config/cert/kafka.server.keystore.jks

ssl.keystore.password=***

ssl.key.password=***

ssl.keystore.type=PKCS12

Based on the configuration above, only SSL connections from authenticated LoadRunner Controller and load generators are allowed. All data in transit are encrypted. Only LoadRunner components with SSL certificates from <LR_Installation_Folder>/dat/certs which are signed by the same CA as defined in the Data Hub’s ssl.truststore.location field will be authenticated. By default, LoadRunner uses the same self-signed CA certificate as the Data Hub, so it can be used out of box until you change the certificates in your environment.

If your LoadRunner environment uses custom SSL certificates after installation, whether they’re self-signed or signed by a valid CA, you need to prepare the SSL certificate for the Data Hub and Web Connector (see below) as well as the CA certificate chain before the installation. Make sure you use the same CA in LoadRunner Controller, the load generators, and the Data Hub. 

If you want to use another port, (e.g., 443, which is usually allowed in the firewall rules by default), you need to manually stop the Kafka service, modify the server.properties config, and start the service again:

sudo systemctl stop kafka

nano /opt/MF/MF_LoadRunner_Data_Hub/app/config/server.properties

Find the following line in the editor, replace 9443 with any port number you want, save the file (Ctrl+O), and start the service.

listeners=SSL://:9443

sudo systemctl start kafka 

Run tests through the Data Hub

Once the Data Hub is installed, you need to configure the settings for the load generators and Controller. Then you can run tests through the Data Hub.


Let’s go through the process step by step:

  1. Set up the load generator (LG) to use the Data Hub

On Windows LGs, open the LoadRunner Agent Configuration tool from the Start menu.

Choose Through LoadRunner Data Hub and click the Settings button beside it.

In the Data Hub server address field, specify <DataHub_FQDN>:<SSL_port>, e.g.: my-datahub.example.net:9443. Keep the load generator name unchanged (it’s filled in automatically with the current machine’s FQDN), and then click OK.

If the load generator is not directly reachable in the Controller’s network, you can also change its name to a user-friendly load generator name or alias, such as loadgen101-datahub. In that case, expect a slower script transfer step in the current release.

Figure 4. Configure the load generator to use the Data Hub 

Click Test Connection to verify that communication can be established with Data Hub.

Figure 5. Test Connection to Data Hub from Load Generator 

Finally, click OK to apply the changes. The LoadRunner Agent will be restarted. 

On Linux LGs, use lr_agent_settings in the bin folder to enable the Data Hub options.

Data Hub options:

[-is_datahub <enable Data Hub settings: 0 | 1>]

[-datahub_addr <Data Hub server address>]

[-lg_name <load generator name>]

[-is_use_web_connector <use Web Connector: 0 | 1>]

[-web_connector_addr <Web Connector address>]

[-web_connector_proxyname <proxy name>]

[-web_connector_proxyport <proxy port>]

[-web_connector_proxyuser <proxy user name>]

[-web_connector_proxypass <proxy password>]

Example:

cd /opt/MF/MF_LoadGenerator/bin

lr_agent_settings -is_datahub 1 -datahub_addr my-datahub.example.net:9443 -lg_name `hostname -f`

Note: Replace my-datahub.example.net with the FDQN of your Data Hub machine. hostname -f will be evaluated to the FQDN of the current LG machine.


 2. Set up the Data Hub Settings in Controller

In Controller, go to Tools > Data Hub Settings to open the Settings dialog.

In the Server address field, enter the Data Hub machine and port number, e.g., my-datahub.example.net:9443, and click Save. You can use the Test Connection button to verify the configuration.

Figure 6. Configure Data Hub Settings in Controller 


3. Create a network profile for the Data Hub

In Controller, open Tools > Network Profile Manager to open the Network Profile Manager dialog box.

Click New to add a new entry. Give the new profile a name, such as “DataHub Profile,” and then enable the Data Hub Profile option. It's also recommended to enable Transfer Files Over TCP for fast file transfer over TCP where possible. This is recommended due to a limitation of transferring scripts over Data Hub in the current release, which from our observation will take a minute or more to copy even a simple script from Controller to a load generator in the same network. However, if Controller cannot reach the load generator directly, this option is irrelevant, and in that case you need to use the Data Hub Web Connector (see Step 7 below).

Figure 7. Create a network profile for Data Hub 

There are two steps to ensure that the Transfer Files Over TCP option will work:

  • Enable Transfer all files to remote hosts before running scenario:

In Controller, select Tools > Options. In the Runtime File Storage tab, enable Transfer all files to remote hosts before running scenario.

Figure 8. Enable option to transfer files more efficiently while using Data Hub 

  • Ensure the Load Generator name field in the Agent Configuration dialog (in figure 4) is the FQDN of the load generator machine, which is specified by default for you and should not be changed. In Controller, specify the same value in the Name column of the Load Generators dialog box (figure 9).

If the two conditions above are not met, file transfer is through the Data Hub, which is slow compared to transfer directly over TCP. 


  1. Add load generators working with the Data Hub

Click the Load Generator icon in the Controller toolbar to open the Load Generators dialog.

Figure 9. Add load generators to work with Data Hub 

In the Name column, type the value of the load generator name you specified previously in the Agent Configuration dialog on the load generator – in this example, winloadgen101.example.net.

Select the network profile that you just created for the Data Hub – in this example, “DataHub Profile.”

Click the Connect button, and you’ll see its status change to Ready with an SSL icon. This means all data sent between Controller and the load generator through the Data Hub is secure by default.


  1. Run the test

Click Start Scenario in the Run tab.

Figure 10. Run a test through Data Hub 

If you enabled Transfer Files Over TCP in the Data Hub network profile, the script and results are transferred over TCP for efficiency if Controller and the load generator are in the same network. Otherwise, the transfer is over Data Hub, which may be slower.


  1. (Optional) Send real-time transaction data to the Data Hub

Figure 11. Collate transaction raw data into a Data Hub topic

This feature provides real-time transaction raw data during the test run, for other applications like external analysis tools to consume.

To send the real-time data to the Data Hub, go to Results > Collate Results > Collate Settings... to open the Collate Settings dialog box. Select At runtime to LoadRunner Data Hub in the dialog, and specify a meaningful name in the Topic name field, such as “Transactions.” Click Save to apply. 


Over Firewall Deployment

  1. Use the Data Hub Web Connector

The Data Hub allows only TCP connections from client applications. When you cannot connect directly to the Data Hub due to a firewall or a required proxy, you can use the Web Connector to expose an HTTPs endpoint.

Install the Data Hub Web Connector on an Ubuntu machine (which can be the same machine as the Data Hub, or a different machine) using the installer:

cd /NAS_NTPP/products/LT/LT-LR/linux32_release/Default/SetupBuilder/Output/UNIX_LR_DataHub_WebConnector/InstallProject

InstallProject/linux_amd64/LR_DataHub_WebConnector_x64_Build_Output/Default_Configuration/Web_Installers/InstData/Linux/VM/

./inst64.bin

Accept all default choices by pressing the Enter key until the install is complete.

This installs the Web Connector in the location /opt/MF/MF_LoadRunner_Data_Hub_Web_Connector.

After installation, a service named “webconnector” starts, and is listening on TCP port 5050 by default.

If you want to use another port, do the following:

systemctl stop webconnector

cd /opt/MF/MF_LoadRunner_Data_Hub_Web_Connector

nano testexec.server/cloud-tunnel/etc/cloud-tunnel.json

Find the following line and change the port number:

"cloudTunnelPort": 5050,

Save the file (Ctrl+O), and run the following command to start the service:

systemctl status webconnector

 

Next, there are some extra configuration steps to set up the Web Connector for the load generators and/or Controller.

If a load generator cannot connect to the Data Hub server address directly, do the following:

On a Windows load generator, open the Agent Configuration dialog and select the Data Hub settings. Check the Connect via Data Hub Web Connector option and specify the Data Hub Web Connector address, e.g., my-datahub.example.net:5050. If a proxy is necessary, fill in the proxy details as well.

Figure 12. Configure the load generator to use the Data Hub Web Connector 

Click OK to apply the changes. 

If Controller cannot connect to the Data Hub server address directly, do the following:

In Controller, open the Data Hub Settings dialog, and specify the same configuration as the load generator example above. For example, in the Web Connector field, enter my-datahub.example.net:5050. Click Save.

Figure 13. Configure Controller to use the Data Hub Web Connector

Run the test as usual. 


Troubleshooting tips

Use the Test Connection button in the GUI to verify the connectivity to the Data Hub.

Controller will show error messages in the output pane if communication with the Data Hub fails.

On the load generator side, you can check the LoadRunner Agent log.

The Kafka logs on the Data Hub machine can be helpful when troubleshooting problems with the client connection or the Kafka broker. The log path is /opt/MF/MF_LoadRunner_Data_Hub/app/kafka.log. The Kafka CLI tool located in /opt/MF/MF_LoadRunner_Data_Hub/app/bin can help you troubleshoot problems at the Kafka level. 

Conclusion

The Data Hub can be deployed to simplify the load test environment from LRP version 2022.

It is a major step forward towards a more modern, scalable, and maintainable architecture in LoadRunner, as it decouples the communication between Controller and the load generators, removes the need to install MI Listener in most cases, prevents interruptions caused by an unstable network, streams real-time test data for other integration purposes, and more.

This feature is available as a tech preview in LRP 2022 and will be supported in future LRE versions as well.

We’re continuing to develop the Data Hub to better support new scenarios in the future, such as a lightweight Controller service runtime to run a much bigger number of Vusers with less hardware investment. Please try it out now and share your feedback and suggestions with us!

Labels:

Performance Testing
Anonymous