Highlighted
Absent Member.
Absent Member.
495 views

Differences in responsetimes among various LoadTest Tools

Hello everybody,

 

I am currently doing L&P testing using LoadRunner ( WebHTTP Protocol ). The test setup:

 

1 Virtual User / Thread

1 SOAP Call every Second

1200 SOAP Calls during the test.

Data is the same among all SOAP Requests

 

To test single request i usually use SoapUI as most of our developers do. During my tests i saw huge differences among various LoadTest Tools. Here's a small Table to illustrate the differences I saw:

 

Tool            | Min   | Avg   | Max   | 90%

------------------------------------------------------------

LoadRunner      | 0.080 | 0.121 | 0.262 | 0.144

SoapUI LoadTest | 0.015 | 0.025 | 0.140 | n/a

JMeter          | 0.056 | 0.064 | 0.175 | 0.067

 

I am wondering, why the response times from the SoapUI LoadTest are so much faster than the ones from JMeter and especially from LoadRunner. Are there differences how they are calculated? I have a problem to Justify my Measurements with LoadRunner when LoadRunner has most of the reported response times above 100 ms (without heavy load ) and the developer sees response times way below 100 ms for most of the SOAP requests.

 

Thank you for your help!

0 Likes
10 Replies
Highlighted
Outstanding Contributor.. Outstanding Contributor..
Outstanding Contributor..

Re: Differences in responsetimes among various LoadTest Tools

Hi MarkusG,

 

I am not sure about SoapUI LoadTest but as Jmeter is implemented in Java and LoadRunner using C++, that might cause big difference in transaction time.

 

Regards,

Chau

Regards,

Chau Nguyen | SW Technical Support Consultant.
LoadRunner and Performance Center


If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember to give them a KUDOS by clicking on the STAR at the bottom left of the post and show your appreciation.
Highlighted
Absent Member.
Absent Member.

Re: Differences in responsetimes among various LoadTest Tools

Hi Chau,
SoapUI is implemented in Java as well. And yeah LoadRunner is C(++) which in many cases is supposed to be faster. The results from LR are the slowest ones. I guess there are differences between those product, when they start the measurement and when they stop. Just wondering if anybody can confirm those differences.
Regards,
Markus

 

 

0 Likes
Highlighted
Outstanding Contributor.
Outstanding Contributor.

Re: Differences in responsetimes among various LoadTest Tools

Are you seeing the LR times in the Controller, or VUGen?

0 Likes
Highlighted
Absent Member.
Absent Member.

Re: Differences in responsetimes among various LoadTest Tools

In the Controller

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: Differences in responsetimes among various LoadTest Tools

Hello MarkusG,

 

Please note that as a general rule, HP Technical support does not provide information in regards of comparisons of LoadRunner and third party tools, nor comments in regards of other manufacturers products, since the arquitectures in which they are based are quite different.

 

My apologies for the inconvenience that this could cause.

Regards,

Daniela Gómez Alvarado
Customer Support Engineer

If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember to give them a KUDOS by clicking on the STAR at the bottom left of the post and show your appreciation.
0 Likes
Highlighted
Absent Member.
Absent Member.

Re: Differences in responsetimes among various LoadTest Tools


@dani-G wrote:

Hello MarkusG,

 

Please note that as a general rule, HP Technical support does not provide information in regards of comparisons of LoadRunner and third party tools, nor comments in regards of other manufacturers products, since the arquitectures in which they are based are quite different.

 

My apologies for the inconvenience that this could cause.


Hello dani-G,

 

I can understand that the HP Technical support does not comment on third party tools. But my intention with my question was to get an Idea if the response times are correct or not.  For example there might be ( and in my case is ) a requirement thagt 90 percentile of the transactions have to be <= 80 ms.  From the point of view of my LoadRunner Measurements the test failed way over SLA. I open a defect and the developer rejects the defect because his measurements (with SoapUI) show him something around 25 ms

 

My Measurement: 

 

lr_start_transaction("MyTransaction");

 

web_custom_request("MyTransaction",  .... )

 

lr_end_transaction("MyTransaction", LR_AUTO);

 

 

When I run the Test in the Controller Logging is disabled. But i also ran a few Tests in VuGen, especially to see how much is Wasted Time ( which is reported separately 😞

 

Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1482 Wasted Time: 0.0070).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1175 Wasted Time: 0.0055).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1264 Wasted Time: 0.0059).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1244 Wasted Time: 0.0072).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1225 Wasted Time: 0.0053).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1261 Wasted Time: 0.0059).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1272 Wasted Time: 0.0104).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1385 Wasted Time: 0.0061).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1232 Wasted Time: 0.0058).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1574 Wasted Time: 0.0058).
Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1831 Wasted Time: 0.0065). 

 

 

Okay even if i subtract Wasted Time, I am still out of SLA. Do I have a problem? Are we out of SLA or am I fooled by false numbers and Wasted Time is not correct or there's more time spent in lr_start_transaction and lr_end_transaction or ???

 

Thank you for more insights!

Best Regards

Markus

 

 

 

 

 

 

 

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Differences in responsetimes among various LoadTest Tools

Hi Markus, 

 

1. Are you performing the load tests with the different tools from one and the same machine?  Network conditions are affecting the response times. This can explain the response time the R&D got with SOUP UI. 

2. Are you seeing the same measurement when you send one request for example each 20 seconds? Is the response time increasing when the request frequency increased to one request each second? The response time from the server is expected to increase when the number and request frequency increases.

3. Are you running the same number of vusers with the same request frequency with the different tools? 

4. Are there other users accessing the server during the test which can also affect the server response time? 

5. Additionally it is a prerequisite that the load generators RAM is not exceeding 70-80% so that a reliable transaction response time data can be gathered during the load test.

 

The factors listed above can affect the transaction response time. 

 

Kind regards, 
Vesela 

 

 

HPE Support
If you haven’t tried it yet, come and join us in our entitled forums at Support Customer Forums

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution and give Kudos to the author for their assistance.
0 Likes
Highlighted
Respected Contributor.
Respected Contributor.

Re: Differences in responsetimes among various LoadTest Tools

Do you use Security on the SOAP request?  If so what is it SAML / WSA what version / implementation?


Create a Web/HTTP LoadRunner script with a Customer Request.  Build the request manually. 
You will need to add some SOAP headers to the Web script.
You may need to Remove or delete the Agent value or the SOAP server will reject the request due to it being generated by a Web Browser.
Capture the request via the Proxy

Use a Proxy and record the Transmission from SOAPUI.
You and use LoadRunner or a third party utility E.g TCPMon
Capture the request via the Proxy.


Save the request to a loadrunner buffer.  lr_save_string or lr_save_param_sprintf
Add the Body to the Custom Request via the buffer
Replay the script to check that it works once.
You should get a HTTP 200 response if all works. Don’t worry about the Processing of the SOAP message at this stage. 
Now run LoadRunner with say 10 Vusers. 
Also I assume that you are using SOAPUI Professional and that you use the same number of Threads in SOAPUI as you do with LoadRunner Virtual Users.
Compare the times and let me know the result.
Add the transaction markers in LoadRunner around the Custom Request only.

0 Likes
Highlighted
Honored Contributor.. Honored Contributor..
Honored Contributor..

Re: Differences in responsetimes among various LoadTest Tools

CptMercury,

 

What do you mean by "Capture the request via the Proxy" and “Use a Proxy and record the Transmission from SOAPUI."

Do you have a specific proxy in mind?

 

0 Likes
Highlighted
Respected Contributor.
Respected Contributor.

Re: Differences in responsetimes among various LoadTest Tools

Hello LoadRunner community

 

I checked my previous post and noticed that I did not explain what I am trying to do / analyse.

Often HTTP capture utilities display the captures in a format that is readable for a human. This may not be the format that was sent to the server.

 

What I am attempting as the first step is: Verify that the SOAP Transmission from SOAPUI identical to the SOAP Transmission from HP LoadRunner 12.0x

 

I asked about Security of the SOAP message because the comparison is not easy if SAML is introduced as the security layer of the SOAP message and if WS-Security implementation.

 

I assume Yes to security, however the previous description looks like NO. I therefore assume a concreate Web Service not a Virtual Web Service. I also assume that there is NO web_set_user statement in the loadrunner script.

 

Steps:

  1. Proxy Configuration

Use LoadRunner Remote Proxy (Important: Remote Proxy Configuration)

OR

Use a Third Party Proxy (eg TCPMon)

The Remote Proxy is used to record the SOAPUI and LoadRunner transmission to the server.

NOTE: record / capture each product in its own session do NOT record both in one session.

You can use LoadRunner Remote Proxy to record non-browser based applications. E.g SOAPUI

 

  1. Compare the Capture

Use a comparison tool to compare the Proxy Log files (Capture files) Eg. WinDif.

Pay attention to

  1. Headers used by SOAPUI
  2. SOAP message format. It should be a complete buffer with no CR/LF. It should look like a complete string.

Please confirm that both the Transmissions are Identical in size and format.

Once the packets are confirmed to be the same. I like to check how the buffer is loaded into the web_custom_request of HP LoadRunner 12.x.

 

NEW in HP LoadRunner 12.x

LoadRunner 12.0x introduces new features; Support capture files. I can now use Third Party proxy for applications that HP LoadRunner will not record.)

 

I notice that it is about 100ms difference.

 

As a long time user of HP LoadRunner, I am very intrested in this Topic / Observation

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.