Differences in responsetimes among various LoadTest Tools

Hello everybody,

 

I am currently doing L&P testing using LoadRunner ( WebHTTP Protocol ). The test setup:

 

1 Virtual User / Thread

1 SOAP Call every Second

1200 SOAP Calls during the test.

Data is the same among all SOAP Requests

 

To test single request i usually use SoapUI as most of our developers do. During my tests i saw huge differences among various LoadTest Tools. Here's a small Table to illustrate the differences I saw:

 

Tool            | Min   | Avg   | Max   | 90%

------------------------------------------------------------

LoadRunner      | 0.080 | 0.121 | 0.262 | 0.144

SoapUI LoadTest | 0.015 | 0.025 | 0.140 | n/a

JMeter          | 0.056 | 0.064 | 0.175 | 0.067

 

I am wondering, why the response times from the SoapUI LoadTest are so much faster than the ones from JMeter and especially from LoadRunner. Are there differences how they are calculated? I have a problem to Justify my Measurements with LoadRunner when LoadRunner has most of the reported response times above 100 ms (without heavy load ) and the developer sees response times way below 100 ms for most of the SOAP requests.

 

Thank you for your help!

  • Hi MarkusG,

     

    I am not sure about SoapUI LoadTest but as Jmeter is implemented in Java and LoadRunner using C , that might cause big difference in transaction time.

     

    Regards,

    Chau

  • Hi Chau,
    SoapUI is implemented in Java as well. And yeah LoadRunner is C( ) which in many cases is supposed to be faster. The results from LR are the slowest ones. I guess there are differences between those product, when they start the measurement and when they stop. Just wondering if anybody can confirm those differences.
    Regards,
    Markus

     

     

  • Are you seeing the LR times in the Controller, or VUGen?

  • Hello MarkusG,

     

    Please note that as a general rule, HP Technical support does not provide information in regards of comparisons of LoadRunner and third party tools, nor comments in regards of other manufacturers products, since the arquitectures in which they are based are quite different.

     

    My apologies for the inconvenience that this could cause.


  • -G wrote:

    Hello MarkusG,

     

    Please note that as a general rule, HP Technical support does not provide information in regards of comparisons of LoadRunner and third party tools, nor comments in regards of other manufacturers products, since the arquitectures in which they are based are quite different.

     

    My apologies for the inconvenience that this could cause.


    Hello dani-G,

     

    I can understand that the HP Technical support does not comment on third party tools. But my intention with my question was to get an Idea if the response times are correct or not.  For example there might be ( and in my case is ) a requirement thagt 90 percentile of the transactions have to be <= 80 ms.  From the point of view of my LoadRunner Measurements the test failed way over SLA. I open a defect and the developer rejects the defect because his measurements (with SoapUI) show him something around 25 ms

     

    My Measurement: 

     

    lr_start_transaction("MyTransaction");

     

    web_custom_request("MyTransaction",  .... )

     

    lr_end_transaction("MyTransaction", LR_AUTO);

     

     

    When I run the Test in the Controller Logging is disabled. But i also ran a few Tests in VuGen, especially to see how much is Wasted Time ( which is reported separately ):

     

    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1482 Wasted Time: 0.0070).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1175 Wasted Time: 0.0055).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1264 Wasted Time: 0.0059).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1244 Wasted Time: 0.0072).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1225 Wasted Time: 0.0053).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1261 Wasted Time: 0.0059).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1272 Wasted Time: 0.0104).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1385 Wasted Time: 0.0061).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1232 Wasted Time: 0.0058).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1574 Wasted Time: 0.0058).
    Action.c(33): Notify: Transaction "MyTransaction" ended with a "Pass" status (Duration: 0.1831 Wasted Time: 0.0065). 

     

     

    Okay even if i subtract Wasted Time, I am still out of SLA. Do I have a problem? Are we out of SLA or am I fooled by false numbers and Wasted Time is not correct or there's more time spent in lr_start_transaction and lr_end_transaction or ???

     

    Thank you for more insights!

    Best Regards

    Markus

     

     

     

     

     

     

     

  • Hi Markus, 

     

    1. Are you performing the load tests with the different tools from one and the same machine?  Network conditions are affecting the response times. This can explain the response time the R&D got with SOUP UI. 

    2. Are you seeing the same measurement when you send one request for example each 20 seconds? Is the response time increasing when the request frequency increased to one request each second? The response time from the server is expected to increase when the number and request frequency increases.

    3. Are you running the same number of vusers with the same request frequency with the different tools? 

    4. Are there other users accessing the server during the test which can also affect the server response time? 

    5. Additionally it is a prerequisite that the load generators RAM is not exceeding 70-80% so that a reliable transaction response time data can be gathered during the load test.

     

    The factors listed above can affect the transaction response time. 

     

    Kind regards, 
    Vesela 

     

     

  • Do you use Security on the SOAP request?  If so what is it SAML / WSA what version / implementation?


    Create a Web/HTTP LoadRunner script with a Customer Request.  Build the request manually. 
    You will need to add some SOAP headers to the Web script.
    You may need to Remove or delete the Agent value or the SOAP server will reject the request due to it being generated by a Web Browser.
    Capture the request via the Proxy

    Use a Proxy and record the Transmission from SOAPUI.
    You and use LoadRunner or a third party utility E.g TCPMon
    Capture the request via the Proxy.


    Save the request to a loadrunner buffer.  lr_save_string or lr_save_param_sprintf
    Add the Body to the Custom Request via the buffer
    Replay the script to check that it works once.
    You should get a HTTP 200 response if all works. Don’t worry about the Processing of the SOAP message at this stage. 
    Now run LoadRunner with say 10 Vusers. 
    Also I assume that you are using SOAPUI Professional and that you use the same number of Threads in SOAPUI as you do with LoadRunner Virtual Users.
    Compare the times and let me know the result.
    Add the transaction markers in LoadRunner around the Custom Request only.

  • CptMercury,

     

    What do you mean by "Capture the request via the Proxy" and “Use a Proxy and record the Transmission from SOAPUI."

    Do you have a specific proxy in mind?

     

  • Hello LoadRunner community

     

    I checked my previous post and noticed that I did not explain what I am trying to do / analyse.

    Often HTTP capture utilities display the captures in a format that is readable for a human. This may not be the format that was sent to the server.

     

    What I am attempting as the first step is: Verify that the SOAP Transmission from SOAPUI identical to the SOAP Transmission from HP LoadRunner 12.0x

     

    I asked about Security of the SOAP message because the comparison is not easy if SAML is introduced as the security layer of the SOAP message and if WS-Security implementation.

     

    I assume Yes to security, however the previous description looks like NO. I therefore assume a concreate Web Service not a Virtual Web Service. I also assume that there is NO web_set_user statement in the loadrunner script.

     

    Steps:

    1. Proxy Configuration

    Use LoadRunner Remote Proxy (Important: Remote Proxy Configuration)

    OR

    Use a Third Party Proxy (eg TCPMon)

    The Remote Proxy is used to record the SOAPUI and LoadRunner transmission to the server.

    NOTE: record / capture each product in its own session do NOT record both in one session.

    You can use LoadRunner Remote Proxy to record non-browser based applications. E.g SOAPUI

     

    1. Compare the Capture

    Use a comparison tool to compare the Proxy Log files (Capture files) Eg. WinDif.

    Pay attention to

    1. Headers used by SOAPUI
    2. SOAP message format. It should be a complete buffer with no CR/LF. It should look like a complete string.

    Please confirm that both the Transmissions are Identical in size and format.

    Once the packets are confirmed to be the same. I like to check how the buffer is loaded into the web_custom_request of HP LoadRunner 12.x.

     

    NEW in HP LoadRunner 12.x

    LoadRunner 12.0x introduces new features; Support capture files. I can now use Third Party proxy for applications that HP LoadRunner will not record.)

     

    I notice that it is about 100ms difference.

     

    As a long time user of HP LoadRunner, I am very intrested in this Topic / Observation