Percentile Calculation in StormRunner Load

Micro Focus Frequent Contributor
Micro Focus Frequent Contributor
0 0 2,084

Post authored by Efrat Bar Nahum

What is a percentile?

A percentile is a number under which a certain percentage of scores fall.  For example, the 90th percentile value indicates that 90% of tested values are less than or equal to the percentile value.

Percentiles in SRL and SLA configuration

In the load test configuration, we can set the percentile that we want to use. This is the percentage of transactions expected to complete successfully.  Another setting, the Percentile TRT, is the expected time it takes the specified percentage of transactions to complete. In the example below, the 90th percentile TRT value is set to 3 seconds. If more than 10% of the successful transactions that ended during a 5 second window have a response time higher than 3 seconds, an SLA warning is triggered.lr1.png

Percentile calculation in SRL

During a test run, each transaction is executed several time. The number of executions is determined by several parameters, such as: test duration, script, number of virtual users, number of regions, number of emulations, and more.

For each transaction we calculate a lot of data for a specific timestamp. For example: SLA information, number of passes/fails, min, max, and average duration, percentile, etc. By default, we calculate and save this information for each transaction every 5 seconds. For each timestamp and transaction, we calculate and save the transaction percentile. For example:  Let’s assume that we calculate and save the transaction values every 5 seconds. We define a test with 10 virtual users and set the percentile to the 90th percentile. Then we define a transaction called “Transaction_1” which runs 3 times every 5 seconds. So we have 10 Vusers running 3 times every 5 seconds, which means that we have a total of 30 executions of “Transaction_1” in a 5 second period. We rank these 30 transactions by duration and find the 90th percentile of this transaction. For each transaction we keep the 90th percentile every 5 seconds.

But what if we also have regions and emulations defined for this test? We have to calculate the percentile for every combination every 5 seconds. For example, if we defined 10 regions and 2 emulations, we will keep 20 (10*2) percentile values for “Transaction_1” every 5 seconds. Then, for every timestamp and transaction, we take all the percentile values (in this case 20) and calculate the average. This calculation becomes the new percentile value that we keep for each timestamp and transaction. To sum it up: We keep a single value for the transaction percentile for each timestamp. If we have more than one region/emulation, this value will be the average of all the percentiles for the specific timestamp.

Let’s return to our example and assume that the test duration is 5 minutes. Since we use a 5 second granularity, we have 60 timestamps. If for every transaction and timestamp we keep a single percentile value, we now have 60 percentile values for “Transaction_1”. 
Again, we take all the percentile values that we have for each transaction, and calculate the average. So now we have a single percentile value for the entire test for all the regions and locations.  This percentile value is the value that we see in our report.lr2.png

SLA Breakers and Status

In the Transactions table in the report you can also find SLA parameters. % Breakers is the percentage of transactions that broke the SLA threshold at any given time in the test run. SLA Status is the status of the SLA: Passed if the SLA was not broken and Failed if the SLA was broken. In the above example, you can see that the first 3 transactions failed. This is because more than 10% of the transactions took longer than 3 seconds. If the SLA status of one or more transactions is Failed, the status of the entire run will also be Failed.

Tags (2)
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.