Post authored by Efrat Bar Nahum
What is a percentile?
A percentile is a number under which a certain percentage of scores fall. For example, the 90th percentile value indicates that 90% of tested values are less than or equal to the percentile value.
Percentiles in SRL and SLA configuration
In the load test configuration, we can set the percentile that we want to use. This is the percentage of transactions expected to complete successfully. Another setting, the Percentile TRT, is the expected time it takes the specified percentage of transactions to complete. In the example below, the 90th percentile TRT value is set to 3 seconds. If more than 10% of the successful transactions that ended during a 5 second window have a response time higher than 3 seconds, an SLA warning is triggered.
Percentile calculation in SRL
During a test run, each transaction is executed several time. The number of executions is determined by several parameters, such as: test duration, script, number of virtual users, number of regions, number of emulations, and more.
For each transaction we calculate a lot of data for a specific timestamp. For example: SLA information, number of passes/fails, min, max, and average duration, percentile, etc. By default, we calculate and save this information for each transaction every 5 seconds. For each timestamp and transaction, we calculate and save the transaction percentile. For example: Let’s assume that we calculate and save the transaction values every 5 seconds. We define a test with 10 virtual users and set the percentile to the 90th percentile. Then we define a transaction called “Transaction_1” which runs 3 times every 5 seconds. So we have 10 Vusers running 3 times every 5 seconds, which means that we have a total of 30 executions of “Transaction_1” in a 5 second period. We rank these 30 transactions by duration and find the 90th percentile of this transaction. For each transaction we keep the 90th percentile every 5 seconds.
But what if we also have regions and emulations defined for this test? We have to calculate the percentile for every combination every 5 seconds. For example, if we defined 10 regions and 2 emulations, we will keep 20 (10*2) percentile values for “Transaction_1” every 5 seconds. Then, for every timestamp and transaction, we take all the percentile values (in this case 20) and calculate the average. This calculation becomes the new percentile value that we keep for each timestamp and transaction. To sum it up: We keep a single value for the transaction percentile for each timestamp. If we have more than one region/emulation, this value will be the average of all the percentiles for the specific timestamp.
Let’s return to our example and assume that the test duration is 5 minutes. Since we use a 5 second granularity, we have 60 timestamps. If for every transaction and timestamp we keep a single percentile value, we now have 60 percentile values for “Transaction_1”. Again, we take all the percentile values that we have for each transaction, and calculate the average. So now we have a single percentile value for the entire test for all the regions and locations. This percentile value is the value that we see in our report.
SLA Breakers and Status
In the Transactions table in the report you can also find SLA parameters. % Breakers is the percentage of transactions that broke the SLA threshold at any given time in the test run. SLA Status is the status of the SLA: Passed if the SLA was not broken and Failed if the SLA was broken. In the above example, you can see that the first 3 transactions failed. This is because more than 10% of the transactions took longer than 3 seconds. If the SLA status of one or more transactions is Failed, the status of the entire run will also be Failed.