Idea ID: 2875185

Generate Reports with Offline Results / Runtime Data

Basava Basava
Status : Under Consideration

Collate and analysis works like a charm for small duration tests but not for tests generating high volume of data , duration tests and tests running through CI/CD pipeline.

we are seeing single point of failure for collate and analyze the results for

1) tests generating too much volume of data - This takes collate and creating analysis from couple of hours to 8 to 10 hours. 

2) duration tests - it takes. 8 to 16 hours to analyze the results.

3) CI/CD pipeline - with above two tests the job should wait 8 to 16 hours to complete LRE job and proceed to next action in pipeline job.

why we need to wait this much of time to do post analysis and to make decisions on the test executed. we have data during runtime and offline results, why don't we take this data to generate the HTML reports and complete the test run.

Also, we see collate and create analysis failures due to 'x' reasons (may be issue with infrastructure, issues with LRE itself, process is slow in generating the reports).  

it looks like collate and creating analysis after the test is legacy way and it won't fit with today's agile process where we need to produce the reports immediately after he test.

one way to achieve agile way is to collate and analyze the data while running the test. or integrate LRE with LRE could reporting module. 

by this way there is no wait time once test is completed and there is no failure point for any test running with LRE. The main idea is how can we run the tests in more agile way without any single point of failure. As of now LRE is not designed like that and it is a single point of failure for analyzing and generating the reports.




    we tried but we are looking over all solution for posttest activities. is it a valid business case to wait till analysis is finished to pass the Ci/CD pipeline job...? why cant we present the data from offline results.

    Also, the runtime data and analyzed results never match. for example, average response times from runtime graphs will never match with analyzed html report. if runtime data is valid why cant we generate the report from offline view results and pass / fail the test immediately.


    Regardless to your idea, have you set your test XML(aka "Coded Test" in LRE UI) with following parameter:

    This will set Collate during run and may decrease collate duration significantly.