(This post was written by Bilal Issa from the LeanFT R&D Team)
In the process of functional testing, one of the most critical steps is the comparison of actual and expected outputs of the application under test as well as the ability to capture the details needed for post-run analysis. It is also important to have the ability to compare current run results with run results of previous tests or test runs, given a different set of input data.
In unit testing, the final result is the only important result—be it pass or fail (and, of course, the reason for failure). In functional testing, both the trace and context are crucial for understanding the flow, and ultimately for performing the root cause analysis.
In this blog post I will specifically touch on how exactly Micro Focus Lean Functional Testing addresses this need for understanding the flow and performing root cause analysis. I will also demonstrate the basic—and some of the advanced— reporting features supported by LeanFT.
Right off the bat, LeanFT continuous testing solution was developed with reporting in mind. This means that the coding process focuses on the functional steps, rather than having to deal with the actual reporting of events. For example, when a button is clicked, the operation event is automatically added to the final run results along with the relevant context and data—but more about that later.
Let’s look at a simple example that shows a basic flow of buying a product from the ‘Advantage Online Shopping’ website. Here are the steps:
- Open the Chrome browser and navigate to http://www.advantageonlineshopping.com/.
- Click on the Headphones
- Select Beats Headphones
- Add the selection to the cart.
- Checkout and verify that the total price is $179.99.
The application elements used in the example are modeled using the Application Model in order to keep the code simple and readable:
After running the flow using your IDE’s JUnit test adapter (in this case, from IntelliJ IDEA), you will find a convenient menu shortcut to open the run results under LeanFT > View Last Run Results:
The report opens up in your browser, showing the following run results in HTML format:
How does LeanFT report provide context from a functional testing perspective?
- Report header:
The upper part of the report provides the overall status of the run, displaying the number of failures, if any. It also provides useful information regarding the machine on which the test ran, as well as other vital information, such as: Host name, Total Memory, OS and the overall duration of the run:
- Error list:
When there are failures in a functional test, the first thing you would want to do is to quickly locate the errors. The ‘Errors list’ helps you by putting the errors up front.
Not only that, but also when selecting an error and switching to ‘Test Flow’, the same error tree entry will also be selected.
- Test flow:
This section of the report displays a tree containing all the run steps, providing the entire flow for each reported test, such as the package/namespace, class name, and test hierarchy.
Under the test node, the actual steps of the test are listed:
- Test object operations:
As mentioned, the actual steps of the test, along with the operations on test objects, are automatically reported without the need for writing any additional code. But there’s more to it: for each operation step, the tree entry provides contextual information by displaying the test object hierarchy short path using icons and tooltips to help skim through the flow.
- Test object operation details:
The right pane provides even more detail about the test object. It displays the full object path and the properties that describe it. Also, when the test object is modeled in the Application Model, the details include the name and the class name of the containing Application Model.:
The snapshot is useful for analyzing the test flow. By automatically capturing the state of the application under test, the LeanFT report provides a highlighted image of the test object which provides the best orientation as to where the operation was performed:
The report provides a special entry for verifications which are vital in functional testing methodologies. These verifications help the tester pinpoint failures and check the state of the application under test. In LeanFT, verifications get special treatment. We have introduced a shield icon highlighting these interesting checkpoints within the often long flow:
With verification details in the right pane, will also have a comparison table to help quickly analyze the actual vs. expected values.
The code for the verification in this example is:
String actual = app.AdvantageShoppingPage().TOTALPRICE().getInnerText().trim();
Verify.areEqual("$179.99", actual, "Checking the Total price is 179.99$");
- Image verifications:
One of the advanced verification features LeanFT introduces is the Bitmap-based verification. You can simply check the existence of an image in a test object, and seamlessly get the result in the report in the special section that displays the expected and actual images:
The code used for the image verification above is as follows:
BufferedImage beatsHeadphonesExpected = ImageIO.read(new File(".\\images\beatsHeadphonesImage.png"; app.AdvantageShoppingPage().verifyImageExists(beatsHeadphonesExpected, (byte)50);
You can find more on image verification and comparison in a dedicated blog post here.
Using the Reporter API
In addition to all the convenient reporting features discussed above, LeanFT also has a built-in Reporter API which opens up more capabilities to enhance the report.
- Additional run information:
Use Reporter.addRunInformation() to add extra information to the report header, accessible from the “Notes & Run Information” link.
Reporter.addRunInformation("Created by", "John Smith");
- Custom events:
Use the Reporter.ReportEvent() method to add custom steps to the report. This provides more reporting flexibility to the reported flow.
- Reporting context:
Use Reporter.startReportingContext() and Reporter.endReportingContext() to group several steps under an indented tree entry, to emphasize the context of these steps. For example:
You can read more about the LeanFT run report in the LeanFT Help Center.
Lean Functional Testing (LeanFT), is a functional test automation solution, which was built specifically for continuous testing and continuous integration.
LeanFT enables advanced automation engineers and dev-testers in agile teams to leverage standard development IDEs to create automated functional tests, using modern programming languages.
In addition to its SDKs, LeanFT provides plugins for these IDEs, including powerful tools like the Object Identification Center and Application Models.
You can find more information about LeanFT in the LeanFT Help Center.
Thanks to Bilal Issa from the LeanFT R&D team for providing this article!
Feel free to leave a comment in the box below.