Can results files be compared in SilkTest?
Yes, there is the facility to compare results files which are the result of having executed a Testplan. The "Compare Two Results" command allows a user to quickly note only the results that have changed from a prior run, without having to look at the same errors over again. The command identifies differences based on the following criteria:
- A test passes in one Testplan run and fails in the other
- A test fails in both runs, but the error is different
- A test is executed in one Testplan run but not in the other
SilkTest uses the Test Descriptions as well as the Test Statements to identify and locate the various cases in the Testplan. Therefore, if the descriptions or statements are changed between runs, SilkTest will not be able to find the test when it runs the "Compare Two Results" feature.
To find differences between Testplan executions:
- Open two results files.
- Make the results set you want to compare to another results set, the active window.
- Select Results -> Compare Two Results.
- On the "Compare Two Results" dialog select a results set from the list box and click OK.
When SilkTest displays the results set again, it positions a coloured arrow in the left margin for every test that is different. A red arrow indicates that the difference is due to the pass/fail state of the test changing.
A magenta arrow indicates that the difference is due to the addition or removal of the test in the compared test run.
- Select Results/Next Result Difference to search for the next difference or select Results/Next Error Difference to search for the next difference that is due to the change in a pass/fail state of a test.
SilkTest uses the Test Descriptions as well as the script, testcase, and testdata statements to identify and locate the various cases in the testplan and in the results set. When test results overlap in the two results set that were merged, the more recent run is used. If a test description is changed between runs or the statements modified, SilkTest might be unable to find the test when attempting to merge results. SilkTest places these so-called orphaned tests at the top of the results set.
NOTE: This facility is only available for results files which have been created from a Testplan execution. Results files which have been created by Script executions have no automated feature with which to compare two sets of results.