TEST DOCUMENT VIEW not usable/correct after tests rerun due to automatic overlay of 'NOT EXECUTED' status
The mechanism to set each test run just launch automatically to NOT EXECUTED result generates a not correct information in the TEST DOCUMENT VIEW. This is true expecially when you are at second or next runs of same tests and its depends by the test execution dynamic.
In our experience the tests are rexecuted several times in same project, and a test run take days to be closed, so the test manager need to work in 'test result continuity', he need to know always the last consolidated result (set by its Tester, not by SCTM automatically) to be aware about the current 'real' status in the project.
For example: we have 100 tests in total with finished EP and this result: 60 PASSED, 30 FAILED and 10 NOT EXECUTED, as shown in Test Documentation view. A new build is installed and now the test manager decides to re-run 50 tests, all the FAILED and 5 NOT EXECUTED and a portion of the PASSED (5 Test for regression scope) tests, so immediately the Test Document view shows that 40 test are PASSED and 60 are NOT EXECUTED.What's happen to the previous 'project status' in term of FAILED and PASSED ? disappear!
Question: is it a correct information about the current status of the project ? It is useful ? what this information says? Help me to understand.. please.
Our opinion (15 test manager opinions) is that this information is incorrect and not useful. The real information that the test manager needs is to understand where is the project 'cross runs and builds', minute by minute, for a angle view and which tests are to be confirmed (because are running again) in another angle views. The currect document view cannot help in any ways.
The test status, when a test is just launched, cannot be 'NOT EXECUTED', maybe we can consider a 'RUNNING' or 'PROGRESS' status, or better NOTHING. Only when the Tester select the test and set a result, it MUST BE consider, never before, as all the other main competitor does (HP QC and IBM QM).
Thanks a lot for your attention.
Deciding which status should be shown at which time is an ongoing discussion topic.
Over the years different approaches have been implemented and released – the current state is that the last status is shown real-time. Hence if you set a status in your manual testing area it is immediately shown in the tests area. This also means that on starting an execution all tests are set to “Not Executed” as this is the initial status for tests within an execution – as the test has not been executed for this specific execution.