Using Performance and Coverage Analysis with Nunit unit testing software



Using Nunit with Performance or Coverage Analysis, some assemblies are listed twice in the session files.


The root cause of this issue is that by default Nunit reloads assemblies before each test run.

Details: When nuint starts up, it loads the test assemblies to read their metadata in order to see what tests are exposed by the assemblies. This process does not invoke any code contained within the assembly. However, since the module was loaded, the CRL reports this to our profiler(s) and we report the assembly as being loaded but since no code was executed, you will see either a 0% coverage entry or a 0% timing entry in your Coverage or Performance session file respectively.

When you perform a test, by default the assembly is unloaded/reloaded and the test is then performed. This action, of course, invokes actual code in your assembly and will result in genuine Coverage or Performance metrics being produced in the session file. If you run a second test, the module will again be reloaded resulting in yet entry for the assembly in the session file. In fact, with Nunits defaults settings, you will get n 1 entries for a given assembly in your session file if you perform n tests.

Solution: If this situation does not suit your needs, you can use the Nunit settings to modify the results in your session file(s). By modifying Nunit's default behavior to not reload the assemblies before each test run (Tools | Options uncheck 'Reload before each test run' under 'Assembly Reload') the assembly will be loaded by Nuint only once and you will no longer see duplicate entries in your session files.

NOTE: Making this change within NUNIT could affect the outcomes of your tests. If you run more than one test per session AND your application makes use of any static data, you might see different behavior in your application/tests with the 'Reload before each test run' enabled than you would if it is disabled. This has nothing to do with Performance or Coverage Analysis but rather the nature of static data. Since static data is shared by all instances of its given class it is allocated and initialized at module load. If your tests rely on the fact that the test modules were being reloaded after each run, and therefore your static data was being 're-initialized' for each run, un-checking the 'Reload before each test run' may very well affect the behavior of your application/tests. However, this issue can be addressed by the [TearDown] and/or [TestFixtureTearDown] attributes offered by the Nunit framework. Since these two attributes are invoked at the end of an individual test and the entire test battery respectively, they can be used to reset your static data to it's desired initial state at the end of each test. If this technique were employed, it would have the same effect as reloading the module for each test. The Nuint documentation provides details on the usage of these attributes.


Old KB# 11172
Comment List
Related Discussions