Your needs for executing automated tests may range from "just have them executed and use whatever execution server is available" to "execute them in a specific order isolated on one specific execution server".
Silk Central 18.5 now supports this whole range with the two execution settings "Exclusive execution of assigned tests" and "Distributed execution of assigned tests".
In this blog, we will take a closer look at what these settings mean for your assigned tests and how they interact with each other.
Exclusive execution of assigned tests
What does it do?
The Exclusive execution of assigned tests setting defines if your assigned tests run alone (= enabled) or concurrent to assigned tests of other execution plans (= disabled) on the matching execution server.
For example, imagine an execution plan "EP1" having 5 tests assigned:
Now, if we enable exclusive execution and run the execution plan, the assigned tests run alone on the matching execution server, without any other tests of other execution plans (for example of "EP2", which has the exclusive execution setting disabled):
However, if we disable exclusive execution for "EP1", the assigned tests can run concurrently with the assigned tests of "EP2":
What do I have to consider?
Of course it depends on the type of your tests, if they need to run exclusively or if they can run concurrently with other tests.
For example, certain functional automation tests that interact with the UI might have a problem if other tests try to interact with the same UI at the same time, whereas unit tests should not have a problem if other tests run in parallel.
You should also consider the available resources of your execution server and how many concurrent test executions it is able to handle, as many concurrent test executions will consume each other's resources and therefore might influence each other's results.
To prevent overload, Silk Central 18.5 comes with a new execution server setting (Maximum concurrent executions), where you can specify how many concurrent executions are allowed on a specific execution server.
Will I see if tests ran concurrently?
You will see if other tests were executed concurrently in the test run results' Timeline tab:
And in the test run results' Messages tab:
What does it do?
The Distributed execution of assigned tests setting defines if your assigned tests are executed in the specified sequence on one matching execution server (= disabled) or in parallel across all matching physical execution servers (= enabled).
Again, imagine an execution plan "EP1" with 5 assigned tests and 3 matching execution servers:
If the setting is disabled, all assigned tests are run sequentially on one of the matching execution servers:
However if the setting is enabled, "EP1" is internally divided into smaller units each containing one test. Those units are handled like execution plans and are then distributed to all available matching physical execution servers:
As those small units are handled like any other execution plan the first three tests will be distributed to ES1, ES2, and ES3. The execution server that finishes first will then be given the next one. With this approach the most efficient execution with minimal execution time is ensured.
What do I have to consider?
Enabling distributed execution will have some overhead in terms of time needed for distribution, check out of sources, and fetching results - therefore very short tests might run quicker sequentially on one execution server than distributing them to several execution servers.
Additionally your tests need to be independent of each other, as you do not know on which execution server they will be executed.
Distributed execution is supported on physical (meaning configured as physical) execution servers only, as dynamically starting up virtual execution servers for an execution plan with a huge amount of assigned tests could have a severe impact on resources and money, not to mention the time overhead for starting those machines.
Furthermore you cannot use setup-/cleanup tests when distributing your tests. The reason for that is that setup/cleanup tests usually prepare some environment for the sub-sequential tests and cleanup afterwards, as the environment is rather dynamic/random with enabled distribution setup-/cleanup tests are not supported.
Will I see if tests were executed distributed?
You will find distributed executions by their "<Distributed execution>" information in the "Executed By" column:
In the execution plan run results you will then see on which execution servers the tests have been executed:
Combination of settings
Now that we know what each setting does on its own, we can take a quick look at what happens if we combine them.
Exclusive & Not Distributed
This is the default setting for a new execution plan and means that the tests are executed in the specified sequence on one matching execution server, without having other tests run concurrently.
Exclusive & Distributed
This combination distributes the assigned tests to all matching execution servers - but ensures no other tests run concurrently. You can use this for example if you have functional automated tests that interact with the UI and you want those tests to leverage all available resources to have quicker results and therefore quicker feedback.
Not Exclusive & Not Distributed
In this combination, the assigned tests are executed in the specified sequence on one matching execution server, but other tests can run concurrently.
Not Exclusive & Distributed
This is the most "open" combination which distributes your tests to all available execution servers and allows other tests run concurrently. As each assigned test is transformed into an "internal execution plan", this also means that tests of one execution plan could run in parallel on one matching execution server.
Visibility of progress for automated test execution
Another benefit that comes with these changes is the immediate visibility of test progress for automated test executions on the Activities page. So far the whole information was stored on the execution server only and written to the database when the whole execution plan run was finished. With Silk Central 18.5, the according data is already created in the database at the beginning of an automated test execution, and updated while progressing through the tests - similar to the behavior for manual tests.
Silk Test integration
Above we already mentioned that UI tests might be sensitive to concurrent test executions.
However with Silk Test it is possible to execute browser (Firefox and Chrome) tests concurrently on one machine, consequently with the new Silk Central settings, you can even run the UI tests of one execution plan concurrently on one execution server.
However you have to keep an eye on the resources, as for each test a new browser will be opened - but as mentioned above, you can control that easily with the Maximum concurrent executions setting of the execution server.
What about mobile devices?
The mechanism for automated test execution on mobile devices via Silk Central and Silk Test is the following: When an execution plan is started, Silk Central is looking for matching execution servers on which Silk Test is installed and if the mobile device is free.
As soon as those criteria are met, the execution plan is started and the whole test execution is handled by Silk Test.
This does not change with Silk Central 18.5 - however the availability of matching mobile devices might be a limiting factor, as with distributed execution the amount of execution plans looking for a specific mobile device will rise. Of course if this demand for mobile devices can be met, the available resources can be leveraged more efficiently, execution time is reduced, and results and therefore feedback are available earlier.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.