Execution configuration suite un-assign tests

[LauraMariaB]

Hello,

I have created an execution configuration suite and a new child execution plan inside of it. However, I am not able to change anything for this child execution; it inherits all of the suite configuration parameters, including the assigned tests. Therefore, I am not able to create a configuration suite with multiple child plans that have different tests assigned. Is this the intended behavior? 

Thank you,

Laura 

[Florian Guenther]

Hi Laura,

yes this is the intended behaviour.

The focus is on the different configurations - I want to know how my test set is doing on different configurations. Therefore the test set is fix but the configurations are different.

Regards,
Florian

[LauraMariaB]

In case of a multiple runs for an execution with automated tests assigned, if at a certain point we will not execute some
of the tests, is there a way to mark their status as N/A ? So a test TC.001 will have status PASSED on first run, but in the second run we will like to have its status as N/A.
I am asking this because we get automatically the status of a test by executing a script. We parse a list of results and search for the status of a specific test in there. If the status is passed (TC.001 - PASSED), the program will exit gracefully and the status will be marked as passed. If the status is failed (TC.001 - FAILED) it will exit with error so the status is FAILED. But what if the test is not found at all in that list? It should be marked as N/A.
We do not run the test from SILK, we just give the result file to SILK and want to get the status in the application. Hope I made myself understood. Thanks!

[Florian Guenther]

No - if an assigned test is not executed it will have the status "Not Executed". It is not possible to have no status (=N/A) as a result.

So your automated tests are executed outside of Silk Central and you then transfer the status to Silk Central - how is this done? Manually? Do you have manual tests in Silk Central representing the external automation?

And those tests are executed on different configurations - e.g. different environments like Windows, Linux, MacOS or IE, Firefox, Chrome, etc.?

Which version of Silk Central are you using - I am asking because in 19.0 we introduced functionality to consume such external results programmatically via a REST API.

[LauraMariaB]

Our tests are Windows Script Host Tests. We run tests externally and we have a script (the Script argument for the Windows script) to parse the results and see if the test have failed or not. We put this result file in a well defined directory for Silk to know where to executed it (an windows execution server). So we do not use Silk for the actual execution, just for result interpretation. We would need to somehow know that even if the test was added to an execution, it was not executed because its status was not found in the results file. Also, is no option to exclude a single test for an execution run. You either run all tests or all the failed ones.

Our Silk version is 18.5 and I do not have any idea if some update is scheduled in the near future.
Thank you!

[Florian Guenther]

The REST API and external execution in SC 19.0 would be a perfect fit for your use case. BTW 19.5 will be released mid of February.

It is not easy to produce a Not Executed status for WSH tests deliberately - you can do it either by having no script or by having some invalid calls in the script.

Would something like the following work? Where you handle the case where you receive no result with some invalid call "WScript.SetNotExecuted()"?

var result = "NoResult";
WScript.Echo("result = " + result);

if (result == "Passed") {
  WScript.Echo("Passed");
  WScript.Quit();
} else if (result == "Failed") {
  WScript.Echo("Failed");
WScript.Quit(-1);
} else {
  WScript.Echo("Test has no result in last test run");
  WScript.SetNotExecuted();
}

[LauraMariaB]

Yes, this might do the trick but I do not get what exactly should be this call: WScript.SetNotExecuted(); I think that Silk knows only if the program exited successfully, so it set the status as PASSED/FAILED. Doesn't this means that the WScript.SetNotExecuted() call will be interpreted like a Failed status?

[Florian Guenther]

WScript.SetNotExecuted() - is an invald call, it does not exist, and will result in a Not Executed status in Silk Central, as it is not that the working script produced a failure and therefore would be a Failed but the script itself (or a part of it) cannot be executed due to invalid statements.