Test Automation in Agile Development Using Silk
Compared to traditional software development projects, the role of test automation in agile development projects is more important and significantly different. The close collaboration between various stakeholders in agile software projects also changes the way that quality, and with it testing, is managed. Managing quality is not seen as a discrete discipline with limited focus and responsibilities; it is viewed more as an integral part of overall project management with multiple stakeholders sharing responsibilities for quality. This approach leads to different requirements for test automation and test management tools. Traditional test automation and test management tools often cannot accommodate the required changes to processes, skills, attitude, and responsibility that the shift to agile management practices brings.
This paper elaborates on key principles of successful test automation within agile development projects. It also attempts to explain how the changes in collaboration and processes in agile projects have affected successful test automation principles and test management tools. Finally, this paper shows how SilkTest and SilkCentral Test Manager (SCTM) fit into agile development projects, explaining the benefits that differentiate these tools from traditional test automation and test management tools.
Continuous Test Automation: Key Principles of Successful Test Automation in Agile Software Development
Manual testing is not an option
Test automation is a must for agile development projects. The goal of agile development is to deliver production-ready code at the end of each sprint. This requires rigorous testing throughout each sprint. The effort required for manual testing is too high to accommodate repeating test cases for existing code (manual regression testing) with each sprint. Agile development projects must therefore choose between automating testing, falling short on quality, or developing code using waterfall methodologies such as relying on long manual regression testing cycles at the end of each release (the more hardening sprints a project requires, the more waterfall it is). Unit testing alone cannot ensure the quality required for each sprint. System- and integration testing are also needed. This means that test automation needs to begin early in agile projects. As agile projects deliver customer value with each iteration, they also deliver features that must be tested from the functional/UI perspective with each iteration.
Detect defects as early as possible
Another reason to begin functional testing early is that the cost of fixing a defect correlates directly with the period of time between introduction of the defect and identification of the defect. The shorter this time span, the better; this is a core principle of agile development methods and test driven programming practices.
Agile teams want to know what failed in the current build, not what failed in previous builds days earlier. Finding the root cause of failures is much easier when you know what changed between a successful test run and a failed test run. This is only possible when you minimize the changes that occur between tests. Changes must be checked in regularly, triggering frequent builds that include test automation runs. The earlier you begin and the more often you run test automation runs, the earlier you will identify defects and support agile development processes.
Minimize test-script maintenance effort
The effort required to maintain test automation scripts is a key reason why traditional test automation is often not successful in environments where an application under test is undergoing rapid change. Starting early with UI testing while an application is still evolving implies that significant effort will be required to maintain test automation scripts as the application changes. Sometimes small adjustments to the application under test can dramatically improve the testability and maintainability of the application with test automation tools.
Minimize results-analysis effort
Another important factor in practicing continuous test automation is the cost of test-result analysis. The more often you run tests, the less effort should be required to analyze test results. Test automation needs to run reliably and deliver results that you can trust. Having to differentiate between test-script failures caused by your automation tool and actual application defects will stall your test automation effort.
Continuous integration (CI) requires continuous test automation
CI is a core principle of agile development. Though CI alone (continuously building and running unit tests in combination with static code analysis) cannot achieve quality goals in the absence of test automation for system- and integration testing. Continuous test automation based on the UI level is essential in ensuring that regular system- and integration testing is performed with minimal effort.
How SilkTest Supports Continuous Test Automation
The faster you test, the more you can test. The faster you test, the more often you can test
Continuous test automation (testing early and often) is only possible when automation runs are performed in sync with builds. If your test-automation runs take more time than your build cycles, you will not be able to get test automation results for each build. Agile teams demand test automation results on a per-build basis (at a minimum, for nightly integration builds). SilkTest delivers the fastest playback speed for UI automation available in the market. This enables you to do more testing in a shorter period of time, thereby providing the timely test results that agile teams require.
Robust & flexible control recognition
As UI’s evolve over the course of an application's lifecycle, particularly during the early phases of development, they present a major challenge to test automation. Because visible UI properties such as captions, labels, styles, or window position are subject to change, they cannot be relied upon to identify UI objects. In agile teams, where there is strong collaboration between developers and testers, developers can improve the testability of their applications by introducing stable automation properties for UI controls that facilitate reliable UI object recognition when other properties of the control may change over time. SilkTest provides the strongest support in the industry for such custom UI properties and is uniquely suited to support collaborative teams that care about application testability.
When the introduction of special test automation properties into applications under test is not an option, for example with applications using third-party UI frameworks that do not provide access to the application layer, SilkTest provides the most flexible approach to building reliable and stable identifiers for UI controls. This is in contrast to other test tools that rely on a small subset of properties for object identification.
For details see also:
Reliable test-automation results
The more test results you receive, the less effort you can expend analyzing test failures. SilkTest provides not only the fastest playback speed, it also provides the most reliable test execution, minimizing test failures related to test-script synchronization issues. SilkTest’s out-of-the box synchronization is unique in the industry.
For details see also:
Collaborative Testing: Key Principles in Successful Agile Team Testing
Test automation is not done in isolation
Traditional development projects divide developers and testers into separate teams. Collaboration in traditional teams is built around the “throw code to testers” and “throw bugs to developers” dynamic. In traditional development projects, the most important tool for team communication and collaboration is the defect tracking system. In contrast, agile teams are cross-functional; they include developers, testers, doc writers, and domain specialists. Testing is not viewed as an activity that is solely the responsibility of testers. Every team member is responsible for quality and so each team member contributes to the assurance of quality. Test automation is often a collaborative effort between product owners, testers and developers. Product owners define acceptance criteria for user stories which serve as templates for the automated tests that are implemented by testers. Developers help testers by improving the testability of applications; they also help testers integrate automated tests into reusable automation frameworks. This however is not the only pattern of collaboration: agile team members often share and swap job functions based on the needs and skills of the individual team members. For example, in teams that have few testers, developers may create test automation scripts.
Test management and test automation tools need to fit into the agile toolbox
Agile teams rely on a range of agile project management tools to manage releases, user stories, defects and tasks. Source control management systems, build management systems that provide continuous integration, and xUnit testing frameworks for unit testing are basic equipment for every agile team. Some teams extend their toolbox with acceptance testing frameworks such as FitNesse. Agile teams also rely on tools to help them manage testing efforts that aren’t directly related to user stories in the current iteration. These testing efforts include regression testing, configuration testing, and performance testing.
To integrate with the testing frameworks that agile teams use, test automation tools need to be lean and flexible enough to not conflict with existing team workflow.
Code and tests are managed jointly
Code and tests need to be managed together. As automated tests are created and used by the same team in the same cycles in which code is created (iterations and builds), the best way to manage tests is within the same source control management system, following the same guidelines for versioning and labeling.
File check-in triggers builds and tests
Continuous integration is built on the principle that checked-in code changes trigger new builds. The same principle applies to automated tests; when tests are checked-in, they need to be immediately available for test execution. No manual interaction, such as adding tests to a test plan, should be required to execute tests with the next build.
Automate testing for both new features and stable existing features
In traditional waterfall projects where testing is done at the end of a release, there is often no time to automate testing for new features. Testers must manually test new functionality, manually identify defects, manually report defects to development, and manually verify fixes. Test automation is often run as a parallel project, automating testing for older stable functionality, lagging one or even more releases behind development in testing new functionality. In agile projects, automated tests are implemented along with the development of new features with the goal of assuring that user stories are implemented as required (automated acceptance testing). Automated acceptance tests gradually accumulate over the course of multiple iterations and releases into robust regression test sets that can be used to verify the quality of all existing functionality. Requirements coverage is still an important metric in assuring that existing functionality (requirements) are tested sufficiently.
Benefits of SilkTest and SilkCentral Test Manager in Agile Project Collaboration
Multiple test authoring interfaces for automation are offered on a common architecture
SilkTest allows agile teams to generate scripts using their preferred method based on the skills of team members – either visually or using a scripting language such as VB.Net, C#, Java, or 4Test. Underlying mechanisms such as UI object recognition, synchronization, test interfaces, and results analysis are exposed through a common engine (SilkTest Open Agent) and remain unchanged. So even when written in different languages and work environments (SilkTest Workbench, Eclipse/Silk4J, Visual Studio/Silk4Net, SilkTest Classic/4Test), test automation scripts share common objects and principles that make them interoperable across environments. This fosters collaboration between team members that work in different environments (i.e., testers and developers).
Built-in support for testing frameworks
SilkTest provides support for all leading xUnit testing frameworks, including JUnit, NUnit, and MSTest. Silk4J tests use JUnit as a testing framework, making it easy to integrate Silk4J tests with CI/Build Management systems such as StarTeam TeamInspector, Hudson and Cruise Control. Silk4Net tests use MSTest or NUnit as their testing framework and thus they are easy to integrate into Microsoft’s Team Foundation Server for build verification testing. Silk4J and Silk4Net also work seamlessly with FitNesse/FIT a popular acceptance testing framework. Silk4J and Silk4Net offer lightweight solutions that do not get in the way of test automation engineers’ work. This is the why Silk4J and Silk4Net are so easy to integrate with agile development tools and processes.
SilkCentral Test Manager, like SilkTest, supports all leading xUnit testing frameworks as well as FitNesse out-of-the-box. Through SilkCentral Test Manager’s unique test package support, SilkCentral Test Manager can consume test suites defined in xUnit testing frameworks within SilkCentral Test Manager test plans without the need to re-create test cases manually. So no rework is required to maintain test plans within SilkCentral Test Manager. Test authors work in their preferred environment adding tests to the xUnit test suite; SilkCentral Test Manager automatically recognizes that new tests are available and adds them to the test plan. Using SilkCentral Test Manager’s distributed configuration testing capabilities, agile teams can increase test coverage applying their unit tests to cover multiple configurations.
Requirements coverage beyond the release and iteration level
SilkCentral Test Manager establishes traceability between tests and user stories managed in agile project management tools such as VersionOne and Rally. Unlike agile PM tools, SilkCentral Test Manager provides a cross-iteration, cross-release view of requirements coverage. Only such a cross-iteration, cross-release view enables agile teams to plan regression testing, particularly when it comes to planning manual tests. Without assessment based on attributes such as requirements risk and testing effort it is difficult to choose the right set of tests. Even when agile teams attempt to automate as much as they can, manual testing tasks remain and require appropriate planning.
Test asset management through Source Control Management
SilkCentral Test Manager is built from the ground-up to support different source control management systems (SCM) for storage and version control of automated test scripts, including popular SCMs such as Subversion, Microsoft TFS, and StarTeam. This assures that tests and code remain in sync; the current version of a test checked in through source control is automatically executed against the current version/build of the code base. Agile teams view tests as part of the code base and manage them jointly.
Text execution through build management/CI systems
SilkCentral Test Manager provides Web services interfaces that allow easy integration into build management/CI systems. This allows build management/CI systems to trigger test executions inside SilkCentral Test Manager while querying the results of test executions that can be displayed within the build management/CI system. Alternatively, when only a subset of UI tests is required to verify an integration build, SilkTest can be integrated directly with the build management/CI system through a xUnit testing framework.