Test Automation is intended to save time and improve quality.  This is achieved by enabling many regression tests to be run often with minimal cost.

Ideally all regression tests would be automated, enabling CoreLink to run every regression test on every release.  However, converting a manual test case to an automated test case has a cost, and it may not make sense to automate every test.

Note: The VS ALM Tooling provides two types of test automation: Action Recording/Playback, and CodedUI Automation.  When this documentation refers to automated tests, it is specifically referring to CodedUI Test Automation.

To decide which tests are the best candidates for automation the following information should be considered:

  • Quality of the Test Case - In this case quality refers to how well-written the test case is.  Are the steps clear and specific?  Is the test data used in such a way to make it repeatable?  Are common flows factored out into Shared Steps? etc.  More details on what makes a high-quality test can be found in the section How to Write High Quality Test Cases
  • How often is the Test Case run - Looking at how often the Test Case has been run historically will give a good idea of how much time savings can be realized by automation. The more often a Test Case is run, the more savings can be realized by automation.
  • How stable is the Test Case - If the Test Case steps are frequently or recently updated it may not be a good candidate for automation.  Once a test has been automated it becomes more expensive to change.  You want to choose Test Cases that are unlikely to change in the near future.

There are no specific guidelines about when or how many tests should be automated, however the Enterprise Test Team should report on how many tests have been automated, and track that number over time, setting specific goals for # of tests automated by specific dates.

It is expected that the automated tests will be run as part of most Release Test Plans.  In addition Stable Team Testers may choose to run Automated Tests over the course of the Implementation Test Phase.  This can be initiated either manually by a Stable Team Tester, or by setting up a Nightly Build to run Automated Tests against the Development Environment.  If a Nightly Build is used Enterprise Test will need to take on the responsibility of reviewing the results each morning, and ensuring that any failing tests are analyzed, and any actions required are assigned out to the appropriate team(s).

To start the Enterprise Test Team will be responsible for converting Manual Test Cases into Automated Tests.  Over time the hope is that stable teams can be trained on test automation.

Part of the automation process will involve potentially combining many manual test cases together into a single Test Case with multiple iterations (see How to Write High Quality Test Cases for more info on iterations).  More generic test cases with many iterations tend to be more effective for test automation.  New Test Case Work Items will be created to represent the automated tests, and linked back to the original manual Test Case(s) using the Automated By link type.

The technical details of creating test automation are out of the scope of this documentation.

See Also: Reporting

See Also: How to Write High Quality Test Cases