How to Identify Test Cases[]
In general Test Cases are created in 2 scenarios:
- Validating new functionality works as intended
- Reproducing a defect to validate the resolution
Test Cases for new functionality should be authored by the same team that implements the functionality during the same Sprint. The User Stories should drive the creation of the Test Cases; specifically the Acceptance Criteria (or whatever mechanism is used to capture requirement details) should be used to identify necessary Test Cases. As the Stable Team Testers are a primary consumer of the Acceptance Criteria (in addition to Developers) it is expected they be closely involved in the elaboration and review of these Acceptance Criteria (typically performed during Sprint Planning). Test Cases should be identified that prove out the natural path through the functionality, but also Test Cases for the various edge cases that result from the requirement details.
An important question is how many test cases are necessary for any given piece of functionality? There is often no limit to the number of scenarios, or permutations of data that could be used to build test scenarios. It is important to have enough test cases to build confidence that the functionality is working as intended. How much is enough is a judgement call that the test author has to make based on the complexity of the functionality, the impact of having it wrong (e.g. is the impact a user gets one too many emails, or somebody doesn't get paid thousands of $'s), and how fragile the area of the system has been historically.
For bugs that are found as a result of running a Test Case (either during Functional or Release Testing phases), when the bug is created using the MTM tool it will automatically be linked to the Test Case that found it.
For bugs that are discovered outside of running Test Cases (e.g. reported by a user in production, found during exploratory testing, etc), a Test Case should be created to represent the repro steps and linked to the Bug Work Item.
Basic Test Case (Title, Steps, Links)[]
The above image is a Test Case in TFS/MTM. The 3 most important pieces of data that belong to a Test Case are pointed out:
A - Test Case Title[]
The title will be used to identify the Test Case when viewing a list of test cases. It should describe the specific scenario that this Test Case is intended to test. It should be descriptive enough to distinguish the Test Case at a glance from other like Test Cases. However, Test Cases will be linked to a User Story, and it can be assumed that the person viewing the Test Case title knows which User Story this Test Case belongs to, so the title only needs to be descriptive enough to differentiate between this Test Case and the other Test Cases belonging to that User Story.
- Example User Story: Plan Members of Plan X Only Qualify for a Maximum of $50,000 Per Calendar Year
- Good Test Case Titles
- Single Claim $52,000
- Multiple Claims, Total of $49,000
- Single Claim: $50,000
- Multiple Claims, Total of $50,500
- Bad Test Case Titles
- Creating multiple claims totalling more than $50,000 when a person is a member of Plan X that only qualifies for a max of $50k
- $49,000
B - Test Steps and Expected Results[]
The Test Steps Actions/Expected Results are the core of the Test Case. These are the step-by-step instructions of the actions to take, and the expected results that should be verified. Not every Action should have an Expected Results, however in the cases where it is used, the expected result should be paired with the Action that causes that result to occur. In some cases where there are many items that need to be verified after an action is taken, several Test Steps may be created to indicate to the user which items to validate and the expected value of each. Every Test Case should have at least one Expected Result (otherwise, what is it actually testing?).
The plan is that Stable Team Testers with deep domain knowledge will be the primary test case authors. However, the goal is to have staff with minimal domain knowledge responsible for re-running those Test Cases, and eventually converting them to automated regression tests. For this purpose we will use a fictional persona called Martha. Martha is a new hire who has received 1 week of systems training. The Test Steps should be written with sufficient detail that Martha would be able to execute the Test Case without any assistance.
C - Test Case Links[]
Work Item links are the primary mechanism used to organize the library of Test Cases. Test Cases should have a link to the User Story (or Defect) that this Test Case was written to test. That is the typical method that will be used to look up and organize Test Cases. In addition, any Defects that are discovered as a result of executing this Test Case should be linked back to this Test Case.
This information is visible in the All Links tab of the Test Case (the Tested User Stories tab contains a subset of this data). These links are used extensively for reporting and analyzing test data (see Reporting for more details on how this is used).
Test Case Best Practices[]
There are some common qualities that make a test effective:
- Independent
- Repeatable
- Valuable
- Small
Independent
An independent test is one that does not require other tests to be run either before or after - it is stand-alone. Writing independent Test Cases provides flexibility when planning out test runs, as any combination of tests in any order is valid.
Repeatable
A test should be repeatable and predictable. Meaning that you can run the same test over and over, many times, over many days/weeks, and the results should continue to be the same. This also implies that the Test Case does not modify the testing data in a way that would prevent the test from being run again.
For example, if there is a Test Case that deletes a specific Customer, that Test Case must also re-create that Customer at the start or end of the Test Case otherwise it would not be repeatable.
Valuable
A test should test a scenario that provides value to the team by building confidence that the software is working as intended. This is is often a subjective quality, but as an example there is probably little value in writing a Test Case that has the tester type a value into a field, then simply verify that the value shows up in that field. What would be more valuable is having the Tester enter several values (e.g. creating a new customer), then navigating to a different screen and verifying that all the values input have been saved and are displaying correctly.
Ultimately, it should be a decision made by the Stable Team Tester whether a given test case is valuable or not.
Small
Assuming you have satisfied the Independent, Repeatable and valuable criteria, the next consideration is size of the test. Should we write one big test, or 3 smaller tests? In general, many smaller tests is better, assuming those tests are all independent, repeatable and valuable. Smaller tests give more flexibility in terms of test planning (ability to just plan/run one of the many small tests), and the test results convey more information (now we can have 2 passing tests and one failing test, instead of one giant test that fails. This gives more specific feedback to where the failure is).
Often there will be a tradeoff to be made between value and size, this decision should be made by a Stable Team Tester as they have the knowledge necessary to make the best decision.
[]
Shared Steps is a feature available in MTM that allows the test author to reuse a set of Test Steps across many Test Cases. It is common for a set of Test Steps to appear in many Test Cases. For example, many tests will start by launching and logging in to the application, which may comprise several Test Steps. Once these common steps have been identified, they can be factored out into something called Shared Steps, that can be reused in many Test Cases.
The benefit of using Shared Steps is that when authoring new Test Cases you can easily "drop in" existing Shared Steps instead of writing the steps from scratch (or copy-pasting from other Test Cases). In addition, should the steps ever change (e.g. the flow of the Login screen changes requiring changes to the Test Steps that describe the login process), by using Shared Steps the change only needs to be made in one place - as opposed to if those steps had been copy-pasted between many Test Cases, then each of those Test Cases would need to be updated.
There are no concrete rules about when to use or not use Shared Steps. It will be the responsibility of the test author(s) to familiarize themselves with the library of Shared Steps and use them as appropriate.
It is important to be careful when editing pre-existing Shared Steps. Any change made to the Shared Step will affect all Test Cases that reference it. When updating a Shared Step the author should be examining all linked Test Cases to ensure that the changes will not have a negative affect.
Keeping the library of Shared Steps organized and small will make it easier for Test Authors to discover and find relevant Shared Steps for use. For this reason is it recommended not to pro-actively create Shared Steps (creating Shared Steps on the hunch that they might be reused in the future), but rather to harvest Shared Steps from recently authored Test Cases where similar steps were identified. The goal is that all Shared Steps should be used in at least 10 Test Cases.
Test Case Parameters / Iterations[]
Test Parameters / Iterations allow the tester to run the same set of steps with different data values. This is common when testing the same screen with slightly different input values to test various edge cases. Rather than authoring a separate Test Case for each set of input values, you can author one test case with parameters, then simply provide a list of different values to use in each test iteration.
When authoring Test Steps, there is the ability to define parameters. This can be done by simply writing in the parameter names prefixed with the @ symbol.as part of the Test Step. At the bottom of the window is a parameters grid that will automatically have columns added as the Test Steps are edited and parameters are defined.
Once the Test Steps have been entered and parameters identified, the test author can create rows in the parameters grid to represent each test iteration and the different input values to be used in each.
Now onto the more difficult questions: when should you use iterations, and when should you not?
Often there will be situations where you could write multiple test cases, or with a few tweaks you could write one general purpose test case with several iterations. Sometimes you may even be able to write a a very generalized test case with many many (many!) iterations.
The main advantage of using one Test Case with multiple iterations is the ease of maintenance. There is only one set of test steps to maintain, and only one actual Test Case to worry about. This generally makes maintenance easier. If the Test Steps need to change for some reason, there is only one place to change them. New sets of input values can be added pretty easily by adding new iterations.
However, the main downside of using multiple iterations is the MTM tooling assumes that when you execute a Test Case you will always execute all iterations. This limits the flexibility for test planning. There is no straightforward way to plan to run only some iterations, it's all or nothing.
Test Authors will need to make a tradeoff between ease of maintenance and test planning flexibility when deciding whether to combine test scenarios into one Test Case with multiple iterations, or create multiple Test Cases.
Even if not using multiple iterations there is some benefit in using parameters. By making input values into parameters (e.g. @claimNumber) instead of typing the actual claim number directly into the Test Step; when executing the test the parameter value will automatically be placed onto the clipboard for easy pasting into the software under test. In addition, if the claim number is used in many steps throughout the test case, by making it a parameter it allows you to maintain the value in one place, and easily change it if necessary.
See Also: Executing Tests with MTM
[]
When creating Shared Steps it may be necessary to include parameters in the steps. For example, if the Shared Steps are performing some actions with a claim, it is common to parameterize the @ClaimNumber rather than embedding a specific number in the Shared Steps. This allows the various Test Cases that use the Shared Step to use different claim numbers, and pass in the specific claim number being used to the Shared Step(s).
When a Shared Step is included in a Test Case all the parameters defined by the Shared Step will automatically be added to the Test Case parameters grid.
Consider the scenario where one Shared Step uses the parameter @claimNumber, and another Shared Step uses a parameter named @claimNum. If a Test Case needs to use both of these Shared Steps, the Test Case will have both parameters in it's list even though they represent the same value. Ideally this situation should be avoided. There will be list of common parameter names that should be used where possible for consistency sake.
Test Case Lifecycle[]
There are 3 important stages in a Test Case Lifcycle:
- In Repository - All Test Cases start in this state once they have been created in TFS/MTM
- In Inventory - This indicates that a Test Case has met a certain quality standard and follows all the practices that define a high quality Test Case
- Automated
When we talk about the Test Case Repository we are referring to all Test Cases that exist in TFS regardless of their quality level. When we talk about the Test Case Inventory it is a subset of the Repository that only includes Test Cases that follow best practices, and have passed a review process. The purpose of differentiating between Test Cases in the inventory and not in the inventory is to allow testers to more easily find test cases to reuse, or to find candidates for automation, without having to spend time looking through many lower quality test cases. By allowing testers to create Test Cases in the Repository, it provides a lower barrier to entry to begin using MTM/TFS for their own personal testing. They can quickly create a test case that is lacking details, for their own personal use. Later on that Test Case can be refined and evolved if we wish to share it with others and/or consider it for automation.
Once a Test Case is In Inventory, there will be controls around who is allowed to modify that Test Case (as modifications would imply the Test Case needs to be re-reviewed). Once a Test Case has been automated it is even more important to control what changes are made, as any changes to the Test Case could require changes to the Test Automation.
In order for a Test Case to be included in the Inventory it must follow the following guidelines:
- Descriptive Test Case Title
- Test Steps detailed enough for "Martha" (see Martha Persona) to execute it without assistance
- Parameter Names using common names from list where relevant
- Only using Shared Steps that are in the Shared Step Inventory
- Link to at least one User Story or Defect
- Accurate links to Shared Steps used
- At least one Expected Result
- Expected Results are used appropriately
- At least one passing Test Result
- At least one video of a successful run that has been attached to the Test Case itself
This list of criteria should may evolve over time. For example, once the Stable Data system is in place it should be required that Test Cases in Inventory use Stable Data where possible.
More Test Case Meta Data[]
There are many additional fields included on the Test Case work item. Most of these fields are optional. The vast majority of them are used to capture additional meta-data about the Test Case. This meta-data is used for querying, reporting and organizing the Test Case Repository.
Most field titles are a clickable link that will take the user to a help page describing the purpose of the field, and providing guidance on what value should be input.
