Gathering metrics is critical in enabling an environment of continuous improvement, and any process change initiative. If you’re unable to measure performance, you’re unable to tell if the changes you're implementing are making things better or worse.

It’s important to gather data and some basic metrics to allow CoreLink to understand the effectiveness of its Testing processes, and evaluate the effect of any improvements.  Some metrics that should be gathered include:

  • # of Bugs in Production – Bugs in production are the only issues that impact the customer, and they are the most expensive Bugsto resolve (compared to Bugs discovered earlier in the lifecycle).  For these reasons it is important to track and report on how many of these bugs are identified over time.  This is also an important indicator of the effectiveness of the Testing process, as it obviously strives to minimize this number.
  • # of Bugs Found in DEV – Bugs that are identified in any phase of QA or DEV (before they reach production).  This is important to understand, as this represents the value that the Testing capability is providing.  If Testing never finds any defects, then they are providing minimal value.
  • Types of Bugs – Specifically if a quality issue is a regression, or an issue with new functionality.  This is important to understand to help the Testing teams direct their efforts on regression testing, or non-regression testing.  Other meta-data such as sub-system, domain, etc can also be valuable to enable the Testing teams to direct their efforts to receive the most value.
  • Reduced Testing Time – Average time from a User Story being development complete, until deployed to production.  This is part of the tradeoff that must be made with Testing, we wish to find as many quality issues as possible, but we must also strive to do so in the shortest time possible.  In order to collect this metric we need to track the date each User Story was development complete, and the date it was deployed to production.
  • Cost of Quality – The amount of time spent performing Testing activities.  This includes Stable Team Testers, Enterprise Team Testers, and any other staff that may participate in the Testing activities.

See Also: Reporting