Can you say your functional test suite is efficient? How much value does it bring to your organization? How much would it cost to have a team of manual test specialists instead of automated test suite? Will manual system verification be more error prone than automated test suite?
Many teams and companies won't be able to answer these questions with numbers. Having automated test suite doesn't mean you are automatically bringing value to your company. It can even be opposite. Without the numbers, it is impossible to understand and make strong decisions on the quality of tests in your project. Even if you polish test code and follow all Clean code principles tests may have a negative return on investment (ROI). As a manager why should I blindly decide to put money in a test automation? That may be a good excuse to say there is no budget to write tests.
In this post, let's think about key performance indicators of automated test suite. These indicators are different for developer\QA specialist, product owner and program manager.
Let's start from developer\QA engineer perspective
As a developer, I want to have reliable tests. Tests that don't fail for a working feature and do fail when the related feature is broken. Numbers of false negative and false positive results for each test should strive to zero. I want my tests to perform as fast as possible to maintain fastest feedback loop. Maintenance costs should be low.
So as a developer, I am interested in an execution speed, a number of flaky runs and maintenance cost. The last one is tied to code quality and can be calculated from static code metrics (e.g. maintainability index, cyclomatic complexity, LoC). Taking into account these metrics I can judge how bad is a particular test.
Each flaky run should be characterized with reason of failure - be it environment issue, incorrectly developed assertion etc. This will help developer make decision on a way to improve quality of the test.
Product owner perspective
Product owner as the main stakeholder wants to maximize product business value. A team won't move forward fast if they continuously look back at broken tests or regression issues.
As a product owner, I would like to know:
- an amount of time spent by the team on supporting flaky tests;
- the number of issues discovered by automated tests and manually;
- type of the issues - new or regression.
If time spent on flaky tests is significant - it's time to improve your test suite.
If you continue to manually discover same regression issues - it's time to increase automated test coverage.
Without the numbers, you don't know when the time is right.
Program manager perspective
Here we look at Program manager as a person responsible for managing several related projects, their risks and much more, including overall quality assurance.
It is important to oversee quality assurance trends and consequences from taken actions. If an organization has standardized QA tool or a service, you need metrics on how well it performs across the teams. Did your suite, testing infrastructure, time of test automation improve?
Program manager would be interested in overall health of QA:
- Number of issues discovered by automated tests, manually per project
- Time spent on manual testing per project per sprint
- Time spent on automated test development per project per sprint
- Time spent on flaky test maintenance
Let's review few examples of cases where metrics could work well
Team continuously has problems with Selenium grid which results in slow or unstable tests. Efforts spent in the scope of a project are not significant, but would be notable looking at the organizational level. Suitable action would be to put efforts on implementing stable grid service at organization.
Flaky test runs prevent code from moving forward in CD pipeline which results in job re-runs, longer feedback loop and distraction.
Flaky test runs happen so often that considered as a norm and real failures can skip through.
Tests are maintained by number of people which don't track tests performance. Invalid cases are not escalated. Team is not communicating enough or has no experience to spot weak unit.
We can crystallize following metrics:
- Flaky test run count
- Test execution time
- Number of issues discovered by automated tests
- Number of issues found manually
- Time spent on automated test development
- Time spent on flaky test maintenance
Dashboard with these metrics could give a different level view for all involved parties - be it developer, PO or PM. Developer could get breakdown of metrics per test which points to direct targets for improvement - bet it slow or unreliable test.