Many factors can influence decisions about test automation implementation and its strategy.
The most crucial factors are:
- Time constraints
- Needed level of expertise and number of TAEs to develop the TAS
- Test hardware
- Test tool licenses
- Adaptability
- Maintenance
- Support for different platforms (e.g., Web, desktop and/or mobile)
- Continuous integration/continuous delivery (CI/CD) support
- Test management integration and test reporting
Selecting test cases for test automation is usually done by a test analyst who either understands which test cases can be automated, or by a TAE who has the necessary domain knowledge to make such decisions.
- The following are considered when selecting and prioritizing test cases for test automation:
- Is it technically possible to implement the test cases in an automated fashion?
- Are there any technical challenges that impact the delivery of automated test cases?
- Is the team prepared and well trained to do the implementation work?
- Does the coding effort provide an adequate ROI?
- Is there value in running the test cases frequently?
- Is it a functional or non-functional test? Is it part of the smoke test suite, regression test suite, or confirmation test suite?
- Is the test case repeatable?
- Is the test case easy to maintain when the SUT changes due to updates?
- Does the test case cover frequently used business workflows?
- Is there a functional overlap between tests that allow reusability of test steps and test data?
There are certain tests that can only be carried out with test automation. These include categories when:
- Manual test execution time takes longer than what is adequate
- Execution of test cases need to be synchronized
- Test results need to be available in a pipeline
- Large log files need to be parsed for defects
- Precision in timing in tests is required
- Test permutations are required across multiple OSs, browsers, devices, locations, or configurations
- A large volume of test executions and/or data input is required to maximize coverage
- Any non-functional testing that requires automated monitoring and analysis, or input from a huge amount of users such as stress testing or reliability testing
The ROI calculation can provide meaningful feedback at any given moment in a project lifecycle by demonstrating the return for an activity for the effort invested. The ROI calculation can be further adjusted by including both manual tester costs and TAE costs by simply multiplying the time spent with their respective labor costs.
To calculate the ROI, one needs to determine the investment of test automation (i.e., time and cost) and the savings achieved with it:
ROI = Savings / Investment
It is important to note that the savings and the investment can be calculated considering different metrics and data, in different measures and units. In the scope of this syllabus, a simple model is used to demonstrate the approach, with time units and not the cost. If certain activities are only measured in cost, that amount can be converted to time using a rate specific to the project.
Generally, the savings achieved by test automation are because the same tests can be run in significantly shorter time than run manually. This also means they can be run more often. Therefore, the number of tests being executed can be increased.
To calculate savings, you need to consider the following metrics:
- Time to run a test case manually
- Time to run an automated test case
- Number of test cases
- Number of test runs
To calculate investment, you need to consider the following metrics:
- Time to set up test automation
- Average time to develop automated test scripts
- Number of automated test scripts implemented
- Average maintenance time of an automated test script
- Time to run an automated test script
- Percentage of failed automated test scripts
- Number of test cases defined
- Number of test runs
Pass-fail ratio
This is a common metric and tracks the ratio of the automated tests that passed to the automated tests that failed to achieve the expected result. The pass-fail ratio = the number of tests that passed / the number of tests that failed.
Ratio of failures to defects
A common problem with automated tests is that many of them can fail for the same reason, i.e., a single defect in the SUT. Measuring the number of automated tests that fail for a given defect can help indicate where this may be a problem. Additionally, one may collect the number of failures per defect and track the reason for failure: defect in the sub-system, in the whole system, issue with the test data, or infrastructure.
Test automation execution time
One of the easier metrics to determine is the time it takes to execute the automated tests. In the beginning of the TAS this might not be important, but as the number of automated test cases increases, this metric may become quite important. This metric includes the build time of the TAS.
Number of automated test cases
This metric can be used to show the progress made by the test automation project. But one has to take into account that the number of automated test cases does not reveal much information; for example, it does not indicate the level of coverage.
Functional coverage of test automation
This coverage indicates the percentage of functional requirements covered by automated test cases.
Code coverage
Code coverage tracks how many lines of code is exercised by lower level (i.e., component) tests.