Oftentimes when we talk about Test Automation in the Product Engineering jargon, one of the first questions from the stakeholders / project sponsors is “What is the Return on Investment”. It seems this is asked because Test Automation is an expensive proposition. Unless careful thought is given to what to automate, when to automate, how and how much to automate, the returns adversely impact the team in terms of time, cost and test coverage. So, it is rather important that the stakeholders ask this question upfront which makes the team think about the business and technical value from the test automation effort, present their data points to the team and get their buy-in instead of diving into a random effort.
Today’s article regarding ROI analysis is based on the findings of the software testing company TestFort, which you can read about in more detail in the company’s blog. This article is going to share a few core areas of returns of investment whenever test automation is undertaken. Hopefully, this information will come in handy for anyone who is undertaking such an ROI analysis to ensure they have covered all areas of ROI.
Think about ROI from both tangible and intangible sources. Several people stop at the tangible side which is very important to show measurable returns to the management, but fail to understand that it is their responsibility to explain the intangible returns too.
On the tangible side typically, the returns focus on
1. Time Savings – savings that can be quantified easily explaining how much time would be saved by having automated tests, which would otherwise have to be run manually.
2. Cost Savings – measurable savings from reduced time to test, reduced number of manual testers needed to complete the job, reduced risk of untested areas and reduced risk of product release with undetected bugs.
3. Extra test coverage – with the current development practices where there is a lot of system and service level communication, manual testing alone is not going to suffice to get the required test coverage. Areas such as web services, voluminous databases to be tested, complex performance test scenarios all are going to be impractical if not impossible to be tested manually. The extra test coverage that test automation delivers can be easily measured through instrumented means and code coverage techniques. Other areas typically considered for test automation include: regression tests, build verification/smoke tests, localization tests, hot fix / patch test suites.
4. Risk Reduction – by closely working with the marketing and business team, the test team can determine the risks that test automation helps reduce by quantifying the impact of bugs found by automation. These are bugs which could have been found only by automation; showcasing the impact of such bugs going undetected, will really help the management appreciate the value of such automated tests. There is an element of intangible customer satisfaction that is involved here, but a rough measure of the impact is often good enough to explain the returns.
Intangible side and its questions
On the intangible side, it is important to consider the following while making a case to the management / stakeholder team:
1. Employee Satisfaction – Test automation often provides bigger and better opportunities to the testers on the job and makes the work more interesting and challenging, rather than being bogged by mundane manual tests. This is especially true when automation is taken up on tests that are repetitive in nature. While the satisfaction is often not measurable, this can be represented to the management team through creative ways such as “feedback from employees, satisfaction index that shows their job satisfaction levels before and after they were exposed to test automation, employee retention percentages that can be attributed to test automation opportunities etc.”
2. Creative, Out of Box Thinking amongst teams – Automation is not a very straight forward testing ammunition. One has to understand the business and technical sides of it to ensure its returns are justifiable. To be able to do this, one has to often think of creative workarounds, new frameworks that they may need to create, freeware/open source tools to leverage etc. Such creative scenarios can be a major morale boost making the job very rewarding for the tester. That said, the tester’s manager herein plays an important role in guiding them in the right direction to give ample scope for creativity yet ensuring the automation effort stays on track.
When such intangible benefits are also considered in explaining the ROI on test automation, the management team will really appreciate the holistic thought that the team has provided increasing the chances of their support and buy-in, for the overall test automation effort.
Summary
On the tangible side, ROI is derived from time savings, cost savings, extra test coverage, and risk reduction. Time savings come from automating tests that would otherwise be done manually. Cost savings result from reduced testing time, fewer manual testers required, and decreased risk of undetected bugs. Extra test coverage is achieved through automation, covering areas that are impractical or impossible to test manually. Risk reduction is quantified by identifying the impact of bugs found through automation.
On the intangible side, ROI is associated with employee satisfaction and the promotion of creative thinking within the testing teams. Test automation provides more engaging opportunities for testers and allows them to think creatively to overcome challenges. When presenting ROI analysis, it is crucial to consider both tangible and intangible benefits to gain support and buy-in from management.
