Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« May 2019 | Main | July 2019 »

June 19, 2019

Test Automation Debacles

In the era of digital transformation and spurring competition, organizations are joining the automation bandwagon without a second thought. Especially in software testing; automation has acquired an important place to address needs of agile and continuous testing processes. While the benefits of test automation are well proven with plenty of mature scripting frameworks available in the market, the death knell to the automation journey comes when testing teams start to struggle with test maintenance fatalities.

The Mayhem

We studied a few QA projects closely, right from test planning to execution, only to realize the damage, that un-managed test automation can do to the automation ROI.

Based on the process maturity and application under test, testing teams use various tools and frameworks implementing automation at varied levels. There are tools and techniques that offer automation in pockets and enable teams to realize instant benefits. However, shortsighted test automation is unable to keep up with the pace with which application features evolve. With teams going agile, short test cycles bedecked with amateur testing practices add to the debacles, hence lowering the automation tests maturity graph. Whereas, application under test continues to evolve with exponential speed. Therefore, the gap between automation tests and application features keeps increasing at alarming rate with time, as shown in Figure 1. With frequent changes in the application, poorly written tests and un-managed 'quick' automation test cycles, the teams get caught into vicious cycle of maintenance costs and decline in test coverage. This upsurge in test maintenance leads to regression defects, hence diminishing ROI from the automation.

TestAutomationdebacles.png

As a matter of fact, project teams land into worse situation with test maintenance during automation cycles as compared to manual ones! We observed teams spend more time in fixing test scripts almost by a factor of 200 as compared to what they spend in manual tests. That explains the steep surge in automation ROI in Figure 1.

The mistake lies in..?

We have blamed automation practices and application dynamics enough. Test automation experts are already looking into streamlining test automation and devising ways to leverage application changes to plan tests better. Did it really help? I don't think so! It may slow down the damage but eventually the irreversible destruction caused by haphazard test automation is realized sooner or later. The primary cause lies in the expectations. Testing teams do realize end to end automation is an incremental process or for that matter 100% automation may/may not be achievable, due to which, the end-user's expectations from test automation solutions stoop to minimal level. That's where the problem starts. It's the automation engineering that needs to be addressed!

We may standardize tools or processes for a testing team to adhere to, wherein test automation could be implemented in pockets and benefits be realized momentarily. However, the teams must realize the tool benefits are as good as the features offered and the automation coverage, which may turn into a nightmare in long-term, with no test maintenance available. Hence it is now necessary to look beyond test automation and address the gaps in how the automation tools and accelerators are engineered to offer better and sustainable automation.

You may find my recommendations on picking up the right ingredients and building a sustainable test automation solution in coming blogs.