Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« January 2012 | Main | March 2012 »

February 14, 2012

Test Case Effectiveness - What does it say?

CXO Guide Series - Topic 4

Test Case Effectiveness - What does it say?

While I was analyzing operational data of testing organizations for FY2011 (my client organizations), one data point caught my attention. On an average 50,000 test cases executed across application portfolios and less than 5,000 defects logged and fixed. To my surprise, the data was very similar across multiple organizations (Plus or Minus 15%).  Test case effectiveness is around 10% which means, your factor of safety is 10. We are writing and executive 90% of test cases which are non productive in a given test cycle. This is in spite of using all advance methods of testing like orthogonal arrays, model based testing, Risk based testing, etc. Couple of thoughts came to my mind

·         Are testing organizations doing something seriously wrong or do they really need this level of safety net?

·         Can Test Case Effectiveness metric provide more insights and provide a better explanation on why we need this degree of safety and in turn the resultant cost of quality?

·          Is the lower Test Case effectiveness one of the reason why the CXO community perceives testing organizations as an expensive cost center?

·         Do CXO Community and Testing Organizations really understand the degree of safety that is necessary to maintain the Quality of the application?

 

Well, if the answers to these questions were so straight forward, we would not have asked these questions at the end of the year with the help of a testing metric. But it is very important that every testing organization find its own answers which will be different from each other. Considering the cost of quality of every complex application landscape today, Test case effectiveness is the king of all metrics because it impacts both test planning and test execution efforts.

 

How can I improve test case effectiveness? I have tried to summarize few basic golden rules which are known to everyone but not always practiced

 

Parameter

Strategy

Metrics

Ensure you collect Test case effectiveness metric data after every release

Test Case Writing Techniques

·   Ensure appropriate test case writing techniques are used by testers (Orthogonal Array, etc)

·   Ensure Requirement Traceability Matrix is created and maintained on an ongoing basis which assists in prioritization of test cases and also track changes in requirement

·   Demand 'use cases' and possible exceptions in real life application usage from business users

Reviews

·   End of every release cycle, validate your test case effectiveness, make course corrections

·   Insist on business review of your test cases with an intent to improve TCE

Defect Patterns

Identify modules/functionality which needs attention based on past defect trends and which are stable

Duplication

Avoid duplication of test execution b/w Dev, Test and UAT

Regression Strategy

·   Adopt and implement, clearly defined, business driven regression test strategy

·   Risk Based Testing is a must

Accountability

·   Insist development teams to publish data on defect injection ratio and Code quality

·   Insist on publishing Unit testing results report to validate your test coverage needs

·   Business Analyst/Dev/Test to take joint ownership on every production defect

·   UI test cases to be executed and tested by development teams

 

Conclusion:

Majority of cases testers write test cases based on experience and less on use case conditions (test case writing techniques). Hence in order to control the number of test case written and executed, an aggressive goal to increase test case effectiveness is essential. This will also push testers to understand application functionalities better, use optimization techniques and will increase effectiveness of test planning and execution. This can effectively reduce testing cost significantly. A metric which is hardly captured, can be your solution to optimize your test phase and an answer to reduce cost of Quality.