Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« STC conference by QAI | Main | Overcoming challenges associated with SaaS Testing »

Test Case Effectiveness - What does it say?

CXO Guide Series - Topic 4

Test Case Effectiveness - What does it say?

While I was analyzing operational data of testing organizations for FY2011 (my client organizations), one data point caught my attention. On an average 50,000 test cases executed across application portfolios and less than 5,000 defects logged and fixed. To my surprise, the data was very similar across multiple organizations (Plus or Minus 15%).  Test case effectiveness is around 10% which means, your factor of safety is 10. We are writing and executive 90% of test cases which are non productive in a given test cycle. This is in spite of using all advance methods of testing like orthogonal arrays, model based testing, Risk based testing, etc. Couple of thoughts came to my mind

·         Are testing organizations doing something seriously wrong or do they really need this level of safety net?

·         Can Test Case Effectiveness metric provide more insights and provide a better explanation on why we need this degree of safety and in turn the resultant cost of quality?

·          Is the lower Test Case effectiveness one of the reason why the CXO community perceives testing organizations as an expensive cost center?

·         Do CXO Community and Testing Organizations really understand the degree of safety that is necessary to maintain the Quality of the application?

 

Well, if the answers to these questions were so straight forward, we would not have asked these questions at the end of the year with the help of a testing metric. But it is very important that every testing organization find its own answers which will be different from each other. Considering the cost of quality of every complex application landscape today, Test case effectiveness is the king of all metrics because it impacts both test planning and test execution efforts.

 

How can I improve test case effectiveness? I have tried to summarize few basic golden rules which are known to everyone but not always practiced

 

Parameter

Strategy

Metrics

Ensure you collect Test case effectiveness metric data after every release

Test Case Writing Techniques

·   Ensure appropriate test case writing techniques are used by testers (Orthogonal Array, etc)

·   Ensure Requirement Traceability Matrix is created and maintained on an ongoing basis which assists in prioritization of test cases and also track changes in requirement

·   Demand 'use cases' and possible exceptions in real life application usage from business users

Reviews

·   End of every release cycle, validate your test case effectiveness, make course corrections

·   Insist on business review of your test cases with an intent to improve TCE

Defect Patterns

Identify modules/functionality which needs attention based on past defect trends and which are stable

Duplication

Avoid duplication of test execution b/w Dev, Test and UAT

Regression Strategy

·   Adopt and implement, clearly defined, business driven regression test strategy

·   Risk Based Testing is a must

Accountability

·   Insist development teams to publish data on defect injection ratio and Code quality

·   Insist on publishing Unit testing results report to validate your test coverage needs

·   Business Analyst/Dev/Test to take joint ownership on every production defect

·   UI test cases to be executed and tested by development teams

 

Conclusion:

Majority of cases testers write test cases based on experience and less on use case conditions (test case writing techniques). Hence in order to control the number of test case written and executed, an aggressive goal to increase test case effectiveness is essential. This will also push testers to understand application functionalities better, use optimization techniques and will increase effectiveness of test planning and execution. This can effectively reduce testing cost significantly. A metric which is hardly captured, can be your solution to optimize your test phase and an answer to reduce cost of Quality.

Comments

This is an excellent analysis to improve TCE and reduce COQ.

I have seen in couple of projects that client (including business team) needs higher degree of safety though we as testing partner propose the les no of test cases using RBT.

Vasu,
A good article. Agree with you that there should be focuse on reducing CoQ and this may be missed when dev and testing are handled by different folks.

My thoughts on some of your points.
1. The term Test Effectiveness should be used in correlation with the defects leaked into production and not base it absolutely on defects found while testing.

2. If the study points out that, the no defects are low in testing as well as in production, then may be the development team is doing a good job.

3. A good testing is one where the test cases are low and coverage max. Usually tendency should be to write many a test cases which skews the metric. So i think No of Test Cases, Test Coverage and defects leaked to production should be taken together and analyzed.

Thanks

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.