What constitutes Test coverage?
As a practitioner, we struggle when customers asks us about the test coverage and number of test cases written to test an application. The client's intention for this question is to check the confidence we have on the quality of testing achieved through test cases and its coverage.
Most times, we start looking at requirement coverage and start associating the level of confidence through traceability of requirements into test cases.
This is a decent measure to demonstrate the minimum testing required for an application, as we can find out if one or more requirements have not been mapped to any test case, and hence not been tested at all. While it guarantees all specified requirements to be tested for basic functionality, it does not guarantee depth and coverage inside a particular functionality. The approach leaves multiple questions unanswered like, how would you determine that the application is tested for all possible configuration permutations and combinations? How would you find out whether the application has been tested for end-to-end scenarios or covering all possible options? How would you find the amount of negative testing done?
While discussing this topic with colleagues of mine, one of them asked if there is a benchmark available to determine the number of test cases to be written for an application, based on number of lines of code or requirements. This got me thing. I knew that I was not aware of any such benchmarks, however, I started thinking on the possibility of relating it with estimation modeling and using this to possibly estimate or determine the number of test cases (and scenarios) one should expect based on requirements, type of application and phases of testing. This could be the closest way of determining the extent of testing and test case writing required. Of course, this can be done only if you have great confidence in your estimation modeling technique and the internal benchmarks used.
However, the confidence in quality of testing does not only relate to test coverage and number of test cases written. It also is determined by the quality and quantity of defects a testing team is able to catch. Any number of test cases can be written, however the only way to determine the quality of testing is to relate this to the number of defects raised. There are various defect prediction models available that can help you estimate or predict the number of defects for a particular phase in testing. There are also many techniques through which we can predict the number of defects. The most popular way however would be relating it with size of the code, function points or overall development effort. Of course, one also needs to determine the maturity of processes adopted in the development life cycle to expect or gauge the quality and quantity of defects flowing into testing phase.
Based on this, we can come to the conclusion that to ensure complete test coverage we need an effective estimation model couple with a solid defect predication technique. Using the typical estimation model we can determine the number of test cases required and hence the amount of test preparation required. Further, by using a defect prediction model, we can determine the extent of testing to be done. Together, these two bring confidence in our testing and also work as lead indicators to the applications quality.