Testing Cycles and Product Stability
Years of experience in software development have not helped reduce anxiety levels whenever a project enters the 'Testing' phase of the Software Development Life Cycle. It feels the same as one would feel when parents accompanied you to school to collect your academic results at the end of term examinations. There is always the anxiety of whether the output of the design and coding phase will be able to successfully sustain the test case bombardment. Besides, you would also be anxious to know if there are enough test cases to traverse all paths of the software while testing functionality and QoS parameters - so as to be confident that all loose ends are covered. An even more difficult pill to swallow is a situation where you realize that a number of your tests are failing, and you will have to get back to the customer with the bad news and request an extension. But even after that, how do you gurantee that your product will be defect free ? How do you gurantee that you have not introduced defects unknowingly while fixing the known ones ? Experts would ofcourse recommend a 'thorough peer code review' - but even after that, you would still need a 'tested and passed' certificate before the software is passed to the customer for his acceptance tests.
I have often felt the need to be able to estimate upfront the number of test cycles that should be executed during the 'Testing' phase. What projects often do is estimate for two cycles of test as an approximation. In Test-Cycle 1, all the test cases are run and all the defects in the sofware are assumed to have been detected. Fix them. Test-Cycle 2 is run to make sure that all the defects detected in Test-Cycle 1 are fixed. It is quite common that some defects are not completely fixed, especially when the number of defects detected is large. In addition, new defects may have also crept in as a by-product of the defect fixes. Naturally, this would call for another test cycle, once the Test-Cycle 2 defects are fixed ..and you end up squeezing in a Test-Cycle 3 with the hope that Test-Cycle 3 would be defect-free. (By-product defects are often seen in GUI intensive products where for example, regression defects are common while controlling state of change of UI elements.)
One can relate the above situation to what Bruce Powell Douglas mentioned during his July 2009 visit to the Bangalore campus. With regard to a different context Bruce had said that - "The number of defects in the software is proportional to the number of defects that you know about" which means testing and quality has to be a continuous process.
Every project in it’s estimation phase has to dwell deeper into the possible ways to counter deficiencies in development during the SDLC. One such problem is to estimate the number of defects that would need to be ironed out during the various phases of the project and the effort required to counter them so that the goal of a zero defect product is achieved before the delivery to the customer. The measure is a method of checking if the output of the particular phase has been reviewed and tested and that it will contribute to the final quality goal.
Most leading companies have metrics with regard to software quality management which are created as a result of observing trends in defect data across projects in similar technologies having similar complexities. One such metric would be for example, the number of defects per KLOC of code that are expected to detected during the complete life cycle (In today's world of function point based estimation, this might be a little primitive).
Let us assume that, as per the expected standards, the total number of defects per KLOC for both GUI and non-GUI applications (developed in C or C++) could range between 15-20 defects. Considering the pessimistic choice of 16 defects per KLOC, for a given application of an estimated size 25 KLOC, the total number of defects that could be expected in the complete life cycle of the project is (25 *16 ) 400 defects.
The total number of defects estimated to be detected during the SDLC is divided across various phases of the project. The confidence that justice has been done to the review of the product and that the product is measuring up to required quality can be obtained by measuring defects at the end of the phase. Assuming a general spread of defects detected to be 30%:40%:30% across the Requirements and Design, Coding and Testing phases (please check your company metrics for the spread if any), 30% of the estimated defects still exists when the software enters the Component/Integration Testing phase.
The Component Testing or the Integration Testing phase is high on the list of defect detection stages simply because this integrates a number of independently developed modules – and is more or less the environment which the user is expected to use the application. This is the last stage of validation and verification – before the application passes hands outside the development team.
Ideally, the CT testing should be divided into cycles of CT to make sure that defects creeping in because of defect fixes - are caught and cleaned.
It is quite common to see three cycles of CT testing being planned for certain products. The first cycle should aim at detecting 60-65% of the defects estimated in the CT stage. Assuming that your defect estimate calculations reveal that there are still 120 defects still lingering in the software as you begin your CT, you should be detecting 120 in the first cycle of CT testing and 68 defects in the second cycle of CT testing. The third cycle of CT testing should be a more confirmation cycle to confirm that there are no regression defects.
But, is this approach enough to gurantee good software ? Time pressed, you are assuming and hoping that you would have neatly fixed the 68-70 odd defects that are lingering in the software at the end of the second cycle of CT thus allowing CT-3 to be more of a confirmation cycle - which is quite a risk!
A better approach would probably be to consider the "divide by two" approach to determining the number of cycles of CT. In this approach, the number of defects estimated to be lurking in the software at the start of CT, may be recursively divided by 2 (until the defects are single digit) to determine the number of cycles.
Hence, in this case – 120, 60, 30, 15, 7 are the number of defects expected in each cycle with the sixth cycle being a confirmation cycle. However, in this approach care must be taken to change the test cases being executed – and also to focus more on problem areas (considering the 80-20 rule) so as to not make testing redundant and wasted by repeatedly testing defect free areas.
The "divide by two" principle might not be acceptable in all situations - and it is understandably difficult to convince about its need in a development process which contains strong design and code review phases. But, it is a good method to forsee upfront how much time and effort you would need during your testing phase to provide an efficient and defect free product.
P.S - External Testing is a phase where the development team completes their CT testing and provides the application to an external team within the unit for independent testing – which will surely comprise unbiased testing. This actually means that even on receiving an OK from the development team, there could still be defects lingering in the system –hidden from familiar eyes-which can only manifest in the eyes of an unrelated tester. This team should not be in anyway involved in the design/development of the product. The test cases developed by this team should be based completely on the FS/FD.
