Test Management by Metrics
Effectiveness 3

Where are you finding the most faults?

Apart from the fact that you are finding and logging faults, do you know at what stage of the Software Development Life Cycle (SDLC) you are finding them? Is it important to know when you are finding them? If you are asking questions like these then you are at the start of the road to making your testing effort more effective.

I will answer the second question first (typical tester… always doing things the wrong way around). As an organisation you should be interested in reducing the overall cost of producing quality software! The earlier the faults are found, the cheaper it is to fix them, and if they are found in specifications (prior to coding), then the more effective the cost reduction. What I am trying to say is that if you find the defects early, you will not pay for the same thing twice, as happens in those never ending "fix - re-test cycles". Ways to accomplish this will be through techniques such as formal reviews and better yet.. (document) INSPECTIONS. As these techniques are subject content for further articles, I will move on to metrics.

How do you know where you are finding defects within the SDLC? You need to be able to conduct an analysis on your defect tracking database, which implies that a field will have to be created indicating stage or phase of the SDLC, i.e. Business Requirement creation, Technical Requirement creation, Unit Testing, Integration Testing, System Testing, Acceptance Testing. Obtaining the metrics for the lower level tests will be a challenge, but is important to the process.

Now, how to make the numbers work? You need the statistics for defects from the start of the project up until one month post-production. Why one month? I use a month as a base line, and this is a reasonable period in which to detect serious defects that were missed by testing. What is important is that you use some period of production defects so that you can be realistic about missed defects. Choose a period post product release and then keep to that for all measurements. If you are not using post-production defects then your results will indicate that you are 100% efficient at finding defects in testing, and we all know that that is not quite true!

For example, if you found 3 defects in the business requirements inspection process, your first stage would be 100% effective (3/3) at that point in time, but if you then find another 2 defects in the inspection of technical requirements, then your first stage would only have been 60% effective (3/5). As you progress through the SDLC, your apparent effectiveness in the earlier stages will be reduced, but that is how the metric is supposed to work. This process is detailed in Table 1.

Table 1

How to interpret these numbers? It can be seen from the example that the most effective defect discovery phase is when the verification techniques (eg. Inspections or Formal Reviews) are used during the Detailed Requirements Specification creation. By the way, this and the previous phase are where you should be finding most of the defects, as these will be the most cost effective phases to repair defects!

The least effective is Acceptance Testing with only 2%, but you must bear in mind that this and production are the worst case scenarios for finding defects as they will cost the most to fix (change requirement, change design, code and test). If this is the case, do you want to change anything in the way defects are found in the preceding phases? Remember that a 9% defect detection effectiveness rate in production is rather high. This indicates to me that there is a need to improve testing in earlier phases, but this decision is difficult to make without further metrics as to when errors are injected during the SDLC (you cannot assume that defects are only introduced during requirements phases … we are human after all).

To find this information, you will need a further field in your defect tracking database, namely, Root Cause / Phase. This will enable you to determine which stages have the highest error injection ratios and enable you to take corrective action (prevention would be better, though, and should be your ultimate goal). Table 2 contains modified data.

Table 2

In this example I have assumed that once code is in production, errors cannot be injected (remember that this could happen due to configuration and implementation issues).

You must be honest and admit that testers can make errors in design and execution of tests, so they also inject errors into the process.

Now, looking at the example, and combining all the metrics, I would suggest that attention must be paid to verification techniques in the first two phases (Requirements and Detailed Requirements Specification) as the areas that need action taken to prevent errors. In addition to this I would also suggest verification of all testing phases (too many errors here), and special attention must be paid to Integration and System Testing phases as these are normally the most effective validation phases of any SDLC.

If you have any queries or would like to contact me in connection with your own metrics efforts, you can email me at info@testfocus.co.za.and mark it for my attention.

Peter Sage

<< July 2002

September 2002 >>