The Most Powerful Lever in Testing

Consider the description of testing given by a Test Manager to a new appointee:

• Check that the running system adheres to the specified requirements describing it.


• Find as many failures in the software as you can.


• Check the performance, usability, security, capacity, availability, reliability and other non-functional aspects of the system.


• Plan the test steps and data that will be used to do the above, and start this planning early in the software development cycle.


• Enter inputs according to your prepared steps and check the results with your documented expected results.


• Test the system in an environment, which mimics the live environment, and duplicates interfaces, data, hardware type, software configuration and every other 'live environment' characteristic.


• Analyse the data and report on the results.


• Record all apparent failures and later check that these have been repaired.


• After any change, check that the software still functions correctly in what is known as 'regression testing'.

(Before reading on, can you add any other significant points to describe testing?)

Don't you wish that this advice had been given to you when you started as a tester? If such pointers had been given to you with detailed explanations of the above, how productive you would have been as a new tester. Wrong!

All of the above testing points describe only forty percent or less of what testing entails!

The above points do describe validation testing fairly comprehensively. If unit, integration, system and acceptance tests involving only validation testing are run, this tester and probably the testing department run by the said Test Manager would have missed the most powerful lever in testing!

Introduction to Formalised Software Inspections

Testing comprises verification and validation. Validation testing described above occurs after the product is created (or after a product increment is created if incremental delivery is used). Verification testing occurs while the product (or product increment) is being created and thus can powerfully influence the intrinsic quality of that which is being created.

Verification is often assented to, acknowledged but never really used. It is too subjective and intangible. We do check our documents and designs, but often these meetings turn into 'spot the spelling mistake' farces or endless debates about trivia, and generally we have very tight deadlines so it is more important for our people to be coding …

If this describes your IT team or department, then the good news is that you have up-to-now missed a lever which can speed you towards the following benefits:

• Productivity increase over all development staff of between thirty to one hundred percent.


• Ten to thirty percent reduction in timescales.


• Five to ten-fold reduction in validation test execution costs.


• Ten-fold reduction in software maintenance costs (corrective, adaptive and perfective maintenance).


• Dramatic improvement of work and product quality.


• Reduction in pre-validation defect rates (by as much as 80%) and consequent savings in rework costs.


• Accelerated learning and experience building of all development team members and of involved users.

These are just some of the benefits of formalised software inspections - the most powerful lever in testing! Software inspections were pioneered by M.E. Fagan of IBM in the early 1970's.

Central to the technique is the concept of statistical process control. Many of the key success factors of Inspections run contrary to the current thinking and practices of the majority of individuals of the software development community. For example, with Inspections, material is often read at a rate of one page per hour!

Fagan came out of an industrial background, and asked the question whether or not statistical process control techniques could be applied to the quality of cognitive entities such as code-listings, requirements specifications, or project plans. He was rewarded with a bonus of $50 000 in 1979 by his employers (IBM) for defining the process of Inspections for software development. They were happy!

Statistical process control can be easily understood if we consider a glass factory example, where glass tubing is manufactured for neon lights. The diameter of the tube has to be within a defined range. In our example between 20,1 mm and 19,9 mm. As the tube of glass comes through the machines, rollers measure the tube diameter. If the measurement goes out of the provided boundaries, the glass tube is pushed off the conveyor belt and shatters on the floor. Together with this noise, alarm bells start ringing, and the manufacturing process is in an emergency state, until the tubing is brought into the required diameter range again.

Compare this process with the graph in figure 1.

The good news is that scientifically measured quality rates and control charts can be used to check and improve specifications, designs, contracts, test plans, unit tests, code and other development, management and test products.

Over the next few months of Test Focus, we will be learning how this powerful verification method works, and how to use it to leverage huge advantages in our software delivery efforts.

Wayne Mallinson