Budgeting for a Testing Methodology

In the USA in 1999 the cry in software development departments was, "let's get a testing methodology". In 2000 the cry changed to "we have a testing methodology, now let's use it". I hope that in 2001 the message from these departments will be, "it works so well, why did we not introduce it earlier?"

Why have so many USA companies invested in testing methodologies, and then not bothered to use them? The answer may be that there was insufficient buy-in of top management or the testing personnel. Perhaps one small group documented a methodology, and others disagreed or did not see its value, or perhaps they were not pressured to apply it. Whatever the reasons, it is important to note that many purchased or created testing methodologies all-to-easily become expensive shelf ware.

I believe that certain key elements help to ensure the successful introduction and maintenance of a testing methodology.

Key 1: The environment into which the testing methodology is introduced must be supportive and this in itself has very real direct and indirect associated costs.

Senior management must have realistic expectations of the methodology introduction initiative and encourage it, even if it has a wobbly start. The management and staff affected by the initiative must be trained and convinced of the benefits that will accrue from a savvy implementation of a testing methodology. Associated training should ideally be Just In Time (JIT) so as to be relevant to current issues.

Configuration management of the test environment hardware, software, test and other project documentation is a necessary for successful ongoing use of a testing methodology.

Quality assurance reviews and audits should ensure continued use and improvement of the testing methodology. Testers and other staff should be encouraged to make suggestions and improvements to the methodology to ensure maximum buy in.

The quality of the software delivered to testing, can itself greatly influence the effectiveness of a testing methodology. Some studies have shown that validation testing effort can be reduced ten or twenty-fold if earlier verification testing efforts have been effective.

Key 2: Generally it is easier and most often wiser to introduce a methodology, purposefully, yet incrementally and at a pace that the staff can manage.

The introduction of a methodology typically will imply changing some of the existing ways of doing things. An initial drop in productivity and increase in costs may be experienced in the early implementation phases. Persist and measure the benefits of one small change after another. Try to express the benefits in rand value, to keep a record of past costs, and a growing list of victories.

Try to select methodology areas that will deliver the best results first. This will psychologically spur team members onwards, and will generate savings to subsidise the addition of new methodology components.

Key 3: Measure, and communicate the past and then the current states as a result of introducing each methodology component.

Attempt to define and review the business case for each methodology component prior to its introduction. Measure to understand the accuracy of the achieved figures against those predicted.

Key 4: Have a champion. Having one or more champions for a methodology can make all the difference to a successful, sustained methodology initiative, or a failed half-steam attempt at having a methodology.

Cost Factors:

•   Training of staff - typical local training rates are currently in the order of R1000 to R1600 per candidate per day. Required training for testers may be 3 to 5 days for a generalised testing course. Specialised training such as test automation, may easily add another 5 to 15 days per candidate. Developers, managers and other solution development staff will benefit from test training in the order of 1 to 3 days each. Note that this training is not merely to satisfy the spending of training budgets, but if properly planned and coordinated, can make a considerable contribution to the successful implementation of a testing methodology.

•   Supportive environment (hardware, tools, configuration management, measurement programs, quality assurance, management review, ongoing learning, communications, shared vision and understanding). These disciplines all have associated staffing, hardware, administrative and management costs. Test hardware should closely reflect that of the live system in type and configuration and therefore also in cost. A software configuration management tool may currently cost as much as one or two million rands, and often these and other tools are linked to foreign currencies.

•   Communication of the methodology will have creation, storage, maintenance and management costs. It may be decided to communicate the testing methodology via a company intranet on the one hand, or via reference documents on the other hand. These alternatives have different cost implications.

•   Staffing of functions related to the chosen methodology components must be costed. This costing should ideally be balanced against savings accrued as a result of improvements introduced by the methodology.

•   Consulting costs in setting up of a methodology. Ensure that your own staff takes ownership of the suggested processes; otherwise it may be an expensive exercise of no lasting value.

Tackle your budgeting for a testing methodology in bite-sized chunks, and you may easily find that you are budgeting for a saving! But don't take my word for it. Measure and make sure.

What about buying an off-the-shelf testing methodology? OK if you are absolutely sure that it will be off-the-shelf, and not become shelf-ware.

Wayne Mallinson