All Requirements are not Equal

Testers and the other project stakeholders can and should prioritise the requirements. Priorities for testing the requirements depend on:

• Risk of the requirement failing: This in-turn relates to the probability and the impact of failure of the requirement, and the knock-on effect of such a failure;
• Business benefit of the requirement: Requirements that could begin business value delivery by virtue of completion have higher priority;
• Cost of the testing: Higher testing costs may result in the cost-to-benefit ratio of the requirement rising, and so reduce the priority to have or test the requirement;
• Visibility: Requirements that will be highly visible to clients, although less severe than other hidden requirements, will often be prioritised for earlier testing so as not to negatively influence the confidence that the clients have in the system;
• Planned release dates: Some very important requirements might have later planned release dates and thus could justifiably have a low priority for the current release.

Creating Test Cases and Scenarios Early Improves the Requirements (and the Design, Code, System, and Business)

When test analysts design tests and test scenarios to check the requirements, they always (I mean always) discover weaknesses (faults) in the requirements. This early detection of faults serves as an early warning of design or business problems that are addressable before the designs, code, and system exist. That’s right! The team is now able to address problems in the design and code (and eventual system) before the design, code, or system exists! Prevention is better than cure... It is testing activities that give the team members this foresight, enabling them to avoid trouble, rather than rushing blindly into it.

Preparation of the Test Environments

Each level of testing, component, component integration, system, acceptance, system integration, maintenance, as well as specific test types, installation, configuration, back-up and recovery, security, performance, usability, and data migration requires a test environment for proper test execution and results documentation. Early testing and good test planning ensure that testing (and the suitability and availability of the test environments) will not delay the project more than is necessary when testing is on the critical path of the project. All other planned testing should occur in parallel with other project work.

Test Estimation and Budgeting

Each stakeholder needs to estimate how long the testing activities under their direction will take, as well as what equipment, training, software, and support will be required and how much this will cost. This activity assumes all stakeholders know exactly what testing tasks are necessary and how long each one normally would take. At least two methods of estimating test duration and effort are required for credible results. Three methods of estimation submitted for team review would be more reliable still.

Details of Software Testing Relating to Design

Reviews, prototyping, test planning, test estimation and budgeting, and preparation of environments all have applicability here as in requirements related testing activities. I will use this section to highlight specific testing activities that give design testing tasks a big advantage.

The Design Review

During the design stage of any project, be it an information technology project or not, conceptual and sometimes actual models are created and used to investigate the future systems and solutions. Aircraft, buildings, and motor vehicle prototypes, or different aspects of computer systems are reduced to models. For information technology systems, object models, state-transition models, entity-relationship models, logical architectures, data-flow models, interface models, and more are used.

Testing activities include role-playing the progress through a business transaction and often finding faults with existing conceptualised solutions. Team review of the design solution elements not only trains the team and gets team optimisation of the intended design; this testing activity also flags faults.

The tester in such meetings is able to contribute likely (and less likely but possible) ‘real life’ scenarios for the team to work through. The tester in such meetings is able to analyse the number and type of faults that are occurring. The tester in such a team is able to spot suspect areas in the designs and also identify team members whose work might need a little extra testing… The tester in such a team is less embarrassed to carry in a small checklist of things to ask and is well trained to know the value of using it.

Model-based Testing

Early testing of these models yields the speedy identification of design defects. Automation of some of these tests allows scarce and expensive human time to be devoted to creating and developing tests for new models. Leading software producers are testing design models by means of automated testing tools before the actual code for the designs exists! Prevention is better than cure! Once again testing activities are identifying problems (in thinking and models) before actual commitment to building the solution is made.

If we think about it, it is not surprising that this is being done. This level of early thinking is very welcome in IT and is more entrenched in other industries. As people, we often do test early - before committing ourselves wholeheartedly. We put our toe into the water before jumping in and - most of us - date before we enter a more serious relationship.

Wayne Mallinson

Details of Software Testing Relating to Requirements

Requirements Specification

The requirements on a project present a good opportunity for testing to prove itself. If the whole team receives the business details, and has time to evaluate and test each requirement specified, huge savings in both time and money will be realised.

Requirements are Difficult to get Right

Understanding, analysing, and documenting requirements are complex activities and suffer from, amongst others, the following problems:

• The ambiguity of language: Many words or phrases adopt new meanings from context. Persons from different backgrounds may (and often do) interpret a word in a document differently from the originator’s intended meaning.
• People make errors: To err is human! The originator of a requirement may have it wrong; or anyone passing the requirement along - say verbally or by creating a design to meet the (wrong) interpretation of it - may have it wrong.
• Time Pressure: Gathering requirements is typically done under severe time constraints, where insufficient time and attention are devoted to getting the requirements completely, clearly, or accurately stated
• Complexity: One requirement has to operate within a collection of other requirements and may often influence (even contradict) previously documented requirements. Requirements have to dovetail within the whole project and product context.

Early Requirements Testing

A team reviewing or examining requirements by careful reading or study would constitute a test activity that will usually yield good results. This is best done while keeping in mind rules that may be broken or by consulting checklists that address common faults. These reviews (attended by all team roles) are instrumental in improvements, in that the team members gain a better understanding of the project goals and details, and will often identify many early problems thus preventing poor quality work products that follow on from incorporating large volumes of requirements-related problems.
These reviews are recognised testing activities that have reported return on investment results of four to 37 times. Early, means reviewing the requirements soon after they are forwarded to the project for attention – not all requirements are put forward at the start of a project.

Testing Requirements Prototypes

Given that testing activities find so many faults in requirements, is there any other testing activity that can reduce these faults pre-emptively? It turns out that the building and testing of requirement prototypes can be hugely beneficial for creating better requirements with fewer embedded faults. This should not surprise us as when requirement prototypes are tested by users and other team members more time, thought, and consultation is going into the process than would be if this phase were skipped. This extra work and investigation results in better exploration and understanding of the need and thus a (measurably) better requirement specification.

Testing Produces Metrics

Testing activities provide the most useful measures of product quality and process productivity. Take rework for example… how many finance officers or project managers know that as much as 42 percent of a project’s salary costs are rework costs? That’s right! If rework can be reduced (and with some effort over years it can fall to around 5 percent) not only do cost and time-to-market reduce, but quality goes up too! Who says we can’t have it all? Rework includes anything you have to fix because of doing it incorrectly the first time.

Time and Cost Savings

Savings result from the fact that all project stakeholders mentioned above will usually detect many faults during such a testing exercise. Faults found before requirements go to design and coding will be relatively inexpensive to repair. The types of faults found in requirement statements are incorrect facts, contradictions, ambiguities, omissions, and unnecessary material. Another advantage of this static testing approach is that knowledge of the specific requirements checked is available to the diverse team members, who will often then think of other complimentary requirements or spot conflicts with other parts of the system of which they have special knowledge.

Different Review Goals

Static testing occurs during reviews of printed material and occurs on any project documentation. The reviews differ in goals and effectiveness at detecting faults. Some reviews are suitable for training team members or encouraging a common team view, others for member participation, and yet others effectively identify faults for repair. Reviews encourage critical thinking and often expose valuable, new ideas. Even if you review a topic you have written yourself, the very act of holding a red pen seems to encourage the discovery of new or better thinking and the highlighting of faults.

Early Communication Helps Test Planning – Reduces Project Risk

If testers get new requirements as they enter the project for processing, it helps them to start test planning or to adjust existing test planning timeously. New project and product risks enter with the new requirements and time, cost, and quality negatives are best mitigated by early identification and planning of the appropriate testing responses.

 

<< July/August 2006