The Heart of a Tester’s Job
Part 2
Requirements for Testing
Last month we learned how to identify and delineate requirements for testing in a specification. By now creating a spreadsheet of the requirements to hold additional information relevant to each requirement we develop a powerful testing planning, status tracking and management tool.
THE REQUIREMENTS SPREADSHEET
The easiest way to track the requirements and their progress, and to also ascertain the risk and effort of the testing project, is to put all the requirements in a spreadsheet with the following columns:
Column 1: Requirement number.
Column 2: Status. The relevant detailed statuses in Table 1 can be entered into the spreadsheet cells. Status group itself is not reported in the spreadsheet .
Column 3: Date - This is the
date on which the status changed.
Column 4: Business Function - This is used to categorise the requirements
into functional areas. For example, if this requirement has to do with
the login function, put 'Login' in this column. If it is a 'check balance'
function, put 'Balance Enquiry' etc. Later on this will assist in grouping
the requirements into appropriate test scenarios of related function.
Table 1
Column 5: Effort - This is a weighting factor. A weighting of 1 may be used for low effort tests (i.e. tests which can be carried out in the test environment without preparation), use a weighting of 3 if testing will require moderate effort, and 10 for the really complex testing situations which require much time and/or preparation. Actual hours of estimated effort may also be used, but ultimately, weighting factors are simpler to use and tend to be adequate.
Column 6: Risk - One can use actual monetary estimates of the potential impact of failure, or a weighting factor. Once again, use of a weighting factor to indicate risk is recommended in most cases. The content of this column needs to be agreed to with the Project Manager and also the User. This communication provides an ideal early opportunity for User involvement and management awareness of the likely testing needs, effort and risks.
Table 2: Project 5% Complete - Additional
Columns may be added at the testers discretion
Projection of staffing, scheduling and budgetary requirements: After the risk and effort columns are filled in, it is easy to see if the current staff complement for testing is sufficient. If there are budget and time restrictions, one can calculate the effort of the high-risk entities, and agree on the minimum testing required. The Project Manager and the Users need to agree on any reduction of the planned testing if this becomes necessary (This might already assist in freeing up some budget!). A huge advantage of this Requirements Management method, is that it can be used to predict testing work hours, schedules, and costs that should be reported to management at a very early stage of involvement.
Note: Function can be extended to
more than one column to include sub-functions or non-functional attributes
like security, usability, performance etc.
Estimate of hours: One can sum the effort over all the requirements,
and translate this into (a fairly accurate) estimate of the total number
hours that it will take to test.
Remember to multiply the effort by 3 in the estimate calculation, as each derived test suite will be run at least 3 times. The example progression below shows the normal sequence of events when running a test scenario (logically grouped requirements).
Figure 1: Succesive Runs of the Same Test
Reasons for the re-testing:
• Error masking - Some failures prevent other requirements from being available for testing. For example if a form does not display because of a failure, items on the form can not be tested until it is displayable (next test run)
• Imperfect fixes - According to statistics, for every six defect fixes, one fix (or 17% of fixes) will either not be adequate, or will introduce an unexpected error (often in a seemingly unrelated application area)
• Changing requirements - With changed requirements, more testing is required
Column 7: Test Procedure (or Test Pack) - This indicates the procedure in which the requirement is included. This is necessary and helpful for cross-referencing the requirements. All requirements need to be included in at least one test pack or procedure for full requirement test coverage. The requirements may be grouped into functions which coincide with release schedules, logical business areas, risk, effort or in other useful and creative ways, for example by those requirements which have failed during test.
Column 8: Issue Report Column - The issue report column is there to cross-reference the issue number, which was raised when a requirement failed during test. This column is also used to cross-reference any minutes where a decision might have been made not to test a requirement or to delete a requirement.
TESTING THE REQUIREMENTS
To make a big dent in the number of requirements to be tested, sort the requirements spreadsheet into low risk and low effort requirement categories. These requirements will include items like tab order, menus, and some error messages for example. They do not need prior preparation in order to be tested. Indicate these low risk and low effort requirements by means of an asterisk on the specification. The testing results are reported on a copy of the specification with its delineated and marked requirements. Note: Copy the specification for each new test run - remember, three or more test runs may be necessary. Now test these requirements.
Figure 2: Effort Versus Risk
Testing the low risk, low effort requirements
first, provides a good indication of the overall quality of application area.
It also provides a clue as to the development effort that will be required
to correct defects based on the number of defects found over the requirements
tested. If 20% are incorrect, it can be expected that the rest of the software
will reflect a similar percentage of defects. These marked specifications, together
with the test results (pass/fail) of the low effort, low risk requirements,
form part of the test report. This activity may be largely delegated to
juniors to test.
In Test Focus next month we will see how test progress planning and reporting is tremendously improved by making reference to the requirements management spreadsheet of table 2, at the beginning of and throughout a project.
Wayne Mallinson