IntroductionIntroductionLet’s check whether you’re paying attention.
Look at Fig. 1. Guess the output values at A and B.
Fig. 1: Black Box Testing
If you got A = 4, well done! You probably recognised a pattern whereby each output is multiplied by 2. Did you think B = 10? If so, you’re wrong! B was the value ‘-1’ when the component was tested!
You agreed to test without knowing the business requirements If you undertake testing without an understanding of the business requirements, you run the risk of making incorrect assumptions.
Many business requirements change patterns at key threshold values; e.g. test scores, 59 – fail, 60 – fail and 61 – pass.
Which programming language was used?
a) Visual Basic?
b) Delphi?
or
c) C?
There is no way to tell! This is the nature of black box testing; we don’t care which programming language was used, we only care what the mapping of inputs to outputs should be, often derived from the business specification. That’s why it is called black box testing; we can’t see into the box to examine the code.
This makes black box testing timeless. During black box testing, testers don’t care how the expected results are derived, they only check that results are ‘as expected’ to pass the test, or otherwise the test fails.
Never test without a good understanding of the exact expected result corresponding to any input. These expected results are best understood from the business requirements specification.
Fig. 2: Black Box Testing With Business Specification as the Test Basis
Fig. 2 shows a good way to predict outputs for the problem in Figure 1. Access the business specification and determine the expected result for each input by understanding the mapping of inputs to outputs from the detailed requirements or business rules described in the specification.
With white box testing one looks into the box to see the code and its specific structure comprising designs, statements, comments, decisions and paths.
Fig. 3 shows white box testing insights; a portion of code in one view, and component calling relationships in another view.
When the explanatory comments are read (the comments lines start with a ‘#’) the reason for the ‘-1’ answer for B becomes clear. The component has a 1 digit display for output. Any answer > / = to 2 digits gets the error message ‘-1’. 5 x 2 = 10, but because it’s 2 digits, the ‘-1’ error message is displayed!
Fig. 3: White Box Testing Showing Internal Code Statements and Component Call Tree
Does this make white box testing a better testing method? No! In white box testing, we refer to the business specification for the test basis in addition to code. Let’s read the business specification to understand why.
We find that the specification asks for a component to be designed that doubles the value of each input. It is to write the results to a 2 digit display! The code in the black and white box testing above was copied by the developer from another project which used a 1 digit display. The developer forgot to update the code to be relevant to a 2 digit display! Neither white nor black box testing could be successfully achieved without reference to the specification.
White box testing can never guarantee that all business requirements have been met. Even if each statement, decision and branch in the code has been executed by a test suite (100% coverage), this doesn’t guarantee correct. complete satisfaction of the requirements. However, if less than 100% coverage of the code is obtained, this indicates more testing could be done. Another test input can be devised to execute a higher percentage of code coverage.
E.g. ATM and Air Traffic Control (ATC) code. Both systems require high reliability. Perhaps both have 100% white box code coverage. Now swop these . The ATM machine would be useless at tracking aircraft, and the ATC code couldn’t give you R10 despite the high code coverage!
Black box and white box testing each have unique strengths and weaknesses.
You could black box test a simple program for months and never discover a hidden input that could result in fraud and loss of money on the one hand. On the other hand, 100% coverage of all the code elements of another program wouldn’t necessarily highlight missing requirements, or the wrong business outcomes for requirements which were present.
Fig. 4: White Box Testing With Business Specification as the Test Basis
The 2 techniques should complement each other. Developers and the more technically-inclined should do white box testing, users and system testers should do black box testing. Each should derive their test suites from requirements as a test basis. White box testing should supplement the test suite used to achieve the highest coverage of the code elements possible in given time, budget, and other project constraints.
To begin to see the power of white box testing, consider the simplified pseudo-code for an ATM cash dispensing function in Fig. 5.
Fig. 5: White Box Test Suite and Expected Results From Running the Test Suite
Only 6 out of the 7 numbered statements would have been executed. Test coverage would be 6 / 7 * 100 %.
The fraudulent statement no. 3 would not have been executed. The tool would have highlighted the statement which didn’t run, and upon investigation of the code that wasn’t run, the fraud would likely be exposed. Thousands of ‘guessed’ black box test inputs, if attempted at all, would have had a very small chance of predicting the exact value of A to perpetrate the fraud AND the system date would have had to be the 5th of the current month during the testing!
Wayne Mallinson
info@testfocus.co.za
The reproduction, adaptation or broadcast, without permission, of any articles
or photographs published in this
publication, is forbidden and copyright
is expressly reserved for Test and Data Services (Pty) Ltd.
Opinions expressed by contributors are their own and do not necessarily express the opinions of the Publisher, Editor or members of the Editorial Panel.