Structured Unstructured Testing?

For a long time, I have been a firm supporter of structured testing. I have advised people to follow this approach. I have written articles about it, and have in fact, been very passionate about structured testing. Yet lately, I have been asking myself "what is structured testing?" I have been a bit surprised at the answers I am getting back … did somebody say that the first sign of insanity is not when you talk to yourself, but when you answer yourself?

I have gone through a process of redefining the way I see structured testing. It may be simpler if I first explain my view of unstructured testing; give my old definition of structured testing, and then explain where I am going with my new interpretation.

Unstructured testing is a chaotic state in which you cannot predict testing effort, repeat tests or measure results. This is also known as ad hoc testing, a process followed by many project teams. This approach does not use quality criteria to determine risk and prioritisation of testing activities. Testing techniques are not used to build good test cases.

I view structured testing as a set of processes or activities based on one of the international standards (IEEE, ISO) or popular testing approaches (TMap, CMMI, TMM etc). The purpose of these processes or standards is to ensure that maximum test coverage is obtained; that if two different testers had to write test cases based on the same specification there is a strong chance that they would be able to produce very similar test cases. These models are based on areas such as requirements coverage, risk and error models.

The structured and unstructured test approaches described above are vastly different and yet have something in common. I have found neither to be satisfactory. Unstructured testing is chaotic and ineffective, and I will leave it at that. But, what about structured testing? Why am I finding it unsatisfactory? I would like to address this by looking at some issues in the following paragraphs.

Requirements: Most models are founded on having good requirements documents. On projects that I consider to be successful, good requirements have certainly been central to achieving success. Yet I have worked on few projects that have had good requirements must allow these to fall into disrepair (I have even had the scenario where, being handed the specification documents for the project, the project manager informed me not to bother reading the documents as the product no longer bore any similarity to the specification!).

The reality is that requirements will often not contain enough detail to support the testing effort. There are signs that some organisations or companies are making efforts to correct this situation, but the change is slow. We will simply have to find a means to conduct good testing without good requirements.

Prioritisation of Risk: Most testing conducted in structured approaches recommends that requirements are prioritised on a risk or quality assessment basis. This is something that I strongly support. However, few clients, users or stakeholders are involved in determining priorities. I see many organisations design high level test strategies that propose that projects base testing efforts on risk prioritisation. Very few projects actually do this mainly due to a lack of understanding of testing by management personnel. Obviously, prioritisation of risk can only occur if there are requirements in the first place.

Quality Characteristics: Structured testing models recommend that quality characteristics are used to drive the testing approach, which is a very good idea. Once again this is part of organisation-wide test strategies, but is seldom visible in projects.

What all of this boils down to is that structured testing is good, but reality bites; and due to lack of requirements, or to project constraints (time, lack of commitment, lack of understanding of testing or poor management), effective testing is not conducted. Where does this leave us? Lately I have been investigating what is known as "Exploratory Testing" or "Context Driven Testing". I think that this is very possibly the solution that will bridge the gap between "Structured Testing" and "Unstructured Testing". There is some very interesting material available on this subject that has been produced by James Bach, Cem Kaner, Bret Pettichord and James Whittaker in various writings: Some of the sites where you will find these articles, are www.testingcraft.com; www.stickyminds.com; www.howtobreaksoftware.com; and www.satisfice.com.

Exploratory testing is viewed by some as unstructured because it is not based on requirements. This is correct. But what can you do when there are no requirements? Exploratory testing is the discipline of designing and executing tests simultaneously. Does this mean that there is no plan? No. The tester must first understand how stakeholders in the product measure quality (quality characteristics) and use this for the basis of the test design.

While testers conduct the tests, they will take notes that will enable an audit trail, which can be reviewed to guide further testing. How does this work? Once the quality concerns are understood, testers will approach the product from two fronts.

On the first front, they will try to gain an understanding as to what the product does, by taking note of what is presented through the application. On the second front, they will then compare this to their own understanding of basic business flows, and system experience. Next, based on this assessment, they will design and execute tests that will best be able to break the software for each given scenario.

These actions will be noted, and then used to design better tests, based on the results of the previous tests. Of course, the success of this approach is largely dependent on the experience of the tester. But then, if your project is in need of testing, and you have not had the time to write specifications, you will need to use experienced testers.

Another approach that I have found refreshing is based on the book, How to Break Software, Addison-Wesley, 2003, and other writings, of James A. Whittaker. This approach uses an error-model to drive testing. Basically, software accepts input, produces output, stores data and computes. If testers know which are the most common areas of error in these processes, the testers will be able to design tests that focus on the errors that will most commonly occur. The author uses the term "attack", which I quite like. It expresses the attitude of the tester towards the application under test. Testing must have the best chance of finding faults, and subsequently, proving that the software does not work.

My favourite book (Testing Computer Software, Cem Kaner, Jack Falk, Hung Quoc Nguyen, Wiley, 1999) taught me that before you test you must THINK. I constantly remind developers to think before they pound keyboards, but this is equally true to myself as a tester: think before you write elaborate test cases. This is one of the most valuable lessons that I have learnt. Not every project is the same, but there are things that go wrong over and over again. I have used these lessons to guide me in projects that were in "trouble", and there was no time to follow the process. But as I think back, these techniques are little different from structured tests that I have conducted.
I use a process that includes the following steps to conduct this type of testing:

Step 1: Quality Attributes

I first take the time to talk to the stakeholders regarding their concerns about the application, and then pay careful attention to the answers. In so doing I will actually be finding out what their perception of quality is. I also use these sessions to get a high level overview of what business or technical processes were meant to be automated by the application. I use this as a high level requirements guide.
If I then take these concerns (quality criteria), and use them to drive the types of tests that need to be run; I will have a strategy upon which I will design my tests.

Step 2: Prioritise Risk

I then list all of the quality attributes in a spreadsheet that will allow me to group high-level functions by quality attribute and main attack category (input, output, data storage, computation). With the stakeholders, I apply a simple risk level (High, Medium, Low) and give an indication of the likely testing effort (this would normally be based on the amount of time that has already been allocated, and then ensuring that the time is used effectively - i.e. the best testing is done in the time available).

Step 3: Design Tests and Execute Tests

As I work my way through the application, I will constantly try to map out the process flows through the application, and at the same time think of tests that will break the code. The way I do this is to bear in mind the four main categories of attack as recommended by James Whittaker. As ideas come to me, I note them down, and then apply those tests immediately. I try to figure out where else those that bear fruit (i.e. faults) would work in other areas of the application (where would the developer have made similar errors?).

Step 4: Review and Design

I have found it best to plan a design and execute test session to last about one hour. I will then take a short break, and review what I have accomplished in the previous period.

I have found that it is a good idea to discuss my findings with a peer, who may give me additional ideas on how to create better tests (sometimes when you are struggling with an idea, the mere act of discussing it will enable you to find an answer). I use this to check to what degree I have strayed from my original plan, and then, by understanding why I strayed off the path, update the strategy.

In general I have found this whole process to be not so different from structured testing. In fact I am prepared to include this in the term "structured testing".

Welcome to structured, unstructured testing!

Peter Sage

<< January / February 2003

May / June 2003 >>