Last year I assisted a client with acceptance testing. By the time I was called in they were expressing disappointment with the response speeds of the system that had been supplied to them. Unfortunately, their expectations of system performance behaviour had never been specified, but rather they had allowed themselves to assume certain acceptable levels without specifically stating them.

The company supplying the software, when approached about the poor response speeds, replied in an e-mail, "…you have what you have, if you had told us your expectations earlier, we might have been able to do something about them…". My next task involved encouraging the supplier to do the best they could to improve responses (contractually, there was no obligation on them, to meet what was assumed and never specified).

I also was able to improve the recipient's software acquisition strategy for next time, by ensuring that they would properly specify performance requirements, and set acceptable levels as acceptance test criteria for these requirements.

In another similar case, a customisation of an e-mail package performed poorly. The recipient of the customised software had expected the customisation to behave with similar response times to the standard e-mail functions, but once again, had not specifically specified this as a requirement. Eventually the slow response times caused the total rejection of the new customised software after much wasted time, energy, debate, resources and stress.

Both of these instances could have been avoided by performance and specification testing.

How does one do performance testing?

In preparing for performance testing, one should ask the following questions: How fast? How many? How accurate, How much bandwidth?

Typically the answers to these questions would not have been accurately or completely specified (if specified at all!), and so testing would have raised the problem early in the software development life cycle where problems are cheaper to solve (they have not yet been designed and coded into the system), and time and budget are still available to the project team.

Performance testing is testing to evaluate the performance requirements or conditions on functional requirements for a software system or component. Example performance conditions on functional requirements include, speed, accuracy and resources used. Resource usage would include memory, telecommunications bandwidth, and off-line storage space.

When measuring performance, the specific operational profile must be known. This profile must describe the operating environment, the period of operation, the expected transaction mix and loads, and the expected behaviour of the system. The operational profile is the live system usage patterns and environment, it is often not exactly known, and must be approximated by a testing profile or profiles, when not testing in the live environment.

Performance test cases

Performance test cases should be designed with the aim of discovering current performance (for information and possible later benchmarking) or for exercising and evaluating specific stated performance requirements of the system. Each test case dealing with a performance requirement should specify system inputs and expected system outputs, whilst the system is under specific load conditions (test profile, environment, and duration).

The following additional factors should be considered when designing performance test cases:

Capacity: The maximum amounts that can be contained, produced, or fully occupy an entity.

Stress: Move a system or component beyond the limits of its specified performance requirements.

Response time: The overall time from when a request is initiated until a response is detected.
Bandwidth: The fixed amount of data that can be passed through a physical entity.

Reliability: Consistently produce the required (expected) results for a given function or combination of functions.

Efficiency: The volume of data processed divided by the resources used for this processing.

Scalability: The ability of the system or component to handle increased loads by adding additional hardware components, without the addition of extra software.

Spike and recovery: To exercise a system or components at high loads, and then to determine how soon the system returns to normal processing after the load is removed or reduced.

Security: The component or system maintains user confidentiality, data integrity, and authorised permissions.

Prototyping or modeling

Low fidelity prototyping involves mental, paper, or spreadsheet calculations. Many performance bottlenecks and problems are possible to discover from theoretical considerations. Entering proposed message lengths and frequencies in a spreadsheet, may quickly (and often inexpensively) indicate that bandwidth or timing requirements are going to be violated.

High fidelity prototyping involves simple coded procedures to for example send a million randomised messages, or query a database one hundred thousand times.

Manual performance testing

Using a number of users with stopwatches can be an effective way to test performance. I have heard of, and experienced numerous examples, where a system will crash (reliability compromised) where a mere five or ten persons have submitted near-simultaneous transactions. For manual and automated timing, it is important to measure the same transaction performance numerous times during the test. This is to establish with reasonable confidence, the variance on the response times, as well as maxima, minima, and average values.

Automated performance testing

A variety of approaches may be used here. One most similar to manual testing involves using test running software to both simulate the interaction of users, and to (with some additional scripting) automatically record response times and other system parameters such as memory usage, disk accesses and message queue lengths.

Batch files, utility programs, protocol analysers, message replicators, cable splitters, and even live feeds can alternatively supply the necessary data loads.

Load tools specialise in replicating network traffic whilst simultaneously allowing testers to build testing profiles, and monitor application responses and system parameters. These tools are technically complex, and require a great deal of expertise for optimal usage.

General points to remember

Transaction sequences and variety are important factors. If a single transaction type is merely repeated with high frequency, then caching (web pages, database queries, memory) may give fast response results that are not repeated in the live environment, which has a larger variety of transactions and transaction sequences, rendering caching less prominent.

Equally important is the test environment's similarity to the live environment. It is no good testing web responses using an ISDN modem, when the typical live user will be using a standard modem. It is nowadays becoming a common practice for some testing to be done within the live environment, to account for some of the inevitable differences that will be experienced between test and live systems.

Establishing a performance baseline

Having an agreed and documented current performance record is very useful for regression testing of performance requirements.

Each time software or system conditions are modified the performance baseline can be used as a measure of expectation and comparison. This allows for rapid and early detection of performance problems prior to the updated system or component going live.

Conclusion

Performance testing is an often overlooked, yet extremely important testing and acceptance consideration. It, like most other testing, requires knowledge, planning, preparation, execution, analysis, and reporting. Good luck as you introduce these measures to your project and company.

Wayne Mallinson

<< January 2002

March 2002 >>