Regression testing is when a subset of previous tests that have been created are executed to ensure that the software is still working and hasn’t degraded or regressed. Normally regression tests would be executed after code, database, operating system, hardware, parameter, environment, interface and/or other changes.
Three types of software change are common: corrective, adaptive, and perfective. Corrective changes are when software code, designs or data structures are corrected. Adaptive changes are when requirements change and, to support this change, functionality or quality attributes are added, subtracted or updatede.g. add a new menu item and screens to meet the new functionality described in the requirements. Perfective changes occur when nothing is broken and the requirements are already satisfied e.g. when a required 4 second response time might be reduced to a 1 second response time as an improvement.
When software system changes are made, tests are deleted, created, and/or updated to accommodate any specific detailed changes. These tests are normal tests to check whether the software and functionality of the new changes work. Often when work is done, code, interfaces and databases closely related to this work could be negatively influenced.
Regression tests are run against the parts of the system most likely to have been unintentionally compromised. To prevent running all of the regression tests each time, which is expensive in terms of time, money and opportunity cost, an impact study of each change is recommended. This allows testing to focus on areas that are more likely to have been damaged and therefore to a smaller subset of regression tests. For major releases it is good practice to run through the full set of regression tests as sometimes problems are introduced in areas no one thought would be touched by any of the changes that had impact studies carried out on them.
Regression testing can be seen as reducing risk either around changes, with the closely associated areas more likely to have been impacted, or over whole systems for major releases. When regression testing misses any failures, each such failure must be thoroughly studied to either create new tests to prevent similar types of failure or bring older tests into the subset of selected regression tests. The main purpose of regression testing is to reduce the risk of defects going to the next stage of any project, and especially to stop defects going live and triggering operational system failures. There is a trade off between the number of regression tests run and the degree to which risks are mitigated. With more regression test coverage, greater confidence can be had in the post regression products, but more tests take longer to execute and cost more to run, especially if they are conducted manually.
Regression testing can occur on each of the different testing and should cover the functional and quality attribute aspects of these testing levels. E.g. if a component or unit has a fixed defect, the regression tests would be the existing component or unit tests or a subset of them which would be run again to ensure existing parts of the component still give the same pass results during test. Because these regression tests test code and functionality which should in principle already work, they are sometimes called confirmation tests. After code is repaired the component will require new or updated tests to test the specific changes.
Regression testing can become very repetitive because essentially you are repeat testing, a high number of times, software and functions which theoretically should still be working. When these tests are done manually effective coverage is normally not achieved because the number of tests is high and the time frame in which to do the testing is often short. Manual testers might also overlook key aspects of the system test outcomes as they struggle to focus with long test sessions, often repetitions of previous tests. In essence most regression tests should be considered for automation, which, if managed and executed well, will expand coverage, ultimately reduce costs after the initial build phase, and execute faster and more accurately, as well as create automatic test progress data for audit.
Regression tests, like many other tests, should be tightly controlled or else most of the testing effort will likely be wasted. For control, regression tests should be run on hardware of known configuration and of known software build version, with limited or no outside interference. If other stakeholders will be using the same environment and system it may still be possible to maintain control by logically isolating users.
Regression testing at the acceptance testing level might need to cater for contract, regulatory, user, operational, factory and site acceptance tests – although testers can assist at these levels, users, and other business stakeholders should conduct or at least hold accountability for such tests. System level regression tests would check important system functionality and transactions. Architectural tests, component integration and component regression tests would be under the accountability of system architects, development leads and individual developers respectively.
Senior management would benefit from seeing the coverage and effectiveness of regression test suites at all testing levels.
Regression testing is of key importance on Agile development projects, where component tests and the other levels of regression test often form a key part of documenting the system and allowing for rapid adaptive changes without undue risk exposure. Regression tests on Agile as well as other software development life cycle models provide a safety net of confidence and assurance to testers, the business and all stakeholders – if all the regression tests still pass after changes to the system, then we can go live with minimum risk. If any tests fail, then we know not to promote the release until these failures are corrected or at least factored into our planning.
Regression testing is probably one of information technology’s best kept secrets. I can not think of a better way to get control of testing and product quality for existing code. Once automated correctly – I can not think of a more efficient way to get control of the quality of existing software products. What do I mean automated properly? Only tests of the highest quality must be automated.
Tests ought to be included in the regression test suite based on their ability to mitigate risk. The automation design maturity must be such that the level of maintenance to keep the automation scripts running must be balanced against the effort to implement the automation design in the first place. For most companies with large regression test volumes, this means that the design maturity should be in the range of data-driven, through action word (keyword), to hybrid models.
Wayne Mallinson
waynem@testdata.co.za
The reproduction, adaptation or broadcast, without permission, of any articles
or photographs published in this
publication, is forbidden and copyright
is expressly reserved for Test and Data Services (Pty) Ltd.
Opinions expressed by contributors are their own and do not necessarily express the opinions of the Publisher, Editor or members of the Editorial Panel.