Historically software testing was a phase squeezed in between coding completion and software production release. We still see these older system development life cycle (SDLC) models even though software testing forms part of each phase of the modern SDLC. The testing phase in these historical SDLC models refers merely to the execution activity of the most important and final testing phases – system testing and user acceptance testing.
Software test automation, in most cases, refers to the automated execution of some of the tests created to test the system under development. Test automation, in general terms, refers to the automation of any testing activity in the SDLC. For the purposes of this article we will specifically refer to the automated execution of a manual test, more specifically known as functional software test automation.
A typical SDLC, as it is practiced in relatively mature software development organisations, has (at least) the following six steps:
Different models might have more phases, but all have these steps in common (the naming might be slightly different). Each of the phases can be broken down into various sub-activities.
It is not our intention to depict a particular development process (e.g. spiral, waterfall, agile, etc.). The idea is simply that these steps are contained in distinct phases – each with its own inputs, outputs, specialists, activities, analysis techniques, and notations – used in the development of any substantial, software-intensive project.
To understand functional test automation development it is important that we take a closer look at the SDLC phases.
The system development life cycle has the following phases:
Gathering of the system requirements takes place in this phase. A detailed study of the business needs of the organisation is included. Options for changing the business process may be considered.
Design focuses on high level design (such as the programs that are needed and how they will interact), low-level design (how the individual programs are going to work), interface design (what are the interfaces going to look like), and data design (what data will be required). During this phase the system's overall structure is defined.
In this phase the designs are translated into code.
The execution of the tests (created for testing the system) takes place in this phase. Normally programs are written as a series of individual modules, each subject to separate, detailed tests. The system is then tested as a whole; that is, the separate modules are brought together and tested as a complete system. The system is tested to ensure that interfaces between modules work (integration testing), that the system works on the intended platform and with the expected volume of data (volume testing), and that the system does what the user requires (acceptance and beta testing).
Inevitably the system will need maintenance. Software will definitely undergo change once it is delivered to the customer. There are many reasons for a potential change. Change could happen because of some unexpected input values into the system. Changes in the system could directly affect the software operation. The software should be developed to accommodate changes that could happen during the post implementation period.
Figure 1
For various models of the SDLC the phase naming conventions might differ; however, they should all incorporate the six system development steps and therefore would have phases similar to the phases discussed in this article. Figure 1 shows the phases within the system development life cycle (SDLC).
In the SDLC we have a phase specifically allocated to do system integration and testing. Is this where testing fits into the SDLC? The SDLC is historically a software development centric model. Since the testing task was primarily left up to the developers, the testing phase was incorporated. Over time the testing profession emerged: testing became more scientific and focused, and became a profession in its own right. The testing phase after the development phase stayed intact in all models; however, its focus changed slightly. This testing phase became the system testing phase, which can incorporate user acceptance testing. This is the phase where all the various test activities reach their intended goal – proving that the system is ready for the production environment. If this is the case, then where does testing – and all its activities – actually fit into the SDLC? To answer this question we need to understand the life cycle of a test. But what does this have to do with functional test automation?
To understand how and where functional software test automation fits into the SDLC we need to understand the test development life cycle (TDLC). The test development life cycle incorporates all the phases and activities of testing and defines the life cycle for the development of a test.
A test has a creation cycle of its own. Certain steps are associated with its creation. Regardless of the specific techniques or tools used for testing, there are six basic steps to testing anything. These steps are:
Testing is a parallel or synchronous activity to system development. Therefore for each system development activity a testing activity exists that supports the development effort and ensures that a verification-validation cycle takes place for each phase in the SDLC. The six steps to testing anything are included in the test development life cycle (TDLC) and are mapped to the different phases of this life cycle.
The different test development phases are as follows:
During the planning phase we define the test criteria. All activities for testing are defined in the test plan. A decision is made regarding which documentation and checklists will be used. The test strategy is determined and the infrastructure is prepared.
Test cases and test scripts are created.
Test cases are executed and static tests are done (it is said that 60-70% of the testing activities can take place without any code present).
During the completion phase the test object is evaluated. The open defects and defect trends are established. The risks of release are determined and advice is formulated regarding the risks of promoting the system release to the production environment.
Figure 2
The TDLC described here does not incorporate all the activities associated with the testing process. The focus is primarily on the activities that include steps to test creation. These activities are the precursors to functional software test automation and are therefore important to our discussion on where and how functional test automation fits into the SDLC.
This brings us to functional test automation, and its steps and activities. The automation of a test has various steps that lead to yet another life cycle.
Functional software test automation development is the automation of the test cases created to test the application/system. In layman’s terms this is the process of creating scripts to automatically execute certain tests created during the test development life cycle (TDLC).
For the development of the automation scripts the following process is followed:
What is required for the functional software test automation project? It might be that we want to automate the regression test packs. What is the scope of the automation?
Creating an automated test is generally more time-consuming (expensive) than running it once manually. It is for this reason that a careful decision should be made with regards to which tests would be beneficial to automate.
In this phase we need to establish which tests would be automated by understanding what tests have already been created for testing the system.
We do not start scripting as soon as we have established what the requirements are. Do we start coding as soon as we have a set of business requirements? No, we plan our system first.
Design the implementation with a focus on reducing the maintenance effort on the functional test automation scripts. To achieve this we need to encourage reuse of the script components as far as possible. We have to plan what utilities will have to be created. These utilities or functions will be reused in our test automation script creation. An example of such a utility script or function might be the error reporting or logging mechanism.
Proper planning will ensure that we create the most cost efficient automated tests in the shortest possible time. At the end of this phase we will know if we will implement a data driven or keyword approach or if we will use the standard capture-playback method.
After careful planning during the design phase we will have a set of reusable utility scripts, which can be used as building blocks to create the scripts that will comprise the automated regression tests. Scripting can now start…
After completing the utility scripts they need to be unit tested and then incorporated into the automated regression suite.
In this phase we integrate the test scripts and utility scripts into the regression suite and execute the test cases automatically. The integration phase also contains test automation development system testing. After successful testing we are now prepared and ready for the reuse of our set of automated tests.
The automated tests can now be run and re-run as required by our testing process.
During the life cycle of our system the system itself will change, requiring us to change our test cases. If the test cases change the functional test automation regression suite will have to be adapted to deal with the changing system. Maintenance will have to be done on our initial automated regression tests and/or utility scripts. The process repeats itself… (refer to Figure 3)
Figure 3
Doesn’t this look a lot like a development project?
If we take a closer look at the system development life cycle (SDLC) we can see the resemblance to the software test automation life cycle or is it the resemblance of the software test automation development life cycle to the system development life cycle?
Functional test automation implementation is actually a mini software development project.
Now that we understand the test automation development life cycle we have reached an important part of our discussion – how do all these life cycle models work together to achieve the greater task of software system development?
The software development life cycle contains various phases with sub phases and activities to accomplish the complex task of creating a software system. The test development life cycle (TDLC) and the test automation development life cycle (TADLC) are only two of the multiple processes in this complex task.
Only the TDLC and TADLC processes will be discussed in this article, together with where they fit into the SDLC. The TDLC has activities that are prerequisites to the start of the test automation development life cycle and it is therefore important to show the phases of the test development life cycle.
Figure 4 shows how the SDLC, TDLC and TADLC all work together to ensure test automation is implemented at the most appropriate stage during the creation of a software system.
Figure 4
During the first phase of the SDLC the development team and the test team start with their individual activities. The two teams work together with other stakeholders to define the requirements for the software system to be developed. The development team focus on defining the requirements to ensure they can design the system from these requirements during the design phase. The test team ensure that requirements are defined that can be tested. They are also responsible for the review of the requirements documentation produced during the requirements analysis phase. We could add more activities for the development and test teams during this phase; however, as this article’s end goal is to understand where functional test automation fits into the SDLC we will keep it simple and only include the most important activities.
The next phase is the design phase and requires that the system requirements are defined and accepted by all involved in the first phase of the SDLC. The development team start the design of the planned system. The test team use the requirements document to start with the design of the test cases that will eventually test the system and ensure that it meets the original requirements (signed off in the requirements analysis phase). The test team also review the design documentation produced during the design phase to ensure it is still in accordance to the signed off requirements documentation. No code has yet been developed. After receiving sign-off on the design documentation the project proceeds to the next phase: the development phase.
The first code emerges during the development phase. The developers start with coding activities and the test team start to document the test cases.
We do not believe test automation tool acquisition is a project that belongs in the general SDLC; to include it would be much like considering the development platform selection as part of the SDLC. This can be done, but these decisions are normally not made on a project to project basis. For the general SDLC process we are discussing we exclude such tasks on the assumption that such decisions have already been made.
The implication is that we have a development platform, development process, test process, and test automation tool infrastructure in place in the organisation.
In the development phase a few important test activities take place. The test case design document is made available to the test team so that they can start with test case creation. The test case design document also provides a good base to determine which tests will be automated. This information is sufficient to start with the design for functional test automation.
During the early stages of the development phase a fair number of changes are still being made to the user interface. These changes are normally object level changes. It is thus too soon to automate tests at this point; however, you can create manual tests with only a small risk that changes that will occur to the manual tests. The planning phase for the functional test automation implementation should start. Test automation engineers analyse the requirements and determine which tests would be beneficial to automate and which tests are better manually executed. The test automation engineers would be responsible for ensuring that the test can be successfully and cost effectively automated.
Automation engineers start with the functional test automation design after analysing the system requirements and specifying the test automation requirements. During the functional test automation design phase the test automation engineers design the test automation scripts in such a way that script code reuse is promoted and a design is constructed which would have the minimum amount of maintenance possible after implementation. If the test team spends most of its time during execution maintaining the test automation scripts the test automation would not be time and cost effective.
The library routines to be used as building blocks during test automation scripting are designed. Development should be reaching their unit test phase and the first code becomes available. The test automation engineers are now ready to enter the scripting phase. They will start by ensuring that the user interface can be ‘seen’ by the automation tool. Library routines that can be used by all the test engineers are created. Some of these routines will consist of data collection/result reporting and defect logging as well as other common functions.
At the end of the development phase the team are ready to make use of the utilities and common functions to automate the tests specified in the test automation planning phase. The system code has gone through unit testing and is therefore more stable than earlier in this stage of the life cycle. Since the appropriate planning for automation has already been completed expeditious script creation should be possible. Individual tests should be 90% automated after completing the development phase. At this stage we won’t be able to do end-to-end testing and functional test automation would still have some manual test components.
Early in the integration and testing phase the test team start with integration testing and then proceed to system testing. Some of the automated tests can be used during the integration testing stage as they become available (after being tested by the test automation engineers). The test automation engineers would integrate the test automation scripts to create end-to-end tests to be used during the system testing stage.
During system testing the functional test automation suite becomes available for automated testing. The test team use the automated tests during the integration phase and system testing phase to test the system. In the SDLC integration and testing phase, multiple test cycles are conducted. The test team finds defects, which are fixed by the developers and re-tested. The test automation would prove to be beneficial during these re-testing activities. Re-testing cycles are where automation test reuse becomes cost effective for the test team.
After successful system testing, user acceptance testing takes place and the system is promoted to the live environment. This is a major milestone (probably the most important milestone) for the project, but it is not the end.
The system enters the operation and maintenance phase. This phase continues until the system is taken out of commission. During this phase the initial four SDLC phases (on a smaller scale) are repeated multiple times. Functional test automation could prove to be invaluable, but would need to keep up with the changing system. As the system changes the test team change the tests and the test automation engineers adapt test automation to keep up with changing system functionality.
Reviewing the SDLC and its phases together with the test automation development life cycle stages showed us that the following six steps are present in both:
The sub-phases and activities might differ slightly; however, the major phases show us the close resemblance of software development to software test automation development.
We also looked at the testing phases and how they are precursors to some of the software test automation activities. This allowed us to get a clear understanding of where functional test automation should be positioned in the SDLC to have the maximum value. We need to start planning as early as possible, although some of the other testing activities need to be completed before functional test automation activities can start.
When test automation phases start time is normally of the essence and the project is at a critical stage. This increases the importance for proper planning and correct test process implementation to make maximum use of the available time to implement functional software test automation. Functional test automation should be completed and in place for the test team to use during integration and system testing.