|
Software Test
Automation Life Cycle
Today, software professionals are faced with the challenge of constructing
and building systems with fewer resources in an ever-shrinking timeframe.
Companies not only want to test software adequately, but also as quickly
and thoroughly as possible. To accomplish this goal, organisations are
turning to automated testing.
When IT
professionals decide to automate software testing, they may not know what
the introduction of an automation tool to a project involves. The Automated
Testing Lifecycle Methodology (ATLM) provides guidance in this area.
By using
the systematic approach outlined within the ATLM, organisations are able
to arrange and execute test activities in a way that maximises test coverage
within the limits of testing resources. This structured test methodology
involves a multi-stage process that supports the detailed and inter-related
activities that are required to introduce and utilise an automated test
tool. The methodology supports the development of test design, the development
and execution of test cases and the development and management of test
data and the test environment. It also supports documentation and tracking
allowing the test team to obtain closure on issue/trouble reports.
The ATLM is comprised of six primary processes or components. Each primary
process is further composed of subordinate processes, as described below.
1. The Decision to Automate Test
The Decision to Automate Test represents the first phase of the ATLM.
This phase covers the entire process that goes into the automated testing
decision. During this phase it is important for the test team to manage
automated testing expectations and to outline the potential benefits of
automated testing when implemented correctly. A test tool proposal needs
to be outlined, which will be helpful in acquiring management support.
Overcoming False Expectations of Automated
Testing
While it has been proven successfully that automated testing is valuable
and can produce a return on investment, there isn't always an immediate
payback. It is important that some of the misconceptions that persist
in the software industry are addressed and that the automated testing
"utopia" is managed.
Benefits of Automated Testing
The test engineer must evaluate whether potential benefits fit the required
improvement criteria and whether the pursuit of automated testing on the
project is still logical, given the organisational needs. There are three
significant automated test benefits (in combination with manual testing).
These include producing a reliable system, improving the quality of the
test effort, and reducing test effort (minimising the schedule).
Acquiring management support
The first step towards a decision to automate a testing project requires
that the test team adjust management's understanding of the appropriate
application of automated testing to specific needs. The test team, for
example, needs to check early on whether management is cost adverse and
would be unwilling to accept the estimated cost of automated test tools
for a particular effort. If so, test personnel need to convince management
of the potential return on investment by conducting a cost-benefit analysis.
If management is willing to invest in an automated
test tool, but are unable or unwilling to staff a test team with individuals
who have the proper software skill level, or provide for adequate test
tool training, the test team will need to point out the risks involved
and may need to reconsider a recommendation to automate test.
Management also needs to be made aware of the additional
cost involved when introducing a new tool, not only for the tool purchase,
but for the initial schedule/cost increase, additional training costs
and for enhancing an existing testing process or implementing a new testing
process.
2. Test Tool Acquisition
Test Tool Acquisition represents the second phase of the ATLM. This phase
guides the test engineer through the entire test tool evaluation and selection
process, starting with confirmation of management support. Since a tool
should support most of the organisation's testing requirements, whenever
feasible, the test engineer will need to review the system's engineering
environment and other organisation needs to determine a list of tool evaluation
criteria. A review of the different types of tools available to support
aspects of the entire testing life cycle is provided in the book Automated
Software Testing, (written by Elfriede Dustin et al and published by Addison
Wesley) enabling the reader to make an informed decision regarding the
types of tests that could be performed on a particular project. The test
engineer then needs to define an evaluation domain to pilot the test tool.
Finally, after all these steps have been completed, the test engineer
can ask the vendor to bring in the selected tool(s). Test personnel then
evaluate the tool, based on sample criteria provided.
3. Automated Testing Introduction Phase
The Automated Testing Introduction Process represents the third phase
of the ATLM. This phase outlines the steps necessary to successfully introduce
automated testing to a new project. These steps are summarised below.
Test Process Analysis
Test Process Analysis ensures that an overall test process and strategy
are in place and modified, if necessary, to allow automated testing to
be successfully introduced. The test engineers define and collect test
process metrics to allow for process improvement. Test goals, objectives
and strategies need to be defined and test processes need to be documented
and communicated to the test team. In this phase, the types of testing
applicable to the technical environment will be outlined and tests identified
that can be supported by automated tools.
During the test process analysis, techniques are
defined. Best practices, such as conducting performance testing during
the unit-testing phase, are laid out. Plans for user involvement are assessed,
and test team personnel skills are analysed against test requirements
and planned test activities. Early test team participation is emphasised.
This helps refine requirement specifications in terms that can be adequately
tested and supports the test team's understanding of application requirements
and design.
Test Tool Consideration
The Test Tool Consideration phase includes steps that investigate whether
incorporating automated test tools acquired by the company with no particular
project in mind may benefit a particular project, or whether other tools
need to be purchased. This is viewed in the context of the project testing
requirements, the available test environment and personnel resources,
the user environment, the platform, and the product features of the application
under test. The schedule is reviewed to ensure sufficient time for test
tool set-up and the development of a requirements hierarchy. Potential
test tools and utilities are mapped to test requirements. Test tool compatibility
with the application and environment is verified, and workaround solutions
are investigated to incompatibility problems surfacing during compatibility
tests.
4. Test Planning, Design and Development
Test Planning, Design and Development is the fourth phase of the ATLM.
These subjects are summarised below.
Test Planning
The Test Planning phase represents the need to review long lead-time test
planning activities. During this phase, the test team identifies test
procedure creation standards and guidelines. The team also identifies
the hardware, software and network types required to support the test
environment. The team considers test data requirements, a preliminary
test schedule, performance measure requirements, a procedure to control
test configuration and environment, the defect tracking procedure, and
associated tracking tool.
Test Design
The Test Design component determines the number of tests that will be
performed, the ways in which those tests will be approached (for example
paths and functions), and the test conditions that need to be exercised.
Test design standards need to be established and followed.
|
Software
Test Automation Life Cycle cont.
The exercise of developing the test procedure definition
not only aids in test development, but also helps to quantify or bound
the test effort. The development of the test procedure definition involves
identifying the suite of test procedures that will need to be developed
and executed in support of the test effort. The design exercise involves
the organisation of test procedures into logical groups and the definition
of a naming convention for the suite of test procedures.
Test Development
In order for automated tests to be reusable, repeatable and maintainable,
test development standards need to be defined and followed.
After performing test analysis and design, the
test team is ready to perform test development. Keep in mind that the
test design and development activities follow an iterative and incremental
approach, in order to address the highest risk functionality up front.
Table 1 correlates the development process phases to the test process
phases. The testing processes and steps outlined in the table are strategically
aligned with the development process. The execution of these steps results
in the refinement of test procedures at the same time as developers create
the software modules. Automated and/or manual test procedures are developed
during the integration test phase with the intention of reusing them during
the system test phase.
Developing test procedures that are maintainable,
reusable, simple and robust can be, in itself, as challenging as the development
of the application under test. Test procedure development standards need
to be in place supporting the structured and consistent development of
automated tests. Test development standards can be based on the scripting
language standards of a particular test tool.
By developing test procedures based on development
guidelines, the test team creates the initial building blocks for an automation
infrastructure. The automation infrastructure will eventually contain
a library of common, reusable scripts. Throughout the test effort and
in future releases, the test engineer can make use of the automation infrastructure
in order to support the reuse of archived test procedures, minimise duplication,
and thus enhance the entire automation effort.
Test Development Architecture
Test team members responsible for test development need to be prepared
with the proper materials. Test team personnel need to follow a test development
architecture that includes, for example, a listing of the test procedures
assigned to them and a listing of the outcome of automated versus manual
test analysis. Also, test team personnel will need to decide when to automate.
At times a test team might want to avoid automating using a GUI testing
tool before the interface (whether API, character UI) or GUI is stabilised
to avoid re-engineering the automated tests in response to non-bug-related
changes. At other times the test team can find workaround solutions when
automating an unstable GUI, such as focusing automation on the known stable
parts only.
Technical Environment
Test procedure development needs to be preceded by several set-up activities.
Test development needs to be supported by a technical environment that
can facilitate it. As a result, the test environment needs to be set up
and ready to go. The test environment includes the technical environment,
which may include facility resources, as well as the hardware and software
necessary to support test development and execution. The test team needs
to ensure that there are enough workstations to support the entire team.
The various elements of the test environment need to be outlined within
the test plan, as discussed previously.
The test team will need to obtain and modify the
test databases necessary to exercise software applications, and develop
environment set-up scripts and test bed scripts. The test team should
perform product reviews and validate all test source materials. The location
of the test environment for each project or task should be defined within
its test plan. Early identification of the test site is critical to cost-effective
test environment planning and development.
5. Execution and Management of Tests
The test team has, at this stage, addressed test design and test development.
Test procedures are now ready to be executed in support of exercising
the application under test. Test environment set-up planning and implementation
are, at this point, consistent with the test requirements and guidelines
provided within the test plan.
With the test plan in hand and the test environment
now operational, it is time to execute the tests defined for the test
program. When executing test procedures, the test team will need to comply
with a test procedure execution schedule. The test procedure execution
schedule implements the strategy defined within the test plan. Plans for
unit, integration, system and user acceptance testing are executed. Together,
these testing phases make up the steps that are required to test the system
as a whole.
Test metrics provide the test manager with key
indicators of the test coverage, progress, and the quality of the test
effort. During white box testing the test engineer measures the depth
of testing, by collecting data relative to path coverage and test coverage.
During black box testing, metrics collection focuses on the breadth of
testing, including the amount of demonstrated functionality and the amount
of testing that has been performed.
6. Test Program Review and Assessment
Test Program review and assessment activities need to be conducted throughout
the testing life cycle, in order to allow for continuous improvement.
Throughout the testing life cycle and following test execution, metrics
need to be evaluated, and final review and assessment activities need
to be conducted so that processes can be improved.
Following test execution, the test team needs to review the performance
of the test program in order to determine where improvements can be implemented
for improved test program performance on the next project. This test program
review represents the final phase of the Automated Test Lifecycle Methodology
(ATLM).
Throughout the test program, the test team collects various test metrics.
The focus of the test program review includes an assessment of whether
the application satisfies acceptance criteria and is ready to go into
production. The review also includes an evaluation of earned value progress
measurements and other metrics collected.
As part of its culture, the test team needs to
adopt an ongoing iterative process of acknowledging the lessons they have
learned. Such a program encourages test engineers to take the responsibility
of raising corrective action proposals immediately, when such actions
potentially have significant impact on test program performance.
Summary
ATLM is a structured methodology that is geared toward ensuring the successful
implementation of automated testing. The ATLM approach mirrors the benefits
of modern rapid application development efforts, where such efforts engage
the user early in the development cycle. The end-user of the software
product is actively involved throughout the analysis, design, development,
and testing of each software build, which is augmented in an incremental
fashion.
In a way that is similar to software application
development, test requirements are specified before test design is constructed.
Likewise, the test program must be mapped out and consciously designed
to ensure that the test activities performed represent the most efficient
and effective tests for the application under test. A test design that
graphically portrays the test effort is developed, in order to give project
and test personnel a mental framework of the boundary and scope of the
test program.
If you have
any queries or would like to contact me in connection with your own automation
efforts, you can email me at info@testfocus.co.za.and
mark it for my attention.
Henk Coetzee
|