Shifting from Manual Testing to Test Automation
During the past five years there has been a noticeable increase in the use of test automation tools particularly in Graphic User Interface (GUI) testing (these tools are sometimes referred to as Record and Playback tools). Four stocks from the American NASDAQ stock exchange indicate the growth of these tools since 1996 (see the graph on page four).
This phenomenal growth is largely attributable to increases in technology and the pressure created on the testing effort by the improved productivity of development tools and languages. It is difficult to convince a project manager that it will take one month to (manually) test a new web site that took developers only two weeks to design and update.
To speed up testing, more and more test teams have turned to test automation tools. Unfortunately test automation tools are often perceived as "silver bullets" to solve all testing issues. The expectations of less time to test, fewer testing resources, and full test automation don't always materialise. Unless a correct implementation approach is followed at the appropriate time, the (expensive) test automation tools can become shelfware and result in disillusioned managements and teams.
In moving to test automation it is imperative that management's perceptions are accurate from the outset, which will then lead to realistic expectations.
- It will initially take longer to develop effective automation scripts than to do manual testing. Evaluating test automation tools in order to make the correct selection can be a significant exercise.
- Not all tests should be automated.
- Introducing test automation will not solve the absence of an effective testing methodology.
- Automated tests do not generally find more defects than manual tests in first-pass testing.
Each of these points will now be elaborated:
Automation initially takes longer than manual testing and thereafter can greatly reduce manual testing times. Firstly, time should be allowed to select the correct test automation tool (see article on "Choosing the Right Test Automation Tool" on page seven). This is a significant exercise, as the selected test automation tool will impact on the overall productivity of the automation effort. Secondly, it should be known that the most effective use of test automation tools requires that tool scripts be specified, designed, coded, and then tested and maintained following their own software development life cycle. It follows therefore that the first tests can be particularly slow, as automation test design features which will save overall automation time later, are created.
Effective scripts have been measured to take between two and ten times longer than the manual creation of equivalent tests. This time will impact on budgeting and scheduling and will only generally be regained in the second, third or subsequent required re-runs of any test. The return on investment can take from 3 to 36 months on a fully implemented GUI test automation system.
There is a learning curve when a tool is used for the first time, and test strategies may have to evolve. Data tables will have to be compiled for data driven testing. Every time the code under test changes the test scripts may have to be changed, and where test design has been weak, totally rewritten. Where a tester could easily cater for changes with manual tests, this is not necessarily true in automated tests. The test code will only execute as designed.
Record and Playback tools are relatively easy to use for simple data capture automation tests, but require to be programmed to make use of features such as comparisons, error handling and repetitions. In addition to this, tests must be robust and reliable, which will require programming skills. It must be realised that not all testers can program and not all programmers can test, so there will be training and staffing implications when there is a decision to use test automation.
Not all tests should be automated. A good test automation effort will generally only automate 40 to 80 percent of the total testing, the remainder being more suited to manual testing. Testers must realise that there will be times when it is quicker to run the test manually. In other cases it is not cost-effective to automate the tests (once-only tests). Aesthetic evaluation of applications, and tests where responses are graphically complex, are generally more amenable to manual testing methods.
Test automation tools should be used to give testers more time to design new and more effective tests and to run manual tests. An effective technique is to let the automation tool do regression testing of the application's basic functionalities and performance, thus allowing time for testers to create more effective tests for automation and to concentrate on areas suited to manual testing.
Test automation does not replace a solid test methodology. If you do not have a test organisation in place that follows sound testing principles and planning, then implementation of a test automation tool is likely to fail. Tests may run faster, and more tests may be run, but not enough defects will be found prior to release.
Good testing principles have been measured to reduce test effort by 83 percent in one project. It is known that a small number of properly selected test cases can be as effective or even more effective than thousands of randomly (or poorly) selected test cases. In general, most test paths should be manually run and refined, prior to being candidates for test automation. This is simply because the automation most-often simulates manual system interactions, only much faster. Therefore these manual interactions guide the automation effort.
Automated tools do not generally find more defects than manual tests. There is a myth that automated tests find many more defects than manual tests. This is not the case. Many defects are found while manually testing or while manually sequencing for a Record and Playback tool. If the defect is then fixed, the tool will not find the defect on even its first automated run. This is also one of the reasons why automation only proves its value in second or later releases of the code under test. Record and Playback tools lend themselves to regression testing and are good at finding defects resutling from (unexpected) side effects from changes to the code under test.
Click on image
to enlarge
In conclusion. Test automation tools, if used correctly, will lead to valuable test (script) assets in an organisation. Running of these scripts will reduce software failure risks and reduce overall testing costs over time. In addition test automation tools can often be highly effective in performing test result analysis and comparison. This valuable saving of repeated testing effort should not be overlooked when performing cost/benefit studies relating to the introduction of test automation tools. One further use for test automation tools is for data creation or conversion.
Peter Sage
info@testfocus.co.za
Reproduction & Copyright
The reproduction, adaptation or broadcast, without permission, of any articles or photographs published in this publication, is forbidden and copyright is expressly reserved for Test and Data Services (Pty) Ltd.
Opinions
Opinions expressed by contributors are their own and do not necessarily express the opinions of the Publisher, Editor or members of the Editorial Panel.