Test Automation Tool Selection

Your company has realised that there are advantages to software test automation. The next logical step is to acquire a tool. As soon as you start looking into the market for Software Test Automation tools you will find a multitude of vendors offering various products that provide for every possible need.

The process of determining which tool will best be suited to your needs is not an easy one. This article is meant to provide some guidance to assist with this important first step into the world of software test automation.

Each organisation will have different needs that should be met by software test automation, which will be complicated by the various applications to be tested and operating systems used.

Whether the tool will be rolled out throughout the organisation, or only used within one department, a simple set of steps to evaluate vendor tools can be followed to determine the correct tool for the job at hand.

The steps to software test automation tool selection are as follows:

1. Determine your requirements

Before looking at any testing tools available in the market make a shortlist of the requirements you have for software testing. What problems will the tool solve? What technical capabilities will the tool need to be compatible with your environment? These are some of the important questions you will need to ask.

Use the following categories to guide the compilation of your requirements list:

Use the following categories to guide the compilation of your requirements list:

Compatibility issues

Any testing tool you choose will need to be compatible with:

• The operating systems your application supports
• The development environments used to create your application
• Third party software with which your application integrates

Tool audience

Who in your organisation will be using the tool? More powerful tools tend to be complex. If the skill level of the people that will use the tool on a daily basis does not con form to the complexity of the tool, a smooth implementation in your organisation would be unlikely.

Will your organisation allow for staff training. Can your organisation, within the implementation time, allow for the learning curve required to become comfortable with the tool?

Business requirements

Vendors need to conform to certain requirements before they can become suppliers to the organisation. Ensure that you are aware of these requirements and measure possible suppliers against these.

Testing requirements

What types of testing problems do you want the tool to address?

• Manual testing problems
• Time constraints when implementing small changes to the system
• Shorter regression testing timeframes
• Test data setup
• Defect tracking
• Increased test coverage
• Increased efficiency of the testing process

List your own testing requirements and specify in detail what problems you need solved.

2. Identify your constraints

The factors that will restrict your choice of tool need to be evaluated early in this process. These factors will allow you to exclude certain tools from your selection process at an early stage:

Environmental constraints

This category will include hardware and software constraints. Consider the operating systems the tool needs to work on as well as the hardware requirements for the tool. Additional hardware may have to be purchased such as disk more space to cope with the additional scripts and data.

Should the tool be co-resident with test applications:

Some tools can be run in a specific environment while exercising an application in a different environment or operating system. What is the future direction of your company in terms of operating systems and hardware?

This needs to be considered to ensure future use of the test automation software.

Supplier constraints

If you experience any problems with the testing tool you would want to have them sorted out quickly and competently. You might want to ask the following questions:

• Is the supplier a bona fide company?
• How mature is the company and their product?
• Do they have adequate technical support?
• How many other organisations have purchased the tool?
• What is the history of the tool?
• Political constraints

Do not underestimate the role of political factors. Although we like to think that the purchase decision is always based on rational technical factors, it is, in many cases, based on a balance of technical, emotional and sometimes irrational factors.

Quality constraints

Quality characteristics of the tool might include the following:

• Skill level required to use the tool
• Multiple user access
• Support and help documentation
• Integration with other tools
• Can it corrupt data
• Frequency of failure during realistic use
• Budget constraints

The amount of money allocated to the project will dictate what is available for the purchasing of a tool. The tool purchase cost may only be a fraction of the cost of fully implementing the tool. Keep in mind that the budget amount available will have to cover the purchase of the tool as well as licensing, training and implementation costs for the use of the tool.

The cost of the tool will also directly affect your Return On Investment (ROI) calculations in the proposed business case to management.

After compiling your list of requirements and constraints you will be in a position to evaluate products in the market.

3. Compile a possible list of candidates

You will need to do some research to determine which tools are available. Use the internet to scan for possible tools. Do a high level evaluation of the tools by matching the abilities of the tools provided in marketing material, to your own requirements and constraints.

Do a feature categorisation by listing each tool according to the following features it provides:

• Mandatory features: These are the features that are essential to accomplish your goal in meeting your requirements within the constraints.

• Desirable features: These are features that will distinguish the best tools from the others.

• Irrelevant features: Features that are not important and will not provide any real benefit to your situation.

Rate these features. This may be an iterative process. You should assess as many tools as possible and eliminate the potential candidates down to a list of no more than ten tools that will form part of the tool evaluation process.

The next step would be to contact the relevant vendors and speak to their sales people. Ask them for a demonstration and possibly an evaluation version that you can use during your selection process.

Be open with the vendors with regard to the requirements against which their tool will be measured.

4. Make your selection

Evaluate your shortlist against the following:

Feature comparison

Your feature list will allow you to compare the tools in terms of performance of the features advertised in marketing material against the requirements of your own system.

You may also want to contact reference sites and ascertain how they feel the tool's features performed in their environment.

In-house demonstrations

Ask vendors to perform an in-house demonstration on your system for the requirements described in step 1.

This will also provide you with the opportunity to gauge the tool vendor's technical capability.

Test of script maintenance

It is important to evaluate how easy it is to maintain the automation scripts on these tools. This is a very important factor that will only become clear during the second or third cycle of release testing after the purchase of the tool.

Competitive trial

Run all your tests on each of the tools on your short list using the same test
applications. This might require more work from your side, but will give you a very good idea of how each tool compares against the others.

At this stage you will have all the necessary information available to make a informed decision with regards the best tool to assist you with your software test automation effort.

Good luck

Henk Coetzee

<< November / December 2002

March / April 2003 >>