Automation Testing

Automating testing is no different from a programmer using a coding language to write programs to automate any manual process. One of the problems with testing large systems is that it can go beyond the scope of small test teams. Because only a small number of testers are available the coverage and depth of testing provided are inadequate for the task at hand. Expanding the test team beyond a certain size also becomes problematic with increase in work over head. Feasible way to avoid this without introducing a loss of quality is through appropriate use of tools that can expand individual’s capacity enormously while maintaining the focus (depth) of testing upon the critical elements.

Consider the following factors that help determine the use of automated testing tools:

• Examine your current testing process and determine where it needs to be adjusted for using automated test tools.

• Be prepared to make changes in the current ways you perform testing.

• Involve people who will be using the tool to help design the automated testing process.

• Create a set of evaluation criteria for functions that you will want to consider when using the automated test tool. These criteria may include the following:

1.       Test repeatability

2.       Criticality/risk of applications

3.       Operational simplicity

4.       Ease of automation

5.       Level of documentation of the function (requirements, etc.)

• Examine your existing set of test cases and test scripts to see which ones are most applicable for test automation.

• Train people in basic test-planning skills.

Approaches to Automation

There are three broad options in Test Automation:

Full Manual	Partial Automation	Full Automation
Reliance on manual testing	Redundancy possible but requires duplication of effort	Reliance on automated testing
Responsive and flexible	Flexible	Relatively inflexible
Inconsistent	Consistent	Very Consistent
Low Implementation Cost	_	High Implementation Cost
High repetitive cost	Automates repetitive tasks and high return tasks	Economies of scale in repetition, regression etc
Required for automation	_	_
Low skill requirement	_	High skill requirements

 

Fully manual testing has the benefit of being relatively cheap and effective. But as quality of the product improves the additional cost for finding further bugs becomes more expensive. Large scale manual testing also implies large scale testing teams with the related costs of space, overhead and  infrastructure. Manual testing is also far more responsive and flexible than automated testing but is prone to tester error through fatigue.

Fully automated testing is very consistent and allows the repetitions of similar tests at very little marginal cost. The setup and purchase costs of such automation are very high however and maintenance can be equally expensive. Automation is also relatively inflexible and requires rework in order to adapt to changing requirements.

 

Partial Automation incorporates automation only where the most benefits can be achieved. The advantage is that it targets specifically the tasks for automation and thus achieves the most benefit from them. It also retains a large component of manual testing which maintains the test teams flexibility and offers redundancy by backing up automation with manual testing. The disadvantage is that it obviously does not provide as extensive benefits as either extreme solution.

 

Choosing the right tool

• Take time to define the tool requirements in terms of technology, process, applications, people skills, and organization.

• During tool evaluation, prioritize which test types are the most critical to your success and judge the candidate tools on those criteria.

• Understand the tools and their trade-offs. You may need to use a multi-tool solution to get higher levels of test-type coverage. For example, you will need to combine the capture/playback tool with a load-test tool to cover your performance test cases.

• Involve potential users in the definition of tool requirements and evaluation criteria.

• Build an evaluation scorecard to compare each tool's performance against a common set of criteria. Rank the criteria in terms of relative importance to the organization.

 

Top Ten Challenges of Software Test Automation

1. Buying the Wrong Tool

2. Inadequate Test Team Organization

3. Lack of Management Support

4. Incomplete Coverage of Test Types by the selected tool

5. Inadequate Tool Training

6. Difficulty using the tool

7. Lack of a Basic Test Process or Understanding of What to Test

8. Lack of Configuration Management Processes

9. Lack of Tool Compatibility and Interoperability

10. Lack of Tool Availability