Test automation can fail for many reasons. Today we will look at a recurring cause of these failures, a bad choice of tool.
Choosing a tool adapted to our context is essential for successful test automation
To automate or not to automate your tests, when you work in Agile, is rarely a question but rather a necessity. In order to answer this need, many tools exist... Each one with its strong points, its weak points but also the skills and environments necessary for its use !
The most common risks with an unsuitable automation testing tool choice are:
- The impossibility of testing all or part of the application. The tools cannot necessarily test all the technologies. Also some objects may not be recognized by some tools. Test results become unreliable or regression testing is not handle properly.
- Necessary skills not adapted to the team's skills. This aspect is sometimes neglected when choosing a tool, even though it is essential. Do the people who are going to use the tool have the necessary technical knowledge to use it correctly? Do they know the automation framework?
- The tool's profitability is not there. A tool can be able to test our entire application based on test cases but with such a high cost of implementation and maintenance that the automation is not financially interesting. This problem can come from the architecture of the automaton but also from the fact that the tool was too "twisted" to meet our needs. Keep in mind that the cost of a tool is not only based on license cost!
Without prior study it is very likely that you will come across a "bad tool". By "bad tool" we do not mean a "bad tool" as such. No, a "bad tool" is a tool that is not appropriate to our context.
Tools such as Cypress or Selenium are designed to do Web testing and are recognized as "good tools". However, if you want to do heavy application testing, these tools are inefficient and therefore "bad tools". Similarly, if the people who write, execute and maintain the tests are not technical, Cypress and Selenium would also be classified as "bad tools".
In both cases (heavy application and not very technical team) Agilitest is a tool that can meet the need, it is therefore, from this point of view, a better tool than Cypress and Selenium.
How to ensure the selection of a good tool?
The issues related to the selection of an automation tool are generally known and to make the right choice we must first take stock of our needs:
- What do I need to automate?
- What is the budget?
- What are the technical skills of the team? Who is able to help?
- Can it be integrated into our development and test environment?
- ...
Following this necessary step it is then possible to make a study of the tools answering our need.
Example of test automation tool research
I need a tool that can test different heavy and web applications. There are technical skills in the team but half of my testers have a rather functional profile.
Following my research I found 3 tools that could meet my needs: XXX, YYYY and ZZZ, a tool offering a high flexibility and working by keywords.
This study allows us to make a first selection by highlighting only the tools that are interesting for our context. Nevertheless, this study, although indispensable, is not sufficient. Indeed, even if, on paper, a tool seems perfect and better than all the others, nothing replaces a real-life situation and the implementation of a pilot project.
For this, it is preferable to select the 2 or 3 tools that seem the most relevant and to start automating (for a few weeks) with them in order to see how they integrate and are used by the team. This experience makes it easier to choose the tool that best suits our needs among the tested tools (note: it is also possible to select none and test others).
Only a real situation and a real pilot can ensure that the selected tool really meets our needs.
Let's go back to the previous example:
XXX and YYY were selected.
The pilot project shows that:
- XXX is cheaper than YYY and ZZZ
- XXX and YYY can automate our tests on all our applications.
- XXX is quite complex and requires more training for the team than YYY.
- Less technical testers write tests faster and more efficiently with YYY, they can't use the ZZZ tool.
- YYY is more expensive, but in the end produces tests that require little maintenance, which reduces the overall cost of licenses + team time compared to the XXX solution.
- ZZZ is free but not well adapted to the project and difficult to master.
In the end the team chooses YYY for ergonomic reasons but also for cost reasons (the difference in license cost being compensated by less training and better productivity of functional profiles)
Conclusion
Choosing the right tool is essential for successful automation. First, you need to know what you need. Then, start a market study to conduct a pilot project. Test the tools that seem to correspond the most to your needs. But keep in mind that tools evolve, applications too. It can change the context. Running tests that are not totally dependent on the automation tool is a real plus in our modern world.
.png)
