Reasons for the failure of test automation: an inappropriate choice of tool

Marc Hage Chahine

Test automation can fail for many reasons. Today we will look at a recurring cause of these failures, a bad choice of tool.

Choosing a tool adapted to our context is essential for successful automation

To automate or not to automate your tests, when you work in Agile, is rarely a question but rather a necessity. In order to answer this need, many tools exist... Each one with its strong points, its weak points but also the skills and environments necessary for its use !

The most common risks with an unsuitable tool choice are:

  • The impossibility of testing all or part of the application. The tools cannot necessarily test all the technologies. Also some objects may not be recognized by some tools.
  • Necessary skills not adapted to the team's skills. This aspect is sometimes neglected when choosing a tool, even though it is essential. Do the people who are going to use the tool have the necessary technical knowledge to use it correctly?
  • The tool's profitability is not there. A tool can be able to test our entire application but with such a high cost of implementation and maintenance that the automation is not financially interesting. This problem can come from the architecture of the automaton but also from the fact that the tool was too "twisted" to meet our needs. Beware, the cost of a tool is not only its license cost!

Without prior study it is very likely that you will come across a "bad tool". By "bad tool" we do not mean a "bad tool" as such. No, a "bad tool" is a tool that is not appropriate to our context.

Tools such as Cypress or Selenium are designed to do Web testing and are recognized as "good tools". However, if you want to do heavy application testing, these tools are inefficient and therefore "bad tools". Similarly, if the people who write, execute and maintain the tests are not technical, Cypress and Selenium would also be classified as "bad tools".

In both cases (heavy application and not very technical team) Agilitest is a tool that can meet the need, it is therefore, from this point of view, a better tool than Cypress and Selenium.

How to ensure the selection of a good tool?

The issues related to the selection of an automation tool are generally known and to make the right choice we must first take stock of our needs:

  • What do I need to automate?
  • What is my budget?
  • What are the technical skills of the team? Can I get some help ?
  • Can it be integrated into our development and test environment ?
  • ...

Following this necessary step it is then possible to make a study of the tools answering our need.

Example:

I need a tool that can test different heavy and web applications. There are technical skills in the team but half of my testers have a rather functional profile.

Following my research I found 3 tools that could meet my needs: XXX, YYYY and ZZZ, a tool offering a high flexibility and working by keywords.

This study allows us to make a first selection by highlighting only the tools that are interesting for our context. Nevertheless, this study, although indispensable, is not sufficient. Indeed, even if, on paper, a tool seems perfect and better than all the others, nothing replaces a real-life situation and the implementation of a pilot project.

For this, it is preferable to select the 2 or 3 tools that seem the most relevant and to start automating (for a few weeks) with them in order to see how they integrate and are used by the team. This experience makes it easier to choose the tool that best suits our needs among the tested tools (note: it is also possible to select none and test others).

Only a real situation and a real pilot can ensure that the selected tool really meets our needs.

Let's go back to the previous example:

XXX and YYY were selected.

The pilot project shows that:

  • XXX is cheaper than YYY and ZZZ
  • XXX and YYY can automate our tests on all our applications.
  • XXX is quite complex and requires more training for the team than YYY.
  • Less technical testers write tests faster and more efficiently with YYY, they can't use the ZZZ tool.
  • YYY is more expensive, but in the end produces tests that require little maintenance, which reduces the overall cost of licenses + team time compared to the XXX solution.
  • ZZZ is free but not well adapted to the project and difficult to master.

In the end the team chooses YYY for ergonomic reasons but also for cost reasons (the difference in license cost being compensated by less training and better productivity of functional profiles)

Conclusion:

Choosing the right tool is essential for successful automation. In order to make the right choice, you must first know what you need. Then you have to start a market study that will lead to a pilot project with the tool(s) that seem to correspond the most to your needs. Be careful: tools evolve, our application too, which changes the context. Having tests that are not totally dependent on the automation tool is a real plus in our modern world.

Photo Credit by Justin on Unsplash

Marc Hage Chahine

Agilitest Partner - Passionate about software testing at Altran - ISTQB Certified (Foundation, Agile, Test Manager)

LinkedIn

Get great content updates from our team to your inbox.

Join thousands of subscribers. GDPR and CCPA compliant.