June 17, 2020

Introduction to test automation failures

Marc Hage Chahine
Blog > Test Automation
Introduction to test automation failures

In this article, we will look at the main reasons of test automation failures.

Automating test execution: a necessary step

The automation of test execution is a hot topic that has become almost unavoidable with the development of incremental methods. The reason is simple: with the democratization of incremental methods, and more generally of Agile methods, regression tests are becoming more and more numerous and executed more and more often. The time required to manually execute regression tests quickly becomes unsustainable for any team working on a product developed in this way.

The diagram below gives an idea of the number of test executions required for both Waterfall / Agility processes :

The multiplication of executions and the time taken by them push teams to automate their tests... and this without necessarily having experience in automation.

This lack of experience is the source of many failures for various reasons. There are for example:

Lack of maintenance

Designing, writing tests and running them once is not enough. The product grows and evolves during the different developments, which generate evolutions, changes, deletions and new features. A test that was valid for version N is not necessarily valid for version N+1, and it is even more likely that it will not be valid for version N+10. The regression tests must therefore be maintained in order to remain up to date.

Agilitest cannot (for the moment) maintain the tests for you. However, it offers to simply factorize your tests to limit their maintenance! The tool has been specifically developed to facilitate this operation: change an identifier, step by step replay and modification, reuse sub-scripts.

Lack of analysis of tests after their execution

Running automated tests is often considered free. Many teams take advantage of this to test as often as possible, which is a very good practice. However, even if the execution is free, the analysis of failed cases is not. The multiplication of executions can make the analysis load too high, which results in the abandonment of this activity and thus makes the execution useless.

Agilitest facilitates the analysis of tests by offering ATSV videos and pdf reports to quickly analyze the causes while the error occured.

The lack of robustness of the tests

The robustness of the tests and their reliability is mandatory with automation. Indeed, when a test fails, we must be sure that it is (this is also the case when it succeeds, but it is more difficult to detect). This robustness must be close to 100% because when it is not the case. Indeed, having a campaign of 100 tests that are 95% reliable and that must all be successful, as in an integration chain, becomes almost impossible.

Agilitest, with its verification and object targeting system, helps you to make your tests more robust and allows you to strongly limit flaky tests.

Tests that do not verify what is required

This problem comes from the fact that automatic checks are not human checks. Something that works with the human eye does not necessarily work automatically. For example, you can't search for "Bus" in a page that also contains the word "Busy".  Agilitest ensures the uniqueness of what is searched for and therefore avoids searching for something that is found several times in the page. The use of regular expressions will allow you to formally identify the elements to be verified. As a last resort or for specific needs, you can use the graphical recognition.

The choice of an unsuitable tool

Some tools are adapted to certain technologies, some tools allow you to do certain types of tests (performance, User Interfaces, Webservices...), some tools allow you to use certain components... The choice of tool is crucial because starting to automate with a tool and then realizing that a mandatory action cannot be done with it leads to a big waste of time, energy and money.

Agilitest is multi-channel on many technologies, with the same way of testing for each technology, and since we know that all our customers have specific needs, we make sure that they will succeed in performing the tests they want by regularly producing new functions at their request.

Too high a cost of automation or wanting to automate everything

Automation is an investment in time and money. In some cases only time is taken into account. In the vast majority of cases, the main goal is to earn money in the medium term. To ensure this return on investment, it is necessary to know what it is possible to automate and maintain at an acceptable cost. This means making choices and not automating everything in order to limit the number of tests to be maintained but also to limit the complexity of automated tests.

Agilitest allows high productivity, and will help you cover  as many scenarios as possible on your application. Deliveries will be even easier.

Want to give Agilitest a try?

See Agilitest in action. Divide by 5 the time needed to release a new version.

Scaling functional test automation for happy teams

  • From manual to automated tests
  • From test automation to smart test automation
  • Finding the right tools
Marc Hage Chahine

About the author

Marc Hage Chahine

Agilitest Partner – Passionate about software testing at Qestit - ISTQB Certified (Foundation, Agile, Test Manager)

linkedin logo

Get great content updates from our team to your inbox.

Join thousands of subscribers. GDPR and CCPA compliant.