Automate everything you can

Description

In the software testing context, when it comes to automation, scripting test cases is the first automation strategy that emerges.

 “It is difficult to claim that you are Agile, if you are not writing lots of automated test cases” [Martin 2007]

Still according to Robert C. Martin, not only is manual testing highly stressful, tedious and error prone, it is also immoral because it turns humans into machines [Martin 2007]. If a test procedure can be written as a script, it is also possible to write a program to execute that test procedure. This leads to cheaper, faster and more accurate testing than the manual work performed by humans, and it also frees humans to do what we do best: create [Salonen 2012].

These late years, the perceived benefits of test automation have drastically increased [Sogeti 2020] and experienced Teams definitely try to introduce test automation in their delivery.

‍

When it comes to test automation, maintenance always comes into the path. Test maintenance is triggered by false positives, the so-called “flaky tests”. Flakiness doesn't just impact testing. It has a direct impact on your team's confidence in software delivery. It is therefore essential to aim for that 100% robustness [Craske 2021]

Lean management has introduced the caring of automated assets as a pillar of the organization. The term used is “Jidoka”. This japanese term is a mix of 2 words: autonomy and automation, thus the translation, “autonomation”. In places where Jidoka takes place such as plants (but not only [Danovaro 2008] [Clinton 2008] [Onetto 2017] [Sridharan 2021] ), robots are improved to progressively face unforeseen situations to reduce manual interventions. 

Therefore, any automation strategy should also include some Jidoka [Moustier 2022] to avoid losing faith in automated tasks [Craske 2021].

DevOps pushes Team to automate things as much as possible, this is why Google promotes in its SRE approach to let Developers contribute to new features half of the time and the other half should be dedicated to introducing automation that would help Ops and improve generated scripts [Beyer 2016]. Not only Developers should support automation and improve production means but also the testing means to increase confidence in the solution because testing means are also part of the production assets since they participate to their quality level.

Impact on the testing maturity

Actually, automation does not start from test scripts automation. When Testers involve test management tools such as Squash, Octane or ALM, this is to replace manual actions that would be difficult to sustain when a massive amount of test cases are available [Bach 2016] [Moustier 2019-1]. Such test management tool already automate activities such as

• the versioning of the test cases
• the run of each step in test cases
• the requirement coverage calculation with the test run results

If your test cases were handled with simple text files or spreadsheets, those tasks will be handled manually with some uncertainty due to human interactions with the test assets.

Testeur should always look for and try some tools that would automate part of their job in order to enable faster and reliable feedback [Bach 2016].

However, automation is not enough because testing is more than repeated scripts. James Bach, the originator of Exploratory Testing with Cem Kaner, demonstrates some funny test approach on a silly toy that scripts could not imagine [Bach 2009]. Moreover, by definition, scripts repeat the same actions without any change. This advantage embeds some drawbacks, especially from the Pesticide Paradox [ISTQB 2018] point of view. After a while, those scripts become useless as bugs get resistant to the pesticide if the same chemical is applied. Variance is required to give testing a chance to find robust bugs.

In this direction, Spotify introduces a data dictionary service which randomly alters original data to introduce some fuzziness and push the limits of existing scenarios [Karl 2013]; naturally, involved data sets are recorded to keep track of tests and enable thorough replays. To go further in changing scenarios, the “Mutation Testing” concept is an attempt to automate test changes to automatically alter existing scripts and try extra testing paths. Although there are some heuristics, this strategy leads to a well known and undecidable problem, the “Equivalent Mutant Problem”, a fully automated solution based on test mutation is unattainable [Kintis 2016]. Trying to completely solve testing through automation is a fallacy. This is only a heuristic.

To prevent such a fallacy agile culture introduced the Automated Testing Ice cream cone metaphor with manual testing that would complete the set of automated tests.

‍

‍

Agilitest’s standpoint on this practice

Agilitest takes part of automation with low maintenance thanks to a #nocode approach. This paradigm participates to reduce maintenance and therefore leave more time to Developers to create added value with other matters [Forsyth 2021] and non coders will also be able to take part of test script automation.

Moreover, Agilitest is gifted with a quite robust engine to adapt to UI changes. The tool can also retrieve data from a CSV file for data-driven testing matters which enables some data mutation like Spotify does with data generated by the DevOps pipeline.

In addition to these features that meet the main principles of test automation, Agilitest recommends that you always complement Agilitest scripts with the following elements 

• much more test at lower levels (integration and unit test)
• some manual testing, notably exploratory testing to fill the gaps between scripts and the Pesticide Paradox principle

updating, improving and decommissioning scenarios on a regular basis in a 5S approach

‍

To discover the whole set of practices, click here.

‍