Automate everything you canAutomated testing
Automation increases reliability and the ability to do more
In the software testing context, when it comes to automation, scripting test cases is the first automation strategy that emerges.
“It is difficult to claim that you are Agile, if you are not writing lots of automated test cases” [Martin 2007]
Still according to Robert C. Martin, not only is manual testing highly stressful, tedious and error prone, it is also immoral because it turns humans into machines [Martin 2007]. If a test procedure can be written as a script, it is also possible to write a program to execute that test procedure. This leads to cheaper, faster and more accurate testing than the manual work performed by humans, and it also frees humans to do what we do best: create [Salonen 2012].
These late years, the perceived benefits of test automation have drastically increased [Sogeti 2020] and experienced Teams definitely try to introduce test automation in their delivery.
When it comes to test automation, maintenance always comes into the path. Test maintenance is triggered by false positives, the so-called “flaky tests”. Flakiness doesn't just impact testing. It has a direct impact on your team's confidence in software delivery. It is therefore essential to aim for that 100% robustness [Craske 2021]
Lean management has introduced the caring of automated assets as a pillar of the organization. The term used is “Jidoka”. This japanese term is a mix of 2 words: autonomy and automation, thus the translation, “autonomation”. In places where Jidoka takes place such as plants (but not only [Danovaro 2008] [Clinton 2008] [Onetto 2017] [Sridharan 2021] ), robots are improved to progressively face unforeseen situations to reduce manual interventions.
Therefore, any automation strategy should also include some Jidoka [Moustier 2022] to avoid losing faith in automated tasks [Craske 2021].
DevOps pushes Team to automate things as much as possible, this is why Google promotes in its SRE approach to let Developers contribute to new features half of the time and the other half should be dedicated to introducing automation that would help Ops and improve generated scripts [Beyer 2016]. Not only Developers should support automation and improve production means but also the testing means to increase confidence in the solution because testing means are also part of the production assets since they participate to their quality level.
Impact on the testing maturity
Actually, automation does not start from test scripts automation. When Testers involve test management tools such as Squash, Octane or ALM, this is to replace manual actions that would be difficult to sustain when a massive amount of test cases are available [Bach 2016] [Moustier 2019-1]. Such test management tool already automate activities such as
- the versioning of the test cases
- the run of each step in test cases
- the requirement coverage calculation with the test run results
If your test cases were handled with simple text files or spreadsheets, those tasks will be handled manually with some uncertainty due to human interactions with the test assets.
Testeur should always look for and try some tools that would automate part of their job in order to enable faster and reliable feedback [Bach 2016].
However, automation is not enough because testing is more than repeated scripts. James Bach, the originator of Exploratory Testing with Cem Kaner, demonstrates some funny test approach on a silly toy that scripts could not imagine [Bach 2009]. Moreover, by definition, scripts repeat the same actions without any change. This advantage embeds some drawbacks, especially from the Pesticide Paradox [ISTQB 2018] point of view. After a while, those scripts become useless as bugs get resistant to the pesticide if the same chemical is applied. Variance is required to give testing a chance to find robust bugs.
In this direction, Spotify introduces a data dictionary service which randomly alters original data to introduce some fuzziness and push the limits of existing scenarios [Karl 2013]; naturally, involved data sets are recorded to keep track of tests and enable thorough replays. To go further in changing scenarios, the “Mutation Testing” concept is an attempt to automate test changes to automatically alter existing scripts and try extra testing paths. Although there are some heuristics, this strategy leads to a well known and undecidable problem, the “Equivalent Mutant Problem”, a fully automated solution based on test mutation is unattainable [Kintis 2016]. Trying to completely solve testing through automation is a fallacy. This is only a heuristic.
To prevent such a fallacy agile culture introduced the Automated Testing Ice cream cone metaphor with manual testing that would complete the set of automated tests.
Agilitest’s standpoint on this practice
Agilitest takes part of automation with low maintenance thanks to a #nocode approach. This paradigm participates to reduce maintenance and therefore leave more time to Developers to create added value with other matters [Forsyth 2021] and non coders will also be able to take part of test script automation.
Moreover, Agilitest is gifted with a quite robust engine to adapt to UI changes. The tool can also retrieve data from a CSV file for data-driven testing matters which enables some data mutation like Spotify does with data generated by the DevOps pipeline.
In addition to these features that meet the main principles of test automation, Agilitest recommends that you always complement Agilitest scripts with the following elements
- much more test at lower levels (integration and unit test)
- some manual testing, notably exploratory testing to fill the gaps between scripts and the Pesticide Paradox principle
updating, improving and decommissioning scenarios on a regular basis in a 5S approach
To go further
- [Bach 2009] : James & Jon Bach - JUN 2009 - “ET with Subtitles #1” - https://www.youtube.com/watch?v=Vy0I2SB5OLo
- [Bach 2016] : James Bach - MAR 2016 - “A Context-Driven Approach to Automation in Testing” - https://www.youtube.com/watch?v=4vcMddaAzDs
- [Beyer 2016] : Betsy Beyer, Chris Jones, Jennifer Petoff et Niall Richard Murphy - « Site Reliability Engineering: How Google Runs Production Systems » - O’Reilly Media - 2016 - ISBN-13 : 978-1491929124 - https://landing.google.com/sre/sre-book/toc/index.html
- [Clinton 2008] : Keith Clinton - SEP 2008 - “Agile Game Development Jidoka, TDD and asset validation” - https://blog.agilegamedevelopment.com/2008/09/jidoka-tdd.html
- [Craske 2021] : Antoine Craske - AUG 2021 - “Test Debt Fundamentals: What, Why & Warning Signs” - https://qeunit.com/blog/test-debt-fundamentals-what-why-warning-signs/].
- [Danovaro 2008] : Emanuele Danovaro & Andrea Janes & Giancarlo Succi - JAN 2008 - “Jidoka in Software Development” - https://www.researchgate.net/publication/221321108_Jidoka_in_software_development
- [Forsyth 2021] : Alexander Forsyth – JAN 2021 - « Low-Code and No-Code: What’s the Difference and When to Use What? » - https://www.outsystems.com/blog/posts/low-code-vs-no-code/
- [ISTQB 2018] : ISTQB - 2018 - “Certified Tester Foundation - Level Syllabus” - https://www.istqb.org/downloads/category/2-foundation-level-documents.html
- [Karl 2013] : Kristian Karl - « How Spotify Does Test Automation » - 24/05/2013 - http://tracks.roojoom.com/r/93313#/trek?page=3
- [Kintis 2016] : Marinos Kintis - JUN 2016 - “On the Reduction of Mutation Testing's Cost” - http://pages.cs.aueb.gr/~kintism/phd-thesis/Kintis-PhD-2016.pdf
- [Martin 2007] : Martin, R.C., 2007. “TDD with Acceptance Tests and Unit Tests” - http://blog.objectmentor.com/articles/2007/10/17/tdd-with-acceptance-tests-and-unit-tests / https://sites.google.com/site/unclebobconsultingllc/tdd-with-acceptance-tests-and-unit-tests
- [Moustier 2019-1] : Christophe Moustier – JUN 2019 – « Le test en mode agile » - ISBN 978-2-409-01943-2
- [Moustier 2022] : Christophe Moustier - APR 2022 - “My Flaky Test is Agile” - https://youtu.be/Ptg5NICosNY?t=2701
- [Onetto 2017] : Marc Onetto & Catherine Chabiron - OCT 2017 - “Marc Onetto on his experience with Jidoka at GE and Amazon” - https://planet-lean.com/jidoka-lean-thinking-amazon/
- [Salonen 2012] : Ville Salonen - OCT 2012 - “Automatic Portability Testing” - https://jyx.jyu.fi/dspace/bitstream/handle/123456789/40043/URN%3ANBN%3Afi%3Ajyu-201210212735.pdf?sequence=1
- [Scott 2019] : Alister Scott - c. 2019 - « Testing Pyramids & Ice-Cream Cones » - https://watirmelon.blog/testing-pyramids/
- [Sogeti 2020] - https://www.sogeti.com/explore/reports/world-quality-report-2020/
- [Sridharan 2021] : Mithun Sridharan - JUL 2021 - “Andon Cord How Amazon And Netflix Use Toyota Principles” - https://thinkinsights.net/strategy/andon-cord/