How to understand what to test
Generating a product requires understanding it beforehand. Say, developers are facing a User Story (US) that they don’t understand, it will take them ages to create what is actually expected. Moreover, they would quickly make false assumptions based on what they thought they understood. Worst of all, making assumptions is inevitable unless the taken path is already known or obvious. Unfortunately, building a system is usually not something obvious but a rather complex task and the farther you will go into the unknown, the bigger the hypothesis you will need to make.
Logic may help to go further thanks to what is named “Modus Ponens”, a deductive approach known from antiquity and formulated by Theophrastus:
Considering two assertions
- If A then B
Therefore B can be claimed
Unfortunately again, the Modus Ponens leads to a reasoning that is correct and reinforces the conviction that we are right although the conclusion is false if the premise is actually false [McGee 1985].
There is another reasoning technique named “Modus Tollens”, an inductive approach still from Theophrastus which helps denying antecedents:
Considering two assertions:
- if A then B
- not B
Therefore A is invalid
To avoid deductive fallacies from the Modus Ponens, its Modus Tollens transposition may be used to exclude hypotheses provided that “Double Loop Learning” [Argyris 1977] [Moustier 2020] is involved. This learning strategy leads people to target a direction thanks to an expected knowledge located at a higher range which infers a clear understanding of what is expected.
Impact on the testing maturity
When it comes to complex environments such as developing a large software, according to the Cynefin framework [Snowden 2007], the mindset should move from categorizing or analyzing to probing and sensing which is the basis of testing.
Another component of testing is the use of critical thinking at each step on a regular basis with short iterations [Bolton 2012] [Bach 2016] [Moustier 2019-1]. This mindset leads to “be a tester, not a checker” and become customer advocates, which is possible only if testers truly understand users and the way they will use the product and how they will check if something works correctly [Laing 2016] [Moustier 2019-1].
In terms of practices, there are several agile tricks that provide double loop learning arrangements:
- Lean Startup helps consolidating hypotheses from the market which is enabled by automated observables and deducted US that implement the hypotheses to be checked
- “3 Amigos” workshop is a very good way to infuse into the US what is to be tested at a higher range of the code; the acceptance criteria then can be automated through an ATDD approach
- Finally, the TDD coding style provides an immediate feedback loop on the assumptions made at code level.
Agilitest’s standpoint on this practice
Agilitest basically targets acceptance criteria at US and product level. But relying only on those tests is not enough to avoid fallacies neither at market nor at code level. Moreover, it would also introduce some feedback delay since automation requires a product ready to be delivered or at least good enough to be testable by a robot.
Therefore, to avoid a testing ice cone and leaving developers deep diving into wrong assumptions, it is highly recommended to perform multiple kinds of testing techniques that should be built-in from the very beginning [SAFe 2021-26][SAFe 2021-27] with a highly testable product.
To go further
- [Argyris 1977] : Chris Argyris - « Double Loop Learning in Organizations » - Harvard Business Review - SEP/1977 - https://hbr.org/1977/09/double-loop-learning-in-organizations
- [Bach 2016] : James Bach - MAR 2016 - "My Brilliant Testing Ideas vs. Your Critical Thinking: A Challenge!" - Vilnius QA Gathering "Bugs'a'loud" - https://vimeo.com/161913983
- [Bolton 2012] Michael Bolton - "Critical Thinking for Testers" - EuroStar Conference - 2012 - https://www.developsense.com/presentations/2012-11-EuroSTAR-CriticalThinkingForTesters.pdf
- [Laing 2016] : Samantha Laing, Karen Greaves - « Growing Agile: A Coach’s Guide to Agile Testing » - Leanpub - 2016 - https://leanpub.com/AgileTesting
- [McGee 1985] ; V. McGee - 1985 - “A Counterexample to Modus Ponens” - https://doi.org/10.2307/2026276
- [Moustier 2019-1] : Christophe Moustier – JUN 2019 – « Le test en mode agile » - ISBN 978-2-409-01943-2
- [Moustier 2020] : Christophe Moustier – OCT 2020 – « Conduite de tests agiles pour SAFe et LeSS » - ISBN : 978-2-409-02727-7
- [SAFe 2021-26] : SAFe - FEV 2021 - « Core Values » - https://www.scaledagileframework.com/safe-core-values/
- [SAFe 2021-27] : SAFe - FEV 2021 - « Built-in Quality » - https://www.scaledagileframework.com/built-in-quality/
- [Snowded 2014] : Snowded - JUL 2014 - “Cynefin as of 1st June 2014.png” - https://en.wikipedia.org/wiki/File:Cynefin_as_of_1st_June_2014.png
- [Snowden 2007] : David J. Snowden et Mary E. Boone - NOV 2007 - “A Leader’s Framework for Decision Making” - https://hbr.org/2007/11/a-leaders-framework-for-decision-making