Understand what to test

State of mind

Rather than checking the functionality

State of mind

How to understand what to test

Generating a product requires understanding it beforehand. Say, developers are facing a User Story (US) that they don’t understand, it will take them ages to create what is actually expected. Moreover, they would quickly make false assumptions based on what they thought they understood. Worst of all, making assumptions is inevitable unless the taken path is already known or obvious. Unfortunately, building a system is usually not something obvious but a rather complex task and the farther you will go into the unknown, the bigger the hypothesis you will need to make.

Logic may help to go further thanks to what is named “Modus Ponens”, a deductive approach known from antiquity and formulated by Theophrastus:

Considering two assertions

  • If A then B
  • A

Therefore B can be claimed

Unfortunately again, the Modus Ponens leads to a reasoning that is correct and reinforces the conviction that we are right although the conclusion is false if the premise is actually false [McGee 1985].

There is another reasoning technique named “Modus Tollens”, an inductive approach still from Theophrastus which helps denying antecedents:

Considering two assertions:

  • if A then B
  • not B

Therefore A is invalid

To avoid deductive fallacies from the Modus Ponens, its Modus Tollens transposition may be used to exclude hypotheses provided that “Double Loop Learning” [Argyris 1977] [Moustier 2020] is involved. This learning strategy leads people to target a direction thanks to an expected knowledge located at a higher range which infers a clear understanding of what is expected.

Impact on the testing maturity

When it comes to complex environments such as developing a large software, according to the Cynefin framework [Snowden 2007], the mindset should move from categorizing or analyzing to probing and sensing which is the basis of testing.

Cynefin framework by [Snowded 2014]

Another component of testing is the use of critical thinking at each step on a regular basis with short iterations  [Bolton  2012] [Bach 2016] [Moustier 2019-1]. This mindset leads to “be a tester, not a checker” and become customer advocates, which is possible only if testers truly understand users and the way they will use the product and how they will check if something works correctly [Laing 2016] [Moustier 2019-1].

In terms of practices, there are several agile tricks that provide double loop learning arrangements:

  • Lean Startup helps consolidating hypotheses from the market which is enabled by automated observables and deducted US that implement the hypotheses to be checked 
  • 3 Amigos” workshop is a very good way to infuse into the US what is to be tested at a higher range of the code; the acceptance criteria then can be automated through an ATDD approach
  • Finally, the TDD coding style provides an immediate feedback loop on the assumptions made at code level.

Double Loop Learning from Lean Stratup to TDD [Moustier 2020]

Agilitest’s standpoint on this practice

Agilitest basically targets acceptance criteria at US and product level. But relying only on those tests is not enough to avoid fallacies neither at market nor at code level. Moreover, it would also introduce some feedback delay since automation requires a product ready to be delivered or at least good enough to be testable by a robot.

Therefore, to avoid a testing ice cone and leaving developers deep diving into wrong assumptions, it is highly recommended to perform multiple kinds of testing techniques that should be built-in from the very beginning [SAFe 2021-26][SAFe 2021-27] with a highly testable product.

To go further

  • [Moustier 2019-1] : Christophe Moustier – JUN 2019 – « Le test en mode agile » - ISBN 978-2-409-01943-2
© Christophe Moustier - 2021