Do everything possible at all levels of the organization to try to shift the testing effort upstream of the solution
The definition of Shift Left
Whatever the name you assign to software quality practices, there are mainly four classes of testing styles for any artifact:
- Preventing bugs before they are introduced in the artifact
- Reviewing the artifact while it is being built
- Reviewing the artifact once it is eventually ready for delivering to Customers - both static and dynamic reviews
- Monitoring dynamically the artifact once it is delivered to Customer - see Shift Right
The test principle “Early testing” [ISTQB 2018] relates to testing as soon as possible, that is to say tests types A if possible, eventually B if Testers can participate in the product ideation process. The idea of “Shift Left Testing” is pulling the testing effort towards A, as much as possible, i.e. shifting the testing effort on the left side of the SDLC. However, with agile development, the product is built iteratively which means that types C and D will participate in “testing early” for the next product increment.
Therefore early testing strategy should be thought of as continuously performing and improving testing wherever in the SDLC. Thus, the Swiss cheese model [Reason 2006] can be used to manage the appropriate quality standard to apply regarding the context [Bach 2019]:
- existing controls may be reinforced to reduce holes in cheese slices
- extra controls may be added if more appropriate or cheaper (Pareto’s law)
The Swiss cheese model highlights the necessity to combine tests that is confirmed by the table here below which provides an average percentage of defects found in type C static tests [Firesmith 2014]:
Average Percentage of Defects Found as a Function of Static Verification Method and Defect Type [Firesmith 2014]
When combining static and dynamic, the total effectiveness would reach only 99.27%.
This figure may seem fair, though it does not take into account the financial impact behind it : between $22.2 billion and $59.5 billion annually [NIST 2002][Firesmith 2014] or even up to $113 billion knowing that only $30 billion would end world hunger [Initial State 2014].
To raise the bar regarding defect identification, we need to tackle the hidden devil in every detail through numerous testing techniques that offer different perspectives on the item we want to test as per the “Testing is context dependent” principle [ISTQB 2018], just as you would walk around a car to inspect for defects when you buy it.
Another benefit of shift left testing is that Non Functional Requirements (NFR) are taken into account very early. This is utmostly critical because NFR are among the “Architecturally Significant Requirements” (ASR) [Chen 2013], i.e. they impact drastically the way the architecture will look like.
Impact on the testing maturity
The so-called “Continuous Testing” is a combined series of activities and mindset which includes
- Introducing testing as early as possible notably through Testability of the Product Backlog
- Product ideation in group with workshops such as 3 Amigos
- Built-in Quality to Prevent bugs [SAFe 2021-27] notably thanks to
- a testable design to enable Testability as soon as possible of the product parts and Test Harnesses to assess those parts
- some Fail Fast mechanism
- including Automation of observables to prepare type D testing and Monitoring the deployed solution on resources, the domain and the business
- testing often with the Continuous Integration (CI) toolchain thanks to unit tests, acceptance criteria and any other automated tests scripts [Devopedia 2021]
- End-to-end testing as soon as possible even before the whole system is available
- Once the product is deployed, you may also voluntarily disrupt the system or trigger a test of your Disaster Recovery Plan (DRP) to see how resilient to disruptions it is and prepare some remediation plan if needed before genuine crashes would arrive
- Last but not least, retrospectives are also a good mean to take the benefit of some experience gained and improve the system to avoid issues occuring
Every time a test is done to prevent some undesirable situations, there should be some kind of gouvernance knowledge that should guide you, beyond the simple and explicit requirements. This is what is named “Double Loop Learning” [Argyris 1977]. The more you try to anticipate, the more fuzzy those explicit requirements may be. This is why X-Teams and PanTesting should be of great help to dive even more into the Shift Left matter.
Agilitest’s standpoint on this practice
Since Agilitest is designed for Functional End-to-End testing, Agilitest is usually involved at the end of the SDLC and there is a risk that those tests would delay the product delivery or lower the flow or happen after the Sprint and deployment.
Shift left is extremely valuable when it comes to test scripts automation because it gives you the opportunity to run your tests during the Sprint. This testing strategy leads towards some agile practices at least regarding team collaboration [Subrahmanyam 2021] because testing at the end appears to be a pitfall [Firesmith 2014][Magowan 2020].
To go further
- [Argyris 1977] : Chris Argyris - « Double Loop Learning in Organizations » - Harvard Business Review - SEP/1977 - https://hbr.org/1977/09/double-loop-learning-in-organizations
- [Bach 2019] : James Bach - MAI 2019 - ”Heuristic Test Strategy Model” - https://www.satisfice.com/download/heuristic-test-strategy-model
- [Chen 2013] : Lianping Chen & Muhammad Ali Babar & Bashar Nuseibeh - FEB 2013 - “Characterizing Architecturally Significant Requirements” - https://www.researchgate.net/publication/255569055_Characterizing_Architecturally_Significant_Requirements
- [Devopedia 2021] : "Shift Left." Version 5, June 28. Accessed 2021-06-28. https://devopedia.org/shift-left
- [Firesmith 2014] : Donald G. Firesmith - AUG 2014 - “Common System and Software Testing Pitfalls: How to Prevent and Mitigate Them: Descriptions, Symptoms, Consequences, Causes, and Recommendations” - isbn:9780133748550
- [Initial State 2014] : www.InitialState.com - « The Cost (and Opportunity) of Product Bugs » - 2014 - https://fr.slideshare.net/initialstate/product-bug-infographic
- [ISTQB 2018] : ISTQB - 2018 - “Certified Tester Foundation - Level Syllabus” - https://www.istqb.org/downloads/category/2-foundation-level-documents.html
- [Magowan 2020] : Kirstie Magowan - NOV 2020 - “Shift Left Testing: What, Why & How To Shift Left“ - https://www.bmc.com/blogs/what-is-shift-left-shift-left-testing-explained/#
- [Moustier 2019-1] : Christophe Moustier – JUN 2019 – « Le test en mode agile » - ISBN 978-2-409-01943-2
- [Reason 2006] : James T. Reason - « Revisiting the "Swiss Cheese" Model of Accidents » - Eurocontrol - Oct 2006 - http://www.eurocontrol.int/eec/gallery/content/public/document/eec/report/2006/017_Swiss_Cheese_Model.pdf
- [SAFe 2021-27] : SAFe - FEV 2021 - « Built-in Quality » - https://www.scaledagileframework.com/built-in-quality/
- [Subrahmanyam 2021] : Gayathri Subrahmanyam - JUL 2021 - “Shift Left Testing: A Secret Mantra for Software Success” - https://www.softwaretestinghelp.com/shift-left-testing-approach/