Feedback from resources, the domain and the businessActive testing
Dashboard on the operation for immediate treatment of alerts
From a DevOps perspective, feedback and measurement are a necessity, and it is rooted in the core values of the practice:
- This is the “M” in SAFe’s “CALMR” [SAFe 2021-43] or the well known “CALMS” coined by Jez Humble [Buchanan 2021]
- Gene Kim, Jez Humble and Patrick Debois (the “father” of DevOps [DevOps Cafe 2010]) also wrote some DevOps principles about feedback
- Google’s point of view on DevOps named “Site Reliability Engineering” (SRE) also has a strong concern about feedback and alerting [Beyer 2016]
The list could be expanded to an even larger set of references that would say the same thing, because the bottom line is that feedback synthesized with metrics is mandatory; and it’s not just tests but activities on the product. This feedback is part of the Shift Right Testing (SRT) approach which provides a “double loop learning” [Argyris 1977] on several items such as the resources, the domain expectations and the business/revenues.
Regarding resources, technical means such as CPUs, network, storage and memory must be monitored while under use. This helps to prevent a service disruption from regular business demands. When it comes to irregular service demands such as DoS attacks, alerts must be raised to handle such security matters and by extension the detection of suspect use of resources notably thanks to tools such as DDoS detection means or honey pots. This so-called “OPSEC” issue raises the need to handle any Non Functional Requirement (NFR) that would be significant on the product such as average response times, API version used to monitor compatibility issues, or even Customer’s feedback on implemented features.
When it comes to the domain, the Lean Startup approach [Ries 2012] or epics descriptions by SAFe [SAFe 2021-34] involve “leading indicators”. Those domain wise feedback provide hints to help predict the business outcome hypothesis [SAFe 2021-44].
Last but not least, the feedback from the business is also expected to ensure revenue and reach the proper balance between income, domain needs and resources. Tools such as APM help to translate IT metrics into business values. To illustrate this, it appears that the response time of an HTML transaction has a huge impact on web giants such as Amazon, Google, Yahoo, or Bing; for example, a delay of 100 ms results in a 1% drop in Amazon's revenue, or about $2 billion [Moustier 2019-1]!
Impact on the testing maturity
Getting some feedback from resources, the domain and the business is pretty useful to avoid service disruption or dissatisfaction due to bad assumptions. If this feedback is quickly provided, any issue can then be handled quickly and negative impacts are then limited to few incidents. Of course, this is possible only when the delivery process is fast and stable enough to afford a “fail fast, big and often” strategy as per a DevOps approach since it leads you to automate everything you can.
Still in an impact limitation perspective, this approach can be mixed with Canary Releasing to combine small impact and fast feedback.
Feedback from the domain is also used in A/B Testing to tell which option is actually prefered by the End Users.
Ultimately, feedback is also used to monitor the customer satisfaction and comply with any SLA with some margin with appropriate SLO’s and the Error Budget management that should go along with the anomaly handling policy such as Andon.
Agilitest’s standpoint on this practice
Agilitest is basically a tool designed for Shift Left Testing purpose; however, the tool can be easily turned into a sanity check automaton which can be launched on a regular basis to ensure the whole system is going right. This smoke testing sequence can simply be scripted on the most significant features to ensure the underlying parts (e.g. web services, databases) are still up and running. Since the tool does not require any coding knowledge, it can be used and maintained by people in a Helpdesk service.
To discover the whole set of practices, click here.
To go further
- [Argyris 1977] : Chris Argyris - « Double Loop Learning in Organizations » - Harvard Business Review - SEP/1977 - https://hbr.org/1977/09/double-loop-learning-in-organizations
- [Beyer 2016] : Betsy Beyer, Chris Jones, Jennifer Petoff et Niall Richard Murphy - « Site Reliability Engineering: How Google Runs Production Systems » - O’Reilly Media - 2016 - ISBN-13 : 978-1491929124 - https://landing.google.com/sre/sre-book/toc/index.html
- [Buchanan 2021] : Ian Buchanan - accessed in SEP 2021 - “CALMS Framework” - https://www.atlassian.com/devops/frameworks/calms-framework
- [DevOps Cafe 2010] : DevOps Cafe - SEP 2010 - “Love it or hate it, the term DevOps is here to stay... and we know who to thank” - http://devopscafe.org/show/2010/9/15/episode-12.html
- [Moustier 2019-1] : Christophe Moustier – JUN 2019 – « Le test en mode agile » - ISBN 978-2-409-01943-2
- [Ries 2012] : Eric Ries - 2012 - « Lean start-up » - Pearson - ISBN 978-2744065088 - http://zwinnalodz.eu/wp-content/uploads/2016/02/The-Lean-Startup-.pdf
- [SAFe 2021-34] : SAFe - FEV 2021 - “Epic” - https://www.scaledagileframework.com/epic/
- [SAFe 2021-43]: SAFe - FEV 2021 - “CALMR” - https://www.scaledagileframework.com/calmr/
- [SAFe 2021-44] : SAFe - FEV 2021 - “Applied Innovation Accounting in SAFe” -https://www.scaledagileframework.com/guidance-applied-innovation-accounting-in-safe/