Feedback from resources, the domain and the business

Active testing

Dashboard on the operation for immediate treatment of alerts

Active testing

Description

From a DevOps perspective, feedback and measurement are a necessity, and it is rooted in the core values of the practice:

  • This is the “M” in SAFe’s “CALMR” [SAFe 2021-43] or the well known “CALMS” coined by Jez Humble [Buchanan 2021]
  • Gene Kim, Jez Humble and Patrick Debois (the “father” of DevOps [DevOps Cafe 2010]) also wrote some DevOps principles about feedback
  • Google’s point of view on DevOps named “Site Reliability Engineering” (SRE) also has a strong concern about feedback and alerting [Beyer 2016]

The list could be expanded to an even larger set of references that would say the same thing, because the bottom line is that feedback synthesized with metrics is mandatory; and it’s not just tests but activities on the product. This feedback is part of the Shift Right Testing (SRT) approach which provides a “double loop learning” [Argyris 1977] on several items such as the resources, the domain expectations and the business/revenues.

Regarding resources, technical means such as CPUs, network, storage and memory must be monitored while under use. This helps to prevent a service disruption from regular business demands. When it comes to irregular service demands such as DoS attacks, alerts must be raised to handle such security matters and by extension the detection of suspect use of resources notably thanks to tools such as DDoS detection means or honey pots. This so-called “OPSEC” issue raises the need to handle any Non Functional Requirement (NFR) that would be significant on the product such as average response times, API version used to monitor compatibility issues, or even Customer’s feedback on implemented features.

When it comes to the domain, the Lean Startup approach [Ries 2012] or epics descriptions by SAFe [SAFe 2021-34] involve “leading indicators”. Those domain wise feedback provide hints to help predict the business outcome hypothesis [SAFe 2021-44].

Last but not least, the feedback from the business is also expected to ensure revenue and reach the proper balance between income, domain needs and resources. Tools such as APM help to translate IT metrics into business values. To illustrate this, it appears that the response time of an HTML transaction has a huge impact on web giants such as Amazon, Google, Yahoo, or Bing; for example, a delay of 100 ms results in a 1% drop in Amazon's revenue, or about $2 billion [Moustier 2019-1]!

Impact on the testing maturity

Getting some feedback from resources, the domain and the business is pretty useful to avoid service disruption or dissatisfaction due to bad assumptions. If this feedback is quickly provided, any issue can then be handled quickly and negative impacts are then limited to few incidents. Of course, this is possible only when the delivery process is fast and stable enough to afford a “fail fast, big and often” strategy as per a DevOps approach since it leads you to automate everything you can.

Still in an impact limitation perspective, this approach can be mixed with Canary Releasing to combine small impact and fast feedback.

Feedback from the domain is also used in A/B Testing to tell which option is actually prefered by the End Users.

Ultimately, feedback is also used to monitor the customer satisfaction and comply with any SLA with some margin with appropriate SLO’s and the Error Budget management that should go along with the anomaly handling policy such as Andon.

Agilitest’s standpoint on this practice

Agilitest is basically a tool designed for Shift Left Testing purpose; however, the tool can be easily turned into a sanity check automaton which can be launched on a regular basis to ensure the whole system is going right. This smoke testing sequence can simply be scripted on the most significant features to ensure the underlying parts (e.g. web services, databases) are still up and running. Since the tool does not require any coding knowledge, it can be used and maintained by people in a Helpdesk service.

Related cards

To go further

© Christophe Moustier - 2021