NFR to test

Planning

Non-Functional Requirements are part of the US to have a potentially deployable product increment

Planning

Description

Everyone knows about functional requirements & testing. However, Non Functional Requirements (NFR) appear to be the key to lovable products, especially if there are competitors on the market.

Culture influences a lot NFRs [Moustier 2019-1] [Moustier 2020]:

  • Banking would promote security a lot
  • Gaming is focussed a lot on the User Experience
  • Medical devices would also provide a bunch of NFR, should it be on materials, ways of working, robustness, safety to people, …

As a reference, the [ISO 25010 2011] is commonly cited. This standard is based on eight “characteristics”:

  • Portability : Adaptability ; Installability ; Replaceability
  • Maintainability : Modularity ; Reusability ; Analyzability ; Modifiability ; Testability
  • Security : Confidentiality ; Integrity ; Non-repudiation ; Accountability ; Authenticity
  • Reliability : Maturity ; Availability ; Fault Tolerance ; Recoverability
  • Functional Suitability : Functional Completeness, Correctness, Appropriateness
  • Performance Efficiency : Time Behavior ; Resource Utilization ; Capacity
  • Compatibility : Co-existence ; Interoperability
  • Usability : Appropriateness Recognisability ; Learnability ; Operability ; User Error Protection ; User Interface Aesthetics ; Accessibility


There are different kinds of NFR to test that can be expanded with more characteristics. It takes some time to take care of NFRs.  Google’s SRE, DevOps vision of Google forecasts that Developers should not spend more than 50% of their time on new features [Beyer 2016]. The rest of their time is assigned to help Ops, that is to say operability and maintainability matters. To help gouvernance, SRE introduces the “Error Budget” to help the Product Owner (PO) to balance new features with technical debt.

NFR also strongly impacts architecture. These so-called “Architecturally Significant Requirements” (ASR) [Chen 2013] should then be known as soon as possible to limit deep refactorings.

Discovering NRF is a matter of

  • culture of groups and individuals
  • experience on the business and End Users’ groundfield that can be gained via Gemba walks or Design Thinking

Actually, NFRs strongly represent the culture of the organization; for instance, companies that would care a lot about providing “green products” as their baseline should include testing how much a given part should also be green. It would be a shame if some part would reject tons of CO2 or would involve highly toxic materials in their process...

Impact on the testing maturity

Handling NFR is something that should impact every activity in the Software Development Life Cycle. In an agile configuration, NFR should not be treated in a staggered approach after the whole release is ready. This matter should be handled “continuously”, that is to say it should included during any activity; for instance, 

  • SAFe introduces NFR as part of the description of an epic [SAFe 2021-34]
  • Architecture should take ASRs into account as early as possible [Moustier 2019-1]
  • The four quadrants of testing includes two quadrants on NFR [Crispin 2021]

This is why NFR impacts both DoR and DoD; however, handling any kind of NFR at once is merely impossible from the start and would make no sense for a MVP to support 1M Users from the start if the product is not to be insanely viral on the mass market. Fortunately, it appears that NFR are usually progressive[Larman 2010]. This property enables a progressive adoption (say, moving from 1k Users to 10k Users) which is usually aligned with products newly deployed on the market. With the time being, reinforcing a NFR should be reflected in both DoR and DoD updates.

Agilitest’s standpoint on this practice

By Agilitest, they know how crucial NFR are. This is why the product can be plugged to a performance measuring tool such as Octoperf [Agilitest 2020].

Related cards

To go further

  • [ISO 25010 2011] : BSI Standards Publication - “Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models” - BS ISO/IEC 25010:2011
  • [Larman 2010] : Craig Larman, Bas Vodde - « Practices for Scaling Lean & Agile Development - Large, Multisite, and Offshore Product Development with Large-Scale Scrum » - Addison-Wesley - 2010 - ISBN-13: 978-0-321-63640-9
© Christophe Moustier - 2021