Transferability tests

Description

Nowadays, there is a huge diversity of configuration onto which software is being run mixing

• Browsers, one major source of portability issues [Brown 2003], along with their plugins 
• Display standards, resolutions and processors (e.g. OpenGL, DirectX) with different versions
• Operating systems with an increasing creation rate 
• Hardware (mobiles, tablets, laptops, servers, …)

Those configurations emerge from technology evolution and market needs which try to adapt to every need. When changing the point of view from an analytic perspective to an external point of view, portability matters is quickly at stake [Letouzey 2012] because Users have different contexts (machines, OSs, display, time zones, languages & locales, …) which is amplified with mobility. Portability issues quickly raise dissatisfaction when configuration doesn't fit your product.

‍

When it comes to enabling your product to work on several combinations, Transferability Testing (TT) is key.

TT is all about ensuring the solution is portable from one environment to another. This is an NFR that helps answer the question "Are we building the right product?" and ultimately "Are we building the product right?" to ensure this characteristic described as [ISO 25010]:

• adaptability: The ability of the product to be adapted to different environments without applying actions or means other than those provided for this purpose for the software considered
• portability testing: test to identify anomalies caused by a transfer to one or more other environments, time matters included such as time zones [Letouzey  2012], and to deduce a degree of portability - Sometimes, portability is close to reuse but it implies reusing a whole product in new environments which infers different strategies than reusing components [Mooney 2008]
• coexistence: also called sociability or compatibility testing, this is the ability of the software product to properly coexist with other independent software or hardware parts in a common environment and sharing common resources
• installability:The ability of the software product to be installed in an environment and connected to third parties already installed that may impact the installed software, the possibility of uninstallation and the duration of the installation process
• replaceability: The ability of the software product to be used in place of another similar software product in a given environment with system features expected to remain unchanged after replacement - goes along with compliance.

Replaceability eases a Lean practice name SMED which consists in being able to replace a tool in less than 10 minutes which agilizes both maintenance activities and Users that may need to change their own ways of work to adapt their business changes.

Software is said to be portable when it can, with reasonable effort, be made to run on computers other than the one for which it was originally written [Brown 2003]. The history of software development has shown a trend which leads towards higher levels of abstraction, from OSs to communication protocols (eg. HTTP) or standards and abstractions. Each level of abstraction allows Developers to focus more directly on solving the problem at hand rather than implementation details [Oglesby 2001].

TT is a direct application of the principle “Testing is context dependent” [ISTQB 2018] which means that whenever there is a need to change the context of the product, TT must be taken into account to install to, use in, and perhaps move to various environments to be able to move and adapt to new environments [Black 2015].

Impact on the testing maturity

Classically, to face the combinations of environments when it comes to TT, different testing techniques may be involved [Black 2015]:

• classification tree, 
• equivalence partitioning, 
• decision tables

All those techniques find some limit when exposed to exponential possibilities. Under those circumstances, heuristics should be added to these techniques, pairwise testing being one of those. However, more pragmatic approaches can be involved. Statistics are of good help to let the Team focus on what the product is more likely to face notably with a Pareto chart. Those statistics should be known from marketing surveys to understand the marketplace and customers needs and wants [Armstrong 2017]. Those informations help to define NFR related to TT in order to produce relevant Epics and User Stories. It is essential to understand the business or technical case in determining which trade-offs are advantageous.

TT lead then to consider [Mooney 2004]

• Language-based strategies - today, some languages provide portable capabilities (Java, C#, Flutter, JavaScript, …)
• Component and Library based strategies - some libraries are compatible across many platforms and ease porting to several environment at once
• Operating system strategies - using Virtual Machines or OS-level virtualization frameworks such as Docker or Kubernetes  are great portability examples
• Architecture strategies - isolating environment-dependent modules from independent parts leads to replaceability matters
• Data portability - the data format may differ between environments that impacts notably file names, media compatibility, character codes, etc. or when upgrading a version, the data migration
• Data Transportation - when data is moved, they may face data portability issues, but also in terms of volume notably with network configurations or when transmission is subjected to interruptions or data loss
• Culture - (languages and locales, adapting to user experience, environment local constraints, …)

It becomes then clear that TT should be integrated within every ideation activities and levels.

System testing is about reliability achievement and evaluation from different environment perspective with several types of TT, notably:

• Automated acceptance tests
• Alpha and Beta Testing along with monitoring solution with observables
• Configuration Testing with classical or statistical methods
• Conformance/Functional/Correctness Testing
• Graphical User Interface Testing notably with internal device farms or public farms such as BrowserStack or SauceLab
• Installation Testing with the help of multiple virtual machines with several configurations
• Penetration testing on different platforms [Salonen 2012], since vulnerabilities may differ from one platform to another
• Performance and Stress Testing on different environments [Salonen 2012]
• Recovery Testing since fault-tolerance may differ and system recovery capacities regarding data backup and restoration; thus, different environments of software should be involved in TT to support “all” defined platforms
• Regression testing with “Back-to-Back” Testing which involves two different versions of the software which are ran through the same set of tests and their test results are compared
• Usability Testing with crowdsourcing services that provide a selection of test users from different demographics.

Integration testing ensures the compliance across parts, which is facilitated with test harnesses to perform local testing in a sandwich approach where integration issues are more likely to happen or where the impacts would be drastic on the whole system. It also appears that portability is an "Architecturally Significant Requirement” (ASR) [Chen 2013] [Moustier 2019-1] that has a huge impact on the solution portability if not taken into account ASAP.

At Unit Testing time, portability can be used in several ways:

• to learn and verify how external libraries behave
• to verify that basic operations work correctly despite changes in the execution environment
• through TDD to provide a robust code designing in a baby-step process [Salonen 2012]

Testability and TT should occur at every activity related with software development [Mooney 2008]:

• at ideation starts when requirements provide portability barriers that could be useless or subject to obsolescence, portability along with other NFR goals which provide constraints that are subject to be altered across several environment
• at design time with external interfaces which can be preserved with SOLID / DDD principles, standards or isolation
• at implementation time with a portable language, standards and discipline
• at testing time with reusable test plan and testing portability itself
• at documenting time with separate system-dependent and independent portions into distinct sections or separate documents and portability processes
• at maintenance time with reusable parts

‍

To establish TT at every level, the Management should facilitate this for TT, as any other NFR, is mainly a matter of culture [Moustier 2019-1]. This focus on the working environment control empowers Teams and leads them to embody TT practices and let TT strategies be fed from both top down and bottom up approaches about 3 principles [Mooney 2004]:

• Control interfaces
• Isolate dependencies
• Think portable

which is facilitated by

• Standardizing the Interface
• Porting the “other side”, notably with mock objects and simulators
• Translating the Interface to enable universal plugs

TT can be monitored through indicators. From an economic point of view, TT happens when the cost of porting to multiple environments is higher than the cost of redevelopment [Mooney 2008]. Ideally, the port should cost zero which is impossible; therefore, the software is characterized by a Degree of Portability (DP) which can be formalized:

DP = 1 - (cost to port / cost to redevelop)

This “1-A/B” formula canvas can be declined into operational parts with 

• “A”: the number of items correctly operable after adaptation
• “B”: the number of total items

For example, items may refer to [Black 2015]

• portability: applicable to data structures, Hardware environment features, Organizational environment functions, System software environmental, or any kind of items 
• Replaceability: Continued use of data, “Function inclusiveness” (i.e. functions expected to remain unchanged after replacement), 
• installability: Ease of set-up retry features, Installation effort, Installation flexibility with customizable installation operations
• coexistence: Available coexistence (with number of entities with which the product is expected to coexist), Replaceability, Continued use of data (with items that are expected to be usable after replacement as confirmed in review), Function inclusiveness
• compliance: implemented items related to portability compliance as confirmed in review

When it comes to coexistence, such issues happen mainly when groups are organized into silos with Team [Black 2015] that are subjected to the Conway Law [Conway 1968]. This can be fixed with Teams that share testing with other project teams to try to avoid coexistence problems [Black 2015], DDD and PanTesting [Moustier 2020].

Since any NFR can be addressed progressively, any portability issue can actually be reinforced with the agile testing quadrants. Ultimately, if changes occur frequently, applying Jidoka to fit what is non desirable and improve automatic adaptability to context changes.

Portability issues raise the bar of independent frameworks and the underlying architecture.

Agilitest’s standpoint on this practice

Agilitest provides a solution to automate test scripts on different OS, browsers & mobiles. This capability facilitates TT with user-friendly switching between testing interfaces ; however, this TT is limited with internal farms since the engine is not based on Selenium. This engine is published on github under an Apache-2.0 License [Pierrehub2b 2021] that which enables running compliance test scripts without any fee; however it is not yet included on public farms for massive TT. Agilitest is therefore a good intermediate solution to start TT with in-house testing solutions.

‍

To discover the whole set of practices, click here.

‍