Adopting the Agilitest solution to create and maintain an automated test repository is a multi-step process. Many of our articles and technical documentation pages discuss the getting started process, or our preferred approach to test robustness and maintainability.
This article specifically addresses a step that is mandatory when you have already chosen Agilitest and want to integrate the solution into your production environment for test execution: what architecture to use for massive test execution, what types of hardware, and what software technologies to use?
What is required to run a test produced with Agilitest ?
Agilitest is a software that runs under the Microsoft operating system, in its desktop version, Windows 10, and its server version, Windows Server 2016 and 2019.
Before running a test, you first need a test case in ATS format created with Agilitest. This ATS test case can be compiled and integrated into a TestNG execution suite thanks to Agilitest for more ease, even if this is possible without.
Two main possibilities exist to execute one or more tests:
- launch a command line related to a TestNG execution
- use by Jenkins a Maven pom.xml configuration file generated in Agilitest
The execution environment of a test consists of an open Windows user session in which the test is executed.
When the test will run, the software to be tested will be launched and all the functional steps will scroll on the screen as if a virtual user was performing the actions with his keyboard and mouse.
Then the prerequisites depend on the channel or channels used in the automated test scenario: desktop, web, webservice, iOS, Android.
Running desktop tests
The first step of a desktop test is to launch the software to be tested, most often in the form of an .exe file to be opened. ATS and Agilitest can handle thi, but it is also possible to connect to a running process when absolutely necessary
A graphical Windows session is required for desktop tests.
Running web tests
The execution environment for a web test is more predictable than for a desktop test since the first step is always to launch a supported browser: Chrome, Firefox, Chromium-based Edge, Opera or Internet Explorer.
A graphical Windows session is required for web tests.
Running webservices tests
Using a webservice through HTTP requests is a lower-level operation than using software with a graphical interface.
Therefore, a Windows console session may be sufficient to play a webservices test.
While it is not essential to have a graphical desktop environment in this case, it is highly recommended in order to benefit from the multi-channel aspect of Agilitest.
Android and iOS mobile test execution
The first step in running a mobile test is to launch an application on a given mobile. It is therefore important to be rigorous in the configuration of your system, with for example a stable IP addressing plan for the mobiles or emulators.
The physical mobiles will have to be connected by USB cable and connected to the same network as the machine running the test.
The mobiles emulated in the Genymotion cloud simply require a functional Internet connection. This solution is very easy to use, and allows you to run the tests on any Windows machine, virtualized or in the cloud, without ever needing a physical mobile farm.
A graphical Windows session is still required, especially to be able to generate test reports with screenshots, and replay videos.
What does it take to replay tests intensively?
The user session, main component of the execution chain
An essential prerequisite for the execution of an Agilitest test is the presence of a graphical Windows session.
It is formally discouraged to execute two tests in parallel in the same user session, on the one hand to avoid any interference between the tests, on the other hand because this does not constitute a realistic functional use pattern. Even a rich functional scenario supposed to reproduce the actions of a user known for his multi-tasking capabilities will be written in sequential form and not in parallel.
It is possible to stack many user sessions on the same computer or on the same Windows server.
The user session allows the isolation of test execution tasks, but also the execution of several tests in parallel at the rate of one test per session.
Massive use of Windows sessions
Under Windows Server for example, the massive use of Windows sessions requires an activated OS with its own license, but also additional licenses such as User CALs, and if the "Remote Desktop" protocol is used, RDS CALs.
Nowadays, the most common method to use many user sessions is to use a remote control protocol such as RDP or Citrix.
The virtual graphics card managed by RDP is RDPUDD Chained DD
The graphical adapter used is then virtual, and managed by the remote control system. There is no relation between the graphics card of the server used and the number of concurrent sessions at the maximum.
Security and login
Need for an open session...
Microsoft Windows security rules do not allow applications to be run on a session that is not open, either through the Task Scheduler or another scheduled or remotely triggered execution system. The Windows session must be open for the test to run.
...but open session does not mean without authentication!
This being the case, it is possible to run tests on an open Windows session, but locked with the authentication request, most often by password.
To lock a local session, one can click on the start menu, then on the user, then on Lock.
To lock a remote session, the same operation is possible (see screenshot below) but closing the connection window is enough. The proof: whether locking the session knowingly or reconnecting via RDP after closing the connection, authentication will be requested.
Once we have configured a session to open automatically or through the connection of a user by a protocol such as RDP, several possibilities are available :
- An automatic login can be configured at server startup, while keeping a password lockout banner. It is important to differentiate between "open session" and "password required". Moreover, an open and visible session without a password is not a problem if no keyboard-mouse screen is connected and access to the session by the remote desktop protocol (RDP, Citrix, ...) requires a password.
- an on-demand login can be performed by scripting the connection request via remote desktop protocol. You can then configure a script that runs at the start of the session and that queries a remote execution service to know if it should launch a test job or not.
Obviously, a strong authentication is required to protect access to sessions by all users. Again, automatic logon does not mean that a password will not be required. The flip side is true: just because your session is closed doesn't mean it's secure, especially if the remote access protocol is accessible over a public IP and the password is weak!
The continuous integration platform: Jenkins
We've covered the environment for running an Agilitest test and the prerequisites.
But while it's possible to run a test in isolation on the DOS or PowerShell command line, or even via the Windows Task Scheduler, that's not the point.
In fact, we recommend scheduling test runs through the Jenkins continuous integration system, which is well known and widely used in the industry. Agilitest is natively supporting Jenkins, both by deploying and installing it on the tester's computer, or by connecting to an existing running instance.
More importantly, Agilitest is capable of communicating with Jenkins to create new Jobs (or items).
If the test execution needs are massive, then the Jenkins server will not execute any tests and will delegate them to the execution servers via Jenkins Master-Slave communication through the Jenkins agent.
To do this, Jenkins allows the use of different communication protocols such as Java Web Start (JNLP) and SSH.
As you can see, most of the considerations in this article are general, not specific to Agilitest. In fact, our approach has always been to offer automation software that is part of a TestOps process, itself composed of interoperable software bricks and the best in their field. It's what we call a "best of breed" implementation.
The Windows session that is the key architecture component provided by Microsoft to parallelize tests within a dedicated infrastructure.
Jenkins is the component that allows you to schedule executions while providing access to campaign reports and tracking the success and failure rate for a given job, even if ATS is compatible with TestNg and can be deployed on any CI/CD system.