Agilitest is able to generate an XML file containing all the important information about the test process, as well as the successive images of the different actions performed. We have chosen to rely on an open format (.xml) and a document generation engine under free license (FOP).

These .xml files can be used to produce all types of reports:

1. Execution reports:

              · Execution report: for a global reading of tests execution

              · Detailed execution report (with or without screenshot): to analyse the reasons for failed tests

                 ◦ Summary report (HTML): for a global reading of test success in a specialized format

                 ◦ ATSV report: Execution images compiled in a video format (played via the ATSV player)

‍

        2. Validation report: to document and to archive the implemented executions

‍

Below is a scheme on how to use the reports according to the needs ot the teams and the way the organisation of the information can be used (plus, an animation video of the scheme https://youtu.be/2UdsYINPk_s)

‍

Agilitest utilizes the open-source ATS (ActionTestScript) engine to execute automated tests. Upon completion of a test run, the system generates a comprehensive suite of visual and technical reports. these are designed to help teams analyze results, troubleshoot failures, and provide quality metrics to stakeholders.

The main HTML report (ats-report.html)

The ats-report.html file serves as the central hub for viewing test campaign results.

Key Interface Components:

• Dashboard: Provides a high-level overview using pie charts to show the number of passed, failed, or blocked tests.
• Script Details: Lists every executed ATS script with its final status. Clicking a script expands the view to show individual steps.
• Action Log: A chronological breakdown of every action (clicks, text input, assertions). Each entry includes:
  • Execution duration.
  • Status (Success/Failure).
  • Associated screenshots (if enabled).
• Visual Media: Integrated screenshots and videos for debugging. Since 2023, the report includes an automatic HTML player for .atsv (ATS Video) files.

‍

Output file structure

Reports are typically generated in the project's /target/ folder following a build or execution.

‍

‍

Advanced feature (ats-core)

Leveraging the ats-core engine allows for more technical reporting capabilities:

• Stability Analysis: Tracks "flaky" tests by comparing current results against historical data.
• Assisted Diagnostics: Analyzes the context of a failure to determine if it is a performance issue (timeout), a network error, or a genuine application bug.
• Report Customization: Users can brand their reports by adding a watermark.png in the src/assets/resources/reports/images directory, which will appear on all generated screenshots.

‍

CI/CD & Devops integration

Agilitest reports are designed to fit into modern automation pipelines:

• XML Export: The Surefire-format XML files are the industry standard for integration with Jenkins, Azure DevOps, and GitLab CI.
• Test Management: Results can be automatically pushed to test management tools like Squash TM or Xray to maintain a centralized "source of truth" for quality assurance.

‍

Configurations

Reports generation

It is possible to configure the information that you want to see appear in your reports (execution or validation) from the project constructor by checking the “Generate report” case or from your Jenkins integration. You will be able to choose to generate execution reports or a validation report.

‍

In side bar in the editor, click on “Executions management”

‍

Then by click on the case “Generate report” in the project’s builder. This will give you access to the “Report detail level” drop-down menu as well as the possibilty to check the “Generate validation report” case.

‍

Reports storage

At the end of the execution, the reports will be stored in one of these default files:

Report generated via Agilitest;

• Java : <Agilitest_project_file>\<project_name>\target\ats-output
• Maven : <Agilitest_project_file>\<project_name>\target\surefire-reports
• Or placed according to the user choice

Report generated via Jenkins integration:

• Jenkins : <jenkins_job_folder>\builds\<numero_de_build>\htmlreports\<nom_du_publisher>\

‍

Execution reports

The level of information in the execution reports can be chosen in Agilitest by using the drop-down menu "Report detail level".

The execution reports provide three level of information from the project’s builder in the Agilitest editor.

‍

Each report level has the following headers:

To the top right you have access to:

• Dashboard: which is the landing page and the high-level summary of your test execution
• Summary report (summary.html): which is the high-performance, bird's-eye view designed for large-scale test campaigns
• Overview: which is the Executive summary, the primary landing page or default view of the ats-report.html file.

‍

Execution report

The execution report provides a global reading of tests execution.

‍

Detailed execution report (with or without screenshots)

The detailed execution report provides a better understanding of the tests design and of potential bugs.

‍

For each playlists it indicates:

• The modifier
• The last time it was modified
• When the playlist started being executed and when it ended
• Test and external ID
• Links to the ATSV file and the HTML file
• A test description
• Various graphs (executions history, actions durations, actions type)

‍

For each actions (with screenshots) it indicates:

‍

• Action type
• External links
• Details of the action
• Execution time
• The status of the action’s execution (passed or failed)
• Screenshot of the action (if the option was selected)

‍

‍

A purple insert corresponding to the action will appear in the image when an executed action corresponds to an user interaction (example : Mouse actions, text input, etc…).

‍

When an action is failed or is not validated but does not prevent the test’ success, a explanation of the error will appear in the information part of the action.

‍

The video report: ATSV

The ATSV_HTML file can be accessed directly from the execution reports and corresponds to a video format of the report presented as a set of compiled images accompanied by animations of the executed actions.

‍

Summary report

The summary report is generated directly from the execution report in a HTML format. It is consulted by managers and the teams project to have a report of the executed tests’ success in a specialized format. It corresponds to a summary by execution Suite and by test in order to have a global status of the software quality assurance.

Directly from the Execution report, click on the Summary link to generate the HTML file (at the top right of the Execution report)

‍

The display of your summary reports can be customized with project report's images.

To access it, open your project properties by clicking on menu located in the side bar of the editor.

Then choose the images you want to display in your reports.

‍

‍

The validation report

The validation report is generated by checking the box "Generate validation report" in the project’s builder:

‍

The validation report is designed to be a proof of execution to be saved or archived.

It includes the same information as the detailed execution report with clickable screenshot, except that:

• It has no link,
• There are all actions (execution time, action execution status) except these: no ActionChannel as well as ActionProperty and ActionComment,
• In the information part, there is no "error" part and in the "element" part, we find the Tag and Criteria but we do not find the number of elements found, the search time, the location, the click position.