Why maintenance of test automation is complex?
Although maintenance of test automation is a complex deliverable from the aspects of the resource-time-cost-effort intensive areas, it has grown as a pivotal task. No matter how innovative the idea is, splendid solutions don’t just emerge overnight by chance. A share of high operating costs comes in software projects of automation code maintenance. For example, the code involves regular maintenance in the sense of:
- Keeping the test code in testable condition for the next sprints,
- Fixing or replacing automated tests,
- Maintainability of environments,
- Maintaining the tools used in test automation code
- Building & maintaining infrastructures for any tests suites, and the automated test suites or cases themselves,
- Supporting and adding more utilities in the test automation when needed,
- Easier setup and update, and installations of the library used in test automation,
- Removing the dead code
- Updating all to keep up with traceability
- Constant knowledge sharing and many more.
While I am observing teams doing all these tasks for the sake of test code maintenance, regression testing is still so valuable for the existing features to release a new version of the software!
The value of maintaining test automation code includes:
- Team collaboration,
- Project stakeholders’ education,
- Team responsibility,
- Having more confidence in the release version of products
- Team members to adopt the new skill set,
- Good software development practices,
- Environment for continuous improvement.
- Keeping/sharing knowledge in the team
- On the whole, the process becomes more agile to adapt quickly to software changes,
- Constant feedback and improving as a team.
This brings joy to the journey of test code maintenance!
How to maintain the test automation code?
Now that we have seen the challenges and values of test automation, let me share with you a couple of aspects of how we did it in one of our projects to maintain the test automation code.
- Identify whether you need corrective maintenance or preventive maintenance for the test automation code.
- Have the timeline set for the activities, and constantly check on your progress for test automation code.
Identify the risk while you are maintaining the test automation code so that we do not sacrifice the release of a new version of the software.
- Having the test automation strategy is very important for the team to think about the maintainability of test automation. For example, not automating all tests or automating the test at targeted layers.
As an example, if this, I recommend this talk about test automation strategy by Richard Bradshaw’s Pyramids Are Ancient - Test Automation Strategy, Richard Bradshaw - Ministry of Testing | Sauce Lab
- Having good testability (as per my understanding of the terminology means, the ability to test with ease) is a must for the team that thinks about maintaining good test automation. “This is because testability is the foundation of all the testing efforts. If the software teams can test the piece of software easily, they can, in turn, determine the ability to automate and maintain the automated piece of code with ease.”
10-ps-of-testability: I like how Rob Meaney’s and Ash Winter’s shared what they did for their team to improve the testability from various perspectives and came up with P’s (P are referred to as MNEMONICS here), and how they explained it (public talks, books, blogs, etc).
- Gaining an understanding of testing debt (as per my understanding of the terminology means, a huge backlog for the test), and working together with team members to prioritize and solve the issue. For example, a situation where a huge test automation task is pending for the development of test automation code, and regression tests become a pain point for the team. The result is a slowdown in the software development process.
- Having a way to ease the process so that the entire team can contribute, which includes developers and business stakeholders. For example, developers can help with writing automation code and help review testers’ automation code, while business stakeholders. can improve the testing coverage by providing areas requiring the attention of the software product.
- Having an opportunity where team members can take ownership in rotations. So that they can have valuable feedback, which could be used for the improvement of maintaining the software test automation code.
- Finding the patterns in the test execution failure of automation test code proves that the team has a flaky test in the test suite. Doing root cause analysis helps a lot to maintain the piece of test automation code.
- Constantly improving and using clean environments for running the test automation. The common successful practice is to use as much as possible the replica of the production environment.
- Having automated test cases independent from test environments (dev, test, UAT environments) from the perspective of test data maintenance. Test data maintenance is very important while maintaining the test automation code to run the tests in different environments. Nowadays, companies/projects have a branching strategy, and for test automation suites, the situation becomes crucial to run them independently and have the test result faster in different environments as per the business needs.
- Ideally, the team should develop the test automation framework in a way where it should be easier to add a test or remove automated tests. Nowadays, companies use different release strategies such as A/B testing in production, roll-out test, Canary Tests, Blue-Green Tests, Configuration Tests, Disaster recovery tests, and Statistical Tests. While maintaining the software test automation code, we should understand which method our team is using to adapt to such a situation.
- Having a good understanding of how to create and maintain the test data for the application will be more helpful in the maintenance of test automation code.
Logging & Reporting:
- Adding more logs in the test report so that any team members can understand the test steps directly from the test report instead of going through the test automation code. The point here is to understand where the test exactly failed. This activity will also help because they will also have a deeper understanding by seeing the exception thrown by the test automation framework in the test report. These will help a lot in doing further debugging in the maintenance of test code.
- Adding logs to the application under test will help to determine why automated tests are failing because the application under test is not working as expected. This is just by seeing the logs around the test reports or having access to the filter/search in the logging system by searching for the exception thrown by the application.
- Sharing the test report constantly with the team, so that the team does not leave the track of or lose interest in the maintenance of test automation code.
- Write simple and understandable test code because maintaining another person’s code can be difficult and time-consuming.
- Code review of test automation code is highly crucial.
- In UI tests, the tools recommended the use of locators should be minimum and more on ID and CSS.
- The tools recommended using a fluent wait state to avoid the stale element exception by Selenium.
- Minimize dependency in the test framework, so that it is easier to maintain the test code.
- Minimize dependency in the test framework so that is easier to update and maintain the dependency of the test automation framework.
Reflection & Actions :
- Lessons I have learned include having constant reflection on what you are contributing to test automation. Recognizing how you are contributing to test automation has a direct effect on your product under test. Learning to find the pattern of failures and improvement will make you a stronger tester.
- Post improving the test automation code based on every stakeholder’s expectation, and by having excellent test results. To keep up the same pace of test automation code, it is recommended practice to have certain rules for yourself, and team members, so that team members have a clear direction on what action to take, and by whom the action is recommended to be taken on the specific situation. Clearly defining ownership is very important for the team to collaborate for the success of the project.
For example, if the test pipeline’s job turns red, that is an alarm to the particular team and the team should analyze the failing test by either reporting the bug or fixing it. But the team should analyze within 6 hours of the time frame window.
Maintain a learning attitude:
- Constantly share knowledge within the team to let people be aware of what is happening in the team. If future releases require improvements to the automation suite, any team member can make updates as needed.
- Keep yourself updated to deliver success for the project/team/company.
- Seek help from the community. It could be inside the company & or outside the company.
Common mistakes to avoid while approaching test automation and test automation maintenance.
- Tools or test automation cannot fix any problem given if the attention to the software testing project is missing in the team.
- Having the mindset to automate all test cases is not the recommended practice, rather than following the test automation strategy, scope, necessary, cost, and time determined by the team/project.
- Owning the attitude that we should automate all bugs found in the different environments and test cases is not the recommended practice.
- Not following the Pesticide Paradox principle (which means not regularly updating the test cases/test suites, which were automated earlier point in time). In some situations, teams automate the workflow. In that case, the workflow should be updated regularly.
- Treating Selenium as the only tool for test automation, and implementing all the automated tests using selenium as a tool for all the layers like API and DB test is not a recommended practice. Ideally, the approach should be automated tests using the right set of tools, and the right layer will benefit the project.
- Not automating with the approach of how real users are going to use the software applications in real-time. For example, many software development teams automate test cases using user agents, emulators, simulators, headless browsers, etc. The real user will use none of those, so why is the team doing such activities just for having the shake of automation tests in place?
- Testers in the software development team have a knowledge gap about the test automation tools. So how would they contribute to such a situation to test code maintenance?
- Lack of ownership and time in the team to tackle the test automation and its maintenance.
- Using a test automation framework just because it is open-source is not a recommended way of working.
- Lack of coordination and communication among team members may affect test automation maintenance efforts.
- We should not use automation as a return on investment, which also includes even using the wrong metrics to measure the success of the project and automated test contribution.
- Not paying attention to the vicious cycle of red-green automated checks provided by any automated test result. This situation will not benefit the team by having automation tests in place.
- Not solving one problem or even having one goal at a time from the aspects of test code maintenance. Tackling everything at the same time jeopardizes the team.
- Using test automation to overcome distinct challenges in the team (such as a process), and without having the deeper insights of using the test automation for its benefits.
- Automating larger/complex tasks as opposed to automation. Automation is a time-taking task while the team can spend their effort solving simple problems.
- Focusing only on automation even when the project demands other testing techniques to be used in the project (such as exploratory testing)
- Insufficient assertion and validation in the existing automated test cases will not benefit the team for having any test automation in place.
- Having a huge knowledge gap about the software application which is being developed in the team, and still has the attitude of approaching automation and maintenance of the automated test cases. This situation can be beneficial for a short duration, but in the long term, this situation may do more harm.
- Not engaging all the stakeholders. Testers automate the checks in silos and waste the effort, time, and cost for the success of the project.
- Lack of effort towards the test automation maintenance in the software development team.
- Having too many expectations from test automation code. Such as, the automated tests would find all the bugs. No, it is not going to!
To summarize this part, I’d like to share this thought: “easy, simple, collaboration, growth mindset as in all things, context matters, and no two projects and people are the same”.
I pen down my thoughts with the motivational thoughts on “maintenance” which I read and got inspired.
'Building takes passion and energy. Maintenance is awful. It’s nothing but fatigue. Once you reach the top, maintaining that beast is awful, but it is crucially important.' - Urban Meyer. 'Similarly, maintenance of test automation code is Maintenance of automation code can be tedious, but it can be highly rewarding when you find your next release results in faster and higher quality testing activities'.
I would like to learn from you. Please share within the comment section your ways/journeys/challenges while approaching the maintenance of the test code?
Reviewer: Deep gratitude to JeanAnn for reviewing this piece of article, by giving your valuable advice and time.