Honestly, aren’t you struggling with your test automation strategy? Well, you’re not alone. Every renowned research from Capgemini to Gartner show it: most companies – small and big – still face significant challenges in implementing a test automation strategy. OK, but why does it feel so daunting at times? Our friends over at Abstracta break it down in pieces in this guest post.
What to Know Before Getting Started with Test Automation
By: Kalei White, CMO at Abstracta
By now, you have probably heard through the grapevine (or over a loudspeaker) the benefits of a good test automation strategy for any software or mobile app development project. Your team may be shifting left testing and/or moving to continuous integration or continuous delivery (CI/CD) environment—both of which rely on test automation to function properly.
Teams that try to implement a test automation strategy without first having certain things clear will find that their efforts, while brave, may quickly turn south. Because, If you don’t set up these tests the right way, you will end up with the same poor results (just delivered a whole lot faster!). Before you get started with test automation, make sure to take into account the following considerations that my colleagues and I at Abstracta have noted while helping various clients of ours to get it off the ground and running strong.
1. Set Realistic Expectations
You can, indeed, expect your automated tests to save your team time, resources, and the headache of having to do everything manually. However, you cannot completely rely on automation when it comes to software testing as not everything can be automated. When you automate testing, you are really just having a computer program check the system in ways that were predetermined ahead of time but, humans are still needed to do the non-automated aspects of testing. Also, know that you must have your manual testing down pat so that you know what you’re automating in the first place. Walk, then run!
It’s important to remember that the value of a test comes from the information that it uncovers about software quality, not from the sheer number of tests executed, nor the frequency with which they’re run. What’s essential is to obtain the right information that will be shared with stakeholders upon which they can make critical business decisions. So, don’t expect that by automating everything, all of your problems will be solved.
Additionally, expect ahead of time that after setting up your test cases, there will still be a considerable amount of man-hours required to maintain scripts. Automation is hardly a “set it and forget it” way to test.
2. Make Sure Everyone’s Onboard
For any project to be successful, it needs the full support of all of those involved. Maybe, instead of your manager being the one to insist on implementing test automation, it’s the other way around. Perhaps you need some help to get the team and your leaders to realize the value of test automation. You might also need to get your developers on board with getting in the habit of automating unit tests, as those will give your team the greatest bang for your buck.
If that’s the case, you can present some information about the possible return on investment of test automation. In his breakdown, Paul Grossman calculates that a team of five senior and five entry-level testers could hypothetically reduce the cost of testing from $78/hour to just $17.58/hour and increase testing from 1,350 hours per month to 5,985, gaining up to $315,000 worth of testing via automation. Not bad!
Remember to be very transparent with your team and make sure all share the same realistic expectations that you set for the automation project, as mentioned earlier. Don’t mislead them by asserting that automation doesn’t require much effort and resources upfront, because it certainly will.
3. Know What’s Worth Automating
Now that we know that not everything can and should be automated, that begs the question, what should we automate?
Automation is particularly useful when your software has accumulated a lot of technical debt, when tests are very time consuming, and when you are working with several complexities, in a long-term context.
Thus, most teams start out by automating regression tests, given the situation in which they already have a defined test suite that must be executed periodically after each product release. In this case, the effort to run them manually becomes repetitive and takes time away from other tasks that are not as easily automatable, yet highly valuable, like running performance tests.
4. Pick the Right Automation Strategy
More important than any tool you may choose to automate with, it’s important to follow a well-thought-out strategy to guide your automation. There are two that I’ll mention here: the test automation pyramid and risk-based testing.
The Test Automation Pyramid
Made popular by Mike Cohn, the agile test automation pyramid improves the ROI of automation and sets a guideline for receiving the most benefits from it.
According to the pyramid, most testing should take place in the development stage, with developers running their own unit tests after every build. These tests are the easiest, cheapest, and fastest to complete. Here, any existing bugs will have the shortest life span, as developers can find and remove them almost immediately. After running unit tests and all of them pass, the pyramid suggests moving onto the API/integration/component testing phase.
This is where the logic and business processes are tested without having to go through the UI. Compared to testing on the UI level, here you’ll have fewer problems, easier maintenance, and faster test execution. Lastly and least often, automate some UI tests. Run as few of these (just the most critical test cases) as possible since they’re the most costly, time-consuming and brittle. Be careful with these tests as they are the most likely to provide false positives and negatives. After reaching the top of the pyramid, completing the UI tests, manual and exploratory testing can be conducted.
Along with the test automation pyramid, it’s a good idea to consider risk-based testing. This test strategy places a higher priority on testing elements of the system that are the most at risk of failing and whose failures would be most devastating to the business.
To get started with this strategy, it’s paramount to run a risk analysis to decide which test cases to automate, taking into account different factors. For categorizing tests by priority, a widely used method is MoSCoW, which is an acronym for Must, Should, Could and Won’t (and yes, there are test cases to which you will say, “No, I won’t automate that”). Once the priority of the tests has been established, it’s advisable to check them every once in a while, given that the business or client requirements might change, which leads me to my last point…
5. Review Your Test Code Often
Test code is still code, and so, it must be tested! With automation comes the risk of encountering false positives and false negatives which may or may not sound the right alarms to inform you of critical errors in your software. When they happen too often, whole teams lose confidence in the automated test cases which means they’re a waste of time.
If you only focus on creating test cases without frequently analyzing their value and aligning them with what the business needs in each moment, then your efforts will be futile. It’s also very important to make sure you have the right test assertions, which are the conditions you want to check against. I hope this has helped you to prepare for putting a test automation framework into place! Are there any other considerations that you think are important? Let me know!