A tremendous number of things in mobile app development and testing are metric-driven. And especially when it comes to mobile app testing, metrics are highly valuable indicators of how well does the app perform, what is the general bug rate, and how many things have been exposed from the code so far. Metrics also reveal how well test automation and infrastructure used around it can expose issues in the code and how quickly and easily developers can fix those.
In this blog, I’ll describe one great example, a real-life story, of how one of our customer uses a great number of identical mobile devices, with the very same OS version, with the same OEM customization setup, to hammer their builds of their mobile app hundreds of times per day, and how each build are getting tested on those devices every time.
The topic of this blog sounds a bit unbelievable, but there is a strong fact and message here of how test cycles can be pushed down, in the context of time. When looking at improving anything by 98% it may sound unbelievable and unrealistic. But the fact is that not all tests are done in consecutive manner: a test case followed by another test case, followed by third test case, and so on.
Some unit tests, for instance, can be break-down to non-consecutive test pattern – a test pattern where each and every test case focus on different aspect of the functionality, regression, performance, or some other aspect of the behavior.
98% Shorter Test Cycle Enables Higher R&D Productivity
First of all, making test cycles gives enormous boost for productivity and efficiency of R&D. Test automation has been widely accepted and adopted to remove repetitive tasks across the entire software development lifecycle (SDLC) along with the smart use of continuous integration, continuous delivery and continuous deployment.
As a result of this sort of DevOps mindset, many organizations have managed to speed up their app development and testing cycles, and just by getting those application developers and operation specialists to collaborate and communicate throughout the end-to-end process.
How can mobile test automation give another significant boost to mobile app development lifecycle to shrink the used time on testing to minimum? The answer is the use of identical mobile devices – and many of those – together with suitable tests. For example, shrinking down 100 tests duration of 1 minute per test to 2 minutes requires more hardware (devices). Let’s drill deeper with this example.
100 tests that take 1 minute of each takes 100 minutes for entire device test cycle. When you have e.g. 50 identical devices in use, and you can deploy that 100 test case pattern on those devices so that each device will only take 2 test cases for execution. In timely manner, you get the very same test pattern executed, results back to your continuous integration tool, screenshots, logs and everything else that enables QA folks to verify the correct functionality and if needed, engineers / app developers to fix any issues.
“To save time and to build better quality apps, divide your test cases on different devices. This enables you to run enormous number of tests, every day. Instead of 1 or 2 devices, use tens or even hundreds.”
This is a real example from one company that actually has over 100,000 test runs EVERY DAY in their internal device lab. This sort of improvement in R&D efficiency has been a game-changer and has reduced all important and inspected test-driven metrics from their development process: bugs, time-to-fix, as well as bugs found per build.
Higher R&D Productivity Enables Frequent Builds (and Releases)
It’s been said that the properly adopted DevOps approach and practices around your organization will help company to produce more frequent releases. Many years ago, it was quite common that app developers tested their apps only close to the release – and then hoped that everything will go fine and whatever problems they’ll get from end-users, they will fixed after release.
Those days are gone and app analytics won’t provide you those important metrics of how well app does across users devices. It’s already too late to start fixing issues with the app and hope that in timely manner app will get fixed and those people who voted 1 star for your app will bounce back and revote. Missing Google’s and Apple’s ‘Featuring’ or ‘Trending’ lists will make the app or game irrelevant against competitors.
This is harsh fact for indie developers, but the very same applies for companies building consumer-facing apps. If the app has any issues with robustness or if users of it think it has quality issues of any sort, they will not use it. And ratings in app markets go inline with the experience users got from that app.
Therefore, it’s crucial to do lots of releases while the app is under development, and test every build on real devices. The significant value-add that frequent builds and testing of those builds brings into development process with more effective and result-driven development will speed up the entire process and makes it possible to publish earlier. The time-to-market will be shorter when testing doesn’t hold it back but is seamlessly integrated with development and provides invaluable results during the whole development lifecycle. These improvements can be measured in three terms of value.
Frequent Builds Equal as Faster Releases and Better Quality Apps
More you test, more you learn. And when you learn what is wrong with the app, you can fix those for your next build. This has been noticed to have the direct impact on app quality.
Frequent builds and enhanced testing process offers various benefits to everyone. Developers will find it easy to integrate with, get instant feedback about their code regressions and can jump on details as logs combined with appropriate screenshots of defect provide all essential information. No need to further document defects nor add humans to confuse things – logs can simply explain where the problem is.