The name of the game in today’s competitive mobile application landscape is a speed combined with robustness. The ‘speed’ means time-to-market and how early developers can publish updates to their applications. It is also a known-fact that the first ‘robust’ application in its own category at Google Play tends to stay on the top of download rankings – that will then expose app to hundreds of millions of mobile consumers. Those later entrants, competitors to this first application in market place are always challengers. What should they do to get attraction for their application?
Software failures in their mobile application can cost their reputation and also push their application down in a search order. It is very important to remember that only those 5-star ratings will keep the application in front of hundreds of millions of downloaders.
Well, that is just the part of the story. Many mobile application developers face economic pressure as they can’t afford (manual testing) resources and there is an imminent need to reduce overall costs. This blog will focus on this theme and scratches the surface in explaining why the test automation actually saves you money, time and resources – when implemented and adopted efficiently.
Background to Automated Software Testing with Mobile Application
For starters, automated testing provides various benefits to developers: coverage to detect bugs and errors – early and later during the development – and significantly reduces the cost of failure, saves time through its repeatability and earlier verification, and leverages the improved resource productivity. Implementing the test automation for your mobile application development process is the best way to gain these benefits and migrate your development to effectively use of resources and time.
As applications under development constantly change and become more complex, the feature set in those keeps also growing and need for test cases should increase with the same tendency. However, in case of so many applications, this is not the case. Just one new critical feature added in your application and all features should be tested to ensure bullet proof behavior and user experience. And this is the most important reason why manual testing can’t keep up. Automation – when built properly and adopted professionally – will solve this for developers.
However, if the automated tests are not implemented to be maintainable when the application changes, it is going to be trickier for testers. If those tests have to be rewritten or significant modifications are needed, users will keep starting over instead of building on prior efforts. Therefore, it is essential to adopt an approach that supports maintainability over the life of the application.
Manual Testing vs. Test Automation – What things to consider?
Automating tests for your mobile application is the best way to increase the effectiveness, efficiency and coverage of your software testing. The true benefit from test automation comes from repeatable nature of those tests, but also from test execution that probably could not even performed manually. For example, by generating test cases, you could yield many more – when none of those might be possible with manual resources. Let’s look at some things to consider when test automation provides you significant value.
1. Costs and Assets – and Time to Market
The test automation for mobile has been considered a critical factor for big mobile app companies but for some reason it is often thought to be too expensive or difficult to adopt for smaller companies. Probably due to historical reasons the cost factor is the first consideration when deciding whether company wants to use automation or stick with manual testing in their development projects. Regardless of if you select manual and automated testing, you’ll need the following assets and resources – and those will cost you money: Time, People, Infrastructure, Tools, and Training.
Depending on the size of the project and application it is quite obvious that test automation will provide you a return on investment (ROI) – how big is the ROI entirely depends on how test automation is adopted and executed in your organization. For example, once those test cases are created, automated tests can be run over and over again at no additional cost and those are rapidly completed than any manual tests. Automated software testing can reduce the time to run repetitive tests from weeks to hours. This is a significant time-saving that translates directly into cost savings.
In average mobile application development project, the costs for manual testing continue to rise when new build is done but the same doesn’t apply that significantly to automated tests. Acquisition costs are typically higher in test automation but those will quickly pay off and increase the ROI.
2. App Development and Automated Testing Cycle – Always together
Increasing the efficiency and productivity with test automation in your application project actually starts with a new mindset. Software tests have to be repeated often during all development cycles to ensure the best possible quality of the application. Every time source code is modified software tests should be repeated. For each release of the software it should be tested on all supported variants of an operating systems and different hardware configurations. Manually repeating these tests is costly and time consuming. For example, running well covering tests on all different variants of Android and actual devices would take a lot of time when done manually.
Test automation is an easy task once adopted from the beginning and it will save you time and money while the development goes on. Determining the number of relevant test cases and releases will also indicate how well will test automation work for your project.
In some projects, it actually makes sense to start with manual testing, especially in early of the development process. This will offer developers a good opportunity to refine their test strategy, set test requirements and done planning on how testing will be done during the development. Later in process when testing processes are standardizes all different types of automated testing will start delivering benefits.
Humans are not infallible. We’re not – and never will be as accurate and precise as machines – and generated test cases. Even the most conscientious tester will do basic mistakes during manual testing. Automated tests perform the same steps precisely every time they are executed and never forget any details.
3. Tools, Test Automation Frameworks and Test Cases
To get the most out of your testing efforts, the selection of the most robust and often cross-platform automated testing method is truly the best way to ensure maximal test coverage, on time and with great results. It’s a well-known fact that automated testing methods can be used for both validating requirements and reducing the costs of testing through automated test case generation. However, the full automatization of large software entities also has a cost that many companies haven’t been ready to pay for. Historically, one of the reasons is the recurring concern of the lack of adequate integration with well-established development life cycles.
Keep in mind that testing frameworks, such as Robotium, uiautomator, Selenium, Calabash, Appium etc. are just methods. Some of those are better suited for native app testing, some better for web and hybrid app testing, but to get the most out of these frameworks and methods, you need professional testing tools and fully utilize benefits that these frameworks have to offer.
Also, one essential tool for mobile developers to use in order to automate their testing effort is Jenkins, which is currently the leading open source continuous integration (CI) server monitoring executions of repeated jobs, such as building a software project or jobs run by cron. Current Jenkins focuses on two jobs:
- Building/testing software projects continuously. In a nutshell, Jenkins provides an easy-to-use continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build increases the productivity.
- Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those that are run on a remote machine. For example, with cron, all you receive is regular e-mails that capture the output, and it is up to you to look at them diligently and notice when it broke. Jenkins keeps those outputs and makes it easy for you to notice when something is wrong.
4. Consider Test Coverage, Reusability & Open standards
Open standards means no vendor lock-in. Automated testing can increase the depth and scope of tests and help significantly to improve software quality. Lengthy and thorough tests – that are often avoided during manual testing – can be run automatically. Ideally, test cases should have full access inside an application and test all aspects of it: memory contents, data tables, file contents, and internal program states to determine if the product is behaving as expected. Automated tests can easily execute thousands of different complex test cases during every test run providing a coverage that is simply impossible with manual testing. Humans freed from repetitive manual tests have more time to create new automated test cases and build more compelling features (and test cases).
Test coverage can be extended with the number of different types of test cases. For example, generating and recording your tests can be done without any knowledge of C, C++, Java or any other programming language. Also, choosing the right device set (not emulators) for testing is a crucial decision and depends on global market that you are reaching with your application. For example, the most popular Android devices on volumes are typically a good pick but tester should also consider platform/OS version, display resolution, memory size of those devices, and even hardware differencies (chipset, set of sensors and so on).
One important thing to consider here is that developers and testers should stick with open standards and only test cases relying on open, transparent APIs should be used for testing. This will help in reusing test cases later on.
5. Learning and Improving effectiveness, and getting there sooner
In a nutshell, a professional automated testing software can be seen as a solution to a common problem: how to produce high-quality, robust and reliable software with ever-growing complexity with technology under massive competitive pressure. In fact, automated software testing provides a cost-effective solution to this problem. Automated software testing improves business results in three ways:
1. increased test efficiency,
2. increased test effectiveness, and
3. shorted time to market.
Testing efficiency is the average number of tests you can run for an hour of tester time. Higher testing efficiency drives down product development time and costs, improving your bottom line.
Testing effectiveness is the rate at which your testing technology reveals bugs before your systems are released. Increased effectiveness reduces costs through a better product, improving your bottom line now and your top line later by building customer satisfaction and loyalty.
If your company sells software products that must be tested before release, reducing your time to market can yield enormous benefits. Replacing manual testing with automated testing can cut weeks or months from elapsed testing time, improving your top line.
In addition, from our experience test automation improves also team spirit and when testing is automated testers and developers have more time to improve their application. And that is the best way to get attraction to their app – as robust and compelling.
Summary – How to Get in Front of Hundreds of Millions of Users
The mobile application ecosystem is very dynamic, but also suffers from both software and hardware fragmentation. OEMs are launching new devices and new customizations that can break your application, and new OS versions are also delivered every now and then, so it is important to remember that your application needs to be updated when something happens on these ecosystems. To meet the market and device compatibility it is recommended to revisit and test you application against these devices then. And for this automated testing has you covered – you have already those test cases/scripts/code that is usable again. Without zero investment costs or manual intervention.
Many of software testing tasks can be laborious and time consuming to do manually. In addition, a manual approach is not always effective in finding certain classes of defects. Test automation offers a possibility to perform these types of testing effectively. Once tests have been automated, they can be run quickly and repeatedly. Many times, this can be a cost effective method for regression testing of software products that have a long maintenance life.
Once automated tests are created they can easily be repeated and they can be extended to perform tasks impossible with manual testing. Because of this, savvy managers have found that automated software testing is an essential component of successful development projects.
Some food for thought:
- Test automation requires knowledge, skills and new mindset – but it will surely pay off!
- Integrating the test automation into overall development lifecycle is a great way to start
- Time spent on a test automation creation will guarantee great return on investment
- Creating and adopting test automation for your mobile app is not difficult – take a look at Testdroid Cloud for starters
Good luck with your testing! And don’t forget to come back for the new blog on 23 October!