The old rule-of-thumb with bug-fixing is simple: Fixing a bug today costs less than fixing that very same bug tomorrow.
Regardless from which point of view you or your customers/end-users are taking, bugs are evil and at the very core of why your mobile application or game doesn’t get those 100 million downloads from Google Play or App Store.
To get the most out of your testing efforts, the selection of the most robust and often cross-platform automated testing method is truly the best way to ensure maximal test coverage, on time and with great results. It’s a well-known fact that automated testing methods can be used for both validating requirements and reducing the costs of testing through automated test case generation. However, the full automatization of large software entities also has a cost that many companies haven’t been ready to pay for. Historically, one of the reasons is the recurring concern of the lack of adequate integration with well-established development life cycles.
In our third post in our series of Deconstructing Mobile Testing Methods, we’re looking into benefits that automated testing and continuous integration can give you in your daily work. And, not only benefits, but what existing products and features can bring to you or your team to help you to easily get up and running with automated testing.
Use Cases – Who Needs Automated Testing? And for what?
Use cases are known to be one of the best bets for any sort of testing. In the past, for some reason, if you wanted to automate the test generation process, there was a large gap in bridging high-level use cases and concrete tests that hammer your software. So with that in mind, here’s a few interesting testing angles to keep in mind when you’re planning the testing methods (and tools) for your awesome mobile application, game or service:
1. Functional automated testing – enhance quality assurance
Functional testing typically involves several steps to describe what the system does: from the identification of functions that software is expected to perform, creation of input data and output based on specification, the actual test case execution and comparison of actual and expected outputs. In order to automate functional testing, it’s highly recommended to adopt professional tools and run test cases again real Android and iOS devices.
2. Performance and stress tests – improve responsiveness of your app
Want to determine how well your application responds to various levels of usage? Then use a stress test – users should create tests which last for hours so they can observe how well the application, as well as device/server combinations, behave during long usage. To determine if your app can run successfully for a sustained duration under heavy load, stress tests can be used. Performance tests measure how responsive your application can be on a real device.
3. Stock or pre-market devices – global access to real devices
Today, the beauty of automated testing for mobile apps is you don’t even need to own or possess the physical device you’re testing your application on. For example, some pre-market devices are located in Hong Kong and CDMA mobile devices are in the US. But you can still have access to these devices, conduct your instrumentation tests and receive results in minutes. Physical location, as well as physical devices are becoming irrelevant for automated testing.
4. Regression testing — uncover new software bugs
Most of the mobile services today have a server-client interaction, requiring a login, uploading of data (e.g. pictures) and downloading of data (e.g. news, data, images). If you’re developing these kinds of services, users need to ensure all their changes are done in code, to either server side or client side do not break the functionality of the service. Automated testing is really the only option in this case, manual doesn’t cut it.
5. Global & Local (Glocal) comprehensive — make sure your app is understood
Testing becomes highly important when your application is targeted for the global markets. The word ‘global’ means your app or game needs to be ‘local’ for everyone. So let’s call it Glocal – be both! When your app needs to be translated and tested with devices in multiple languages, these types of tests are easily performed automatically to change the language of devices, and even do it again with the same test runs. The problems layouts can be easily picked from screenshots.
Keep in mind testing frameworks, such as Robotium, uiautomator, Selenium, Calabash, etc., are just methods. Some of those are better suited for native app testing, some better for web and hybrid app testing, but to get the most out of these frameworks and methods, you need professional testing tools to fully utilize benefits that these frameworks have to offer.
And now, a little word about that….
Continuous Integration – the most effective with the right professional tools
The most popular automated testing environment for mobile apps and games have been Testdroid Enterprise, which provides its users an in-house server product and a private device cloud, to enable developers and testers to run their tests in their own test lab. Testdroid Enterprise can be – and typically is – used as a part of continuous integration environment that enables test execution to start automatically after every code check-in.
Testdroid integrates seamlessly with Jenkins, which is currently the leading open source continuous integration (CI) server monitoring executions of repeated jobs, such as building a software project or jobs run by cron. Current Jenkins focuses on two jobs:
- Building/testing software projects continuously. In a nutshell, Jenkins provides an easy-to-use continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build increases the productivity.
- Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those that are run on a remote machine. For example, with cron, all you receive is regular e-mails that capture the output, and it is up to you to look at them diligently and notice when it broke. Jenkins keeps those outputs and makes it easy for you to notice when something is wrong.
Despite a bit old and focused on Hudson (Jenkins’ predecessor), you can find one great blog about how to create automated UI testing for Android applications here.
Costs of Automated Testing – guaranteed return-on-investment
How do you produce high-quality, robust and reliable software with increasing complexity with technology under massive competitive pressure? Automated testing software is the solution. Automated software testing improves your business results in three ways:
- Increased test efficiency
- Increased test effectiveness
- Reduced time to market
So what is testing efficiency versus testing effectiveness? Simply put, testing efficiency is just the average number of tests you can run in an hour of tester time. So logically, higher testing efficiency drives down product development time and costs, improving your bottom line.
Testing effectiveness is the rate at which your testing technology reveals bugs before your systems are released. So, increased effectiveness reduces costs through a better product, improving your bottom line now and your top line later by building customer satisfaction and loyalty.
Improve your bottom line + building customer satisfaction + reduced development costs equal automated testing – it can improve your top line.
Isn’t that what we all want?
What are our conclusions for you?
Testing, we believe, is a crucial activity to ensure success in highly competitive landscape of mobile apps and games. But “poorly” planned testing can be 20-50% of the total development effort, and in that case, it’d also be the biggest single cost in your budget. In order to ensure that testing can be done extremely efficiently, using agile continuous integration with direct access to more than 230 real devices but not leaving anything untested, Testdroid is the only, smart option available.
See you again next week!