We discussed about the required architecture and infrastructure for mobile game testing last week. In that article, I briefly covered those different types of testing that are required in order to build better and more robust games – that simply works on those devices that the actual gamers use. For this, it is essential to take a look at process how you generally build and test your game while it’s still going through different phases of the development.
Things that get done manually are hardly agile. Manual testing is not an exception – but it will significantly slow down the delivery if each and every change or new feature add-on have to be manually tested. So, how do the most successful game development teams do their game development and testing?
They do Agile – developers concentrating on delivery of new features, enhancements, bug fixes. The use of automation is in an epicenter of how developers already build their software and especially how this process can be significantly boosted, with automated testing included in the process and using real devices – the same ones that gamers use.
Why Testing Must Be Included in Day-to-Day Process?
The common approach, even today, with mobile game testing is that all of that important QA and testing work gets done at very late of the development process. Typically, companies are even dedicating specific time for this when QA, testers, beta testers do their work post-production. While testing is very important, nearly all of it can be included as part of the agile development process and every aspect of the game can be tested during the development – and during the production. Some certain foundations can be even tested (e.g. frameworks, platform setup) during pre-production, but the most important thing in order to improve the efficiency is to seamlessly integrate testing as part of the production phase.
The basic definition of continuous quality and delivery would be simply delivering the better games faster and do the majority of the testing while actual development is on going. As part of our research about mobile app robustness, we’ve also investigated how well the continuous integration, delivery and testing can improve the outcome of those games and result as a more robust end-product.
By comparing two similar game development projects, we’ve summarized that continuous testing using real devices can save up to 60% of overall development and testing time when testing is done as an integral part of the development. This has a direct impact on staffing, costs and faster time-to-market. And when things get fixed earlier, debugging time can go from months to hours.
Here is the typical case – when testing is not part of the actual development: In the first sprint you do the regular things and get the development going. Before the testing of even smaller parts or standalone features gets done, you simply don’t have visibility of the actual robustness of your mobile game. At some point, these bugs will become obvious and then stirs up again your sprint procedures and causes an halt of the overall development. In this example, developers spend much more time on firefighting issue that was part of the code for weeks or even months. Even the worse example would be that the bug gets revealed when the game out for users – e.g. as an user feedback or by using some other post-release analytic systems.
Developers should always spend more time on new features, incremental improvements, than fire fighting the old bugs and problems. Thing with bugs is that they escalate when noticed later on, and that will have a direct impact on developers daily tasks.
How to Add “Extra Days” for Agile Development?
Effective mobile game testing derives from a well-structured and systematic approach, use of test automation framework(s) and real mobile devices, and seamless integration with the agile development process. By adding an ‘extra time’ for the process – e.g. nights, when typically development work is not going on – you can easily dedicate this time for test automation. Written test scripts and automatic pre-processing of test data (logs, screenshots, results) will provide a report of how well the recent regression worked, what were the exact problems and even suggestions on how to fix those issues will be part of the deal with test automation.
After the package (APK or IPA) is built, it can be automatically send for the first test – the smoke test. The smoke test can be also seen as a compatibility test when it will be executed on all possible device variants. Also, each regression can be tested with the existing test cases/scripts for certain specific function. For example, if you mobile game relies on back-end service, this connection can be after each build regression tested with automated scripts. In addition, many of aspects can be instantly automated: localization (testing different languages), different types of performance tests (load, spike, stress and others are typically tests when game contains more ‘beef’ and are truly stressing the hardware), connectivity and hermetic testing (isolation of game from network, other services etc.).
Instead of fixing bugs later in the following sprints, this approach focus on fixing bugs instantly after the regression. Developers will be notified about their enhancements, new source code/build problems during the next day, and will have a chance to fix those problems even early in the on going sprint.
Smoke testing with automatic test exerciser. Automatic test exercisers provide a great way to smoke-test games. They need no specific tests but focus is in testing the user interface logic (e.g. opening menus, clicking buttons, doing swipes, multi-gestures etc.). Automatic test exercisers provide the least exact results, but those are very powerful in terms of providing quick feedback on each iteration/regression of your mobile game.
Regression testing after every code change. Regression testing can be used to ensure that the quality of mobile game is still good after any features, add-ons or changes in game. Each feature of mobile game should be tested on all available platforms before development of the next feature. Thus, regression testing is a part of testing life cycle and very important element in effort to build more robust games.
Connectivity testing. E.g most of the mobile games today have a server-client interaction, requiring a login, uploading of data (e.g. results) and downloading of data (e.g. data, images). If you’re developing these kinds of services around your game, developers need to ensure all their changes are done in code, to either server side or client side, do not break the functionality of the service.
Performance testing and all variants of it are important for mobile game testing. Sluggish and badly performing games can ensure that game won’t be successful, gets bad ratings and won’t make its creator name. Test automation can help to understand the real performance and many other aspects of game’s performance (e.g. battery consumption). For example, developers can create patterns to see how graphics assets gets loaded and used – and based on
that information to optimize handling of those.
Localization and language settings testing becomes highly important when your game is targeted for the global markets. The word ‘global’ means your game needs to be ‘local’ for everyone. When your game titles, texts and content needs to be translated and tested with devices in multiple languages, these types of tests can be easily performed automatically (with help of cloud-based device access and test automation) to change the language of devices, and even do it again with the same test runs. The problems layouts (e.g. fonts behaving bad) can be easily picked from screenshots. Real mobile devices can be also easily switch languages so all possible languages are testable with the exact same game content.
Functional testing is probably the most common method in mobile game testing. Typically functional testing is associated with manual testing and playing ‘game through’. However, to be agile, all functional testing should be automated. Functional testing – with help of test automation frameworks – requires some basic understanding of programming and setting up environment for testing. Automated functional testing can reveal issues related to user interface (and graphics), stability, game flow/mechanism, and integration of graphics assets.
More Advanced Test Methods for Mobile Games
Today, one of the hottest method for mobile game testing with lots of graphics (e.g. OpenGL ES) content is the use of image recognition. It can solve the well-known issue of instrumentation not being able to recognize UI elements based on IDs, descriptions and other typical handles that work with majority of test automation frameworks.
Here is an example video of how we use image recognition for well-known mobile game:
Go through the basics of Calabash, how to create proper Calabash tests and how to make the most of them.Download