Mobile Game Testing and Test Automation Using Real Devices

Bitbar, the mobile devops company. Logo, large

Dear Testdroiders,

Let’s start by discussing a bit about mobile game testing in general. Testing mobile game means ensuring that it is running properly, it meets its all specific requirements and provides fantastic user experiences to gamers. This may sound difficult – especially when there are bazillion different device configurations where the game must run well. If you consider that significant portion of Google Play and App Store revenues are generated by mobile games, there is absolute need to automate as many mobile game components as possible.

The Challenges in Testing Mobile Games

Mobile game segment is fiercely competitive and users have short attention span. Customer lifetime value for each game is heavily dependent on additional content, user collaboration etc which means that everything has to work over several app updates, back-end updates over extended period of time to recover the initial marketing investments.

Screen Shot 2015-11-05 at 11.42.50 AM

What are those challenges to automate all testing aspects of those games? Firstly, majority of games use direct screen access in form of OpenGL or ActiveX bypassing the OS level services. This leads to the problem that all of the native mobile test automation frameworks are pretty useless with mobile games. Without access to the object level information the automation can only utilize xy clicks without much feedback or validation about the internal state of the game.

Secondly, performance is often key driver for user experience and the real performance can only be observed on real hardware. Frame rates do matter a lot and with the richness and variety of mobile devices every model counts. In fact, one of our customers’ told that supporting just one more popular Asian device model can bring additional $5 million of revenue during the lifetime of a game.

Screen Shot 2015-11-05 at 12.06.28 PM

Also many application binaries are massive (up to 3Gb) and games consume a lot of memory, CPU, GPU and battery – all this need to be accounted for so that the real end user experience does not suffer due to physical realities of mobile devices.

Finally, majority of games utilize sensors or other HW features that are not consistent from one device to another and if this is left unchecked there will be nasty surprises down the road. One good example of this is that we found out that one of the popular Chinese mobile hardware platforms has acceleration sensor that for some reason uses inverted x and y coordinates. You can imagine what that does to all games that rely on gesture recognition based on the data from acceleration sensor!

For the past two years, we’ve been working with the biggest mobile game companies in the world to enable their game testing on real devices. The spectrum of used tools, methods, practices and frameworks is pretty huge, but there are certain things that pretty much all game developers do the same way. Despite of what game engine they have selected, for example, over 90% of game developers use some form of image recognition. This maybe as simple as taking a screenshot and analyzing it automatically or manually – OR – it can be fully automated gameplay where the test script drives the game, and outputs stuff to logs for further inspection – OR – even the most advanced way and combination of these two, using the real-time screenshots to find certain graphical elements, comparing those to pre-set graphics assets and then playing the game through based on the test script.

In case you’re interested to try out this for your game, please contact me personally and I’ll be happy to share some great examples how this can be done for your game. Furthermore, we’ve been highlighting the example of image recognition implementation in our blogs.

Graphics Performance is One of Your Top Priorities

Good graphics performance of any mobile game is tightly related to good user experience. Stunning graphics, pictures, animations and all those beautiful effects will make the mobile game shine, but if the performance lags, they are pretty much useless. In the gameplay, gamers want to see constant progress, feel that game is fully implemented for their device, and nothing slows down the game experience.

In short, there are variety of different methods and ways to test graphics performance of a mobile game. Naturally, that requires a real device (phone, tablet) and preferable many of those.

Screen Shot 2015-11-05 at 10.11.27 AM

It’s never exaggerative to start performance tests while game is still in very early phase under development. The sooner you spot those performance bottlenecks, the sooner you’ll get that optimized. Before you publish your game in Google Play or App Store, you should have run different types of performance tests for your game on all possible device configurations.

For example, the most typical performance tests done for mobile games are general stress tests to determine how the game performs in terms of responsiveness, stability and refresh rate under certain workload or user interaction pattern. If you are looking what performance test would work the best, take a look at this list of different performance tests.

Furthermore, we’ve also enabled GameBench in Testdroid Cloud for a specific graphics performance tests. This will offer you lots of great details about FPS during the execution of the game on real devices.

White-Box vs. Gray-Box vs. Black-Box Testing for Mobile Games

White-box testing focuses on the architectural, integration and systemic aspects of mobile game, and it requires knowledge of the code requirements, your internal and external software architecture and libraries, databases, game engines that is used as part of the game. This typically takes place in the earlier phase of the development and can be considered as unit testing. Naturally, test automation provides lots of value for white-box testing, but the focus it primarily on underlying architecture, software and integrations.

Gray-box testing provides instrumentation aspect to mobile game testing. The instrumentation can run through test on mobile game but doesn’t get in great details in terms of underlying architecture, integrations, libraries and other software. Check out more about gray box test automation with Unity games

Black-box testing focuses on mobile game functionality without actual visibility into the underlying implementation. It will uncover the logic of the game and focus on design issues, graphics assets, and gameplay/usability of the game. With black-box testing testing can reveal flaws in visual appearances, layout problems, potential usability or inconvenience of the gameplay. Furthermore, to accommodate

Black Box Testing with Appium & OpenCV on Android and iOS Devices

Let’s look at the black-box testing in more details. You can also check our Appium example for Android and iOS devices, running against popular Clash of Clans mobile game.

First, we wanted to use some open source components to bridge the gap between image recognition, handling of graphics assets, and combining those with Appium framework. The fact that Appium is a cross-platform framework works really well here because mobile games usually work exactly the same way on Android and iOS devices enabling us to use the same script on both major platforms. Appium does a lot of the heavy lifting from the installation to providing necessary input events and managing the test automation session while providing sufficiently high level interface for the next layer of our solution.

Screen Shot 2015-11-05 at 11.03.20 AM

The second step for building this combination to provide value for mobile game testing, we were looking a way to recognize graphics elements and how test script could access that. After that, based naturally what test script is meant to do, we have to enable interactions on these graphical elements and figure out a way to validate if the view has changed after any interaction.

In order to access the game from the outside we chose OpenCV image recognition library that is used for reading the screen buffer and giving X & Y coordinates as an input for the Appium script. OpenCV is really cool open source library that enables resolution agnostic image recognition, it is very customizable and can even recognize images that are stretched or at an angle.

The idea is to simplify the script creation so much that it includes only two types of tasks: Clipping reference images and defining what sort of click is performed when the match is found from the screenshot. No writing of complex scripts, compiling code or any other computer science skills are needed.

You can check how seamlessly it works on this YouTube video. Script snippet can be found here.

Happy Game Testing!

The Fundamentals of Mobile Game Development and Testing

Learn the importance of infrastructure and architecture aspects, as well as the improved agile process and the value of testing in mobile game development and testing


  • jenny howard

    Can you guide how open CV can be used to do the image recognization for games?