Skip to content

Speaking of mobile game testing, it is a common misconception that manual testing is the only way to go forward. In many cases, it is the first thing to start with, but to really get all issues spotted out and fixed before the game is published, test automation built into the process, hammering each and every regression and advancement can yield significant results when considering the customer-ready game.

Test Automation for Mobile Games

Let’s face the fact: manual testing has still too many disadvantages as it’s way too time-consuming, tedious, error-prone and not being able to thoroughly, systematically do the testing of whole software entity. Manual testing is somewhat like scratching the surface and you can’t really find out what’s going on underneath the UI. Besides, you would need an army of programming-capable guys to debug all bits and pieces – something that automated testing can deliver you 24/7 – without any manual efforts.

For a reason or another, some people still associate manual testing as the only way to test games. Sure, there are business reason for many companies out there to emphasize it as they are eager to sell their services (or crowdsource it to unknown quasi-testers) but again, it is the technology and test automation that can get in depth and make sure app works across the ecosystem of different devices, with different software setups, hardware configurations and so on.

This time we’ll take a look at it by using a different framework – Appium – and new advanced features that test automation can bring into testing – image comparison and recognition.

Appium on real devices

UI and Functional Testing – with Appium

In a nutshell, Appium is a mobile UI testing framework supporting cross-platform testing of native, hybrid and mobile web apps for iOS and Android. In fact, Appium is a pretty good choice for mobile games as in many cases those games tend to be identical – or at least very similar – on both platforms, Android and iOS – and the same test script can apply for both. But there are also some other reasons why Appium is a great choice for mobile game testing. For example, you can write tests using your favorite development tools/environment and programming languages, such as Java, Objective-C, Javascript, PHP, Ruby, Python, C# and so on.

Appium enables you to execute your tests on mobile device irrespective of the device OS. This is because the framework is basically a wrapper that translates Selenium Webdriver commands into UIAutomation (iOS) or UIAutomator (Android, API level>=17) or Selendroid (Android, API level <=16) commands depending on the device type. For example, in the context of Android, this is how Appium compares to other test automation frameworks:

Android Family Tree

Example: Using Appium to Test Clash of Clans

In this example, we are using Supercell’s Clash of Clans game. A fantastic game and I bet many of you have played it so you should be pretty familiar with how the game looks and so on. We’re also going to use Appium as a selected test automation framework to basic clicking-through of Clash of Clans tutorial.

 

##
## Example script that tests Clash of Clans tutorial first steps
##
## Works on different resolutions, both iOS and Android
##
 
import unittest
from time import sleep
from TestdroidAppiumTest import TestdroidAppiumTest, log
from selenium.common.exceptions import WebDriverException
 
class ClashOfClansTest(TestdroidAppiumTest):
    def setUp(self):
        # TestdroidAppiumTest takes settings (local or cloud) from environment variables
        super(ClashOfClansTest, self).setUp()
 
    def test_tutorial(self):
        driver = self.get_driver() # Initialize Appium connection to device
 
        sleep(10) # Wait that game loads
 
        # Use this to get detected screen hierarchy
        # print self.driver.page_source
        # Dismiss the in-app purchases dialog if it shows
 
        okayButton = None
        if self.isAndroid():
            try:
                okayButton = driver.find_element_by_id('button3')
                okayButton.click()
                sleep(5)
            except WebDriverException:
                log("There was no in-app purchases dialog")
        else: # iOS
            self.driver.implicitly_wait(5)          # wait only 5 seconds to find it
            try:
                okayButton = driver.find_element_by_accessibility_id('Okay')
                okayButton.click()
                # No need to sleep here since for iOS we wait the Game Center to popup...
            except WebDriverException:
                log("There was no in-app purchases dialog")
            self.driver.implicitly_wait(30)
 
        # Cancel iOS Game Center login
        if self.isIOS():
            #print self.driver.page_source
            try:
                self.driver.implicitly_wait(5)
                cancelButton = driver.find_element_by_accessibility_id('Cancel')
                log("Canceling iOS Game Center login...")
                cancelButton.click()
                sleep(2)
            except WebDriverException:
                log("The Game Center login was not displayed")
            self.driver.implicitly_wait(30)
 
        self.screenshot("welcome-chief")
 
        # Check that there is a goldmine on screen
        rect = self.find_image("queryimages/tutorial_goldmine.png", screenshot_match="screenshots/goldmine_match.png")
        self.assertIsNotNone(rect, "There should be a goldmine on screen in beginning of tutorial")
        log('Gold mine found at %s %s! Tapping tutorial forward...' % (rect[0], rect[1]))
 
        # Dismiss the bubbles
        self.tap_middle()
        sleep(2) # second blabla
        self.tap_middle()
        sleep(2) # Goblin appears
        self.tap_middle()
        sleep(1)
 
        # Go to shop
        # NOTE: tap_image does also assert, fails test if target not recognized
        self.tap_image("queryimages/shopbutton.png")
        sleep(1)
 
        # Buy cannon
        self.screenshot('cannon')
        self.tap_image("queryimages/cannon.png")
        sleep(2)
 
        # Place the cannon
        self.screenshot('place_the_cannon')
        self.tap_image("queryimages/place_the_cannon.png", width_modifier=0.75)
        sleep(2)
        self.screenshot('finish_now')
        # Use gem to finish right away
        self.tap_image("queryimages/finish_now.png")
        sleep(3)
        # Bring it on!
        self.screenshot('bring_it_on')
        self.tap_image("queryimages/bring_it_on.png", height_modifier=0.75)
        sleep(10)
        self.screenshot('battle')
        sleep(10)
        self.screenshot('end_of_battle')
 
        # To be continued...
 
if __name__ == '__main__':
    unittest.main()
 

Let’s look at some stages in this script. The test_tutorial contains the following steps:

1. It first figures out if the test is executed either on Android (self.isAndroid()) or iOS. As you can see, it looks content different, on Android it is trying to find by element ID and on iOS by accessibility ID with description (‘Okay’). The same check happens for iOS Game Center login.

2. Screenshots are taken in various steps and stored in files entered as a parameter in a function call.

3. We do a check if “goldmine” exists on screen by comparing two .png files using self.find_image call. If these pictures match (=goldmine exists on screen), we’ll go forward with the tutorial.

4. We proceed with the tutorial with the following steps: 1) Go to the shop, 2) Buy cannon, 3) Place the cannon. The information about all these three items is stored in those png files: shopbutton.png, cannon.png, place_the_cannon.png.

5. Finally, we finish the tutorial and start a battle! After the battle, the application is brought down.

Okay. That’s about the programming and scripting for now and let’s see how that script looks on real devices. We used one iOS (iPhone 4S) and two Android phones (Samsung Galaxy S3 Mini and HTC One X) for this script. The video footage is taken at our office and is not a high-quality but you should see something going there on devices:

Using Image Recognition

What we just went through in Appium and the Clash of Clans example was the basic image recognition flow for enabling the mobile game to be tested on real devices, regardless OS was different (Android and iOS). There are different ways to recognize content from images. For example, template matching is a technique for finding small parts of an image which match a template image. It is actually very handy even for recognizing UI elements and graphics resized and/or rotated in a different form. This is one way how some game developers use image recognition today.

 

Image recognition

Let’s say the template image has some strong features – e.g. text easy to be abstracted from the background content – a feature-based approach can be used. In this example, “Button 1” text was resized and rotated (or if it is otherwise transformed) this can be quickly and easily identified and further actions can be taken.

Ville-Veikko Helppi

Mobile Testing Product Expert