Mobile Game Testing – Part #2: UI and Functionality + Image Recognition

Test Automation for Mobile Game Testing

Speaking of mobile game testing, it is a common misconception that manual testing is the only way to go forward. In many cases it is the first thing to start with, but to really get all issues spotted out and fixed before the game is published, test automation built into the process, hammering each and every regression and advancement can yield significant results when considering the customer-ready game.

Test Automation for Mobile Games

Let’s face the fact: manual testing has still too many disadvantages as it’s way too time-consuming, tedious, error-prone and not being able to thoroughly, systematically do the testing of whole software entity. Manual testing is somewhat like scratching the surface and you can’t really find out what’s going on underneath the UI. Besides, you would need an army of programming-capable guys to debug all bits and pieces – something that test automation can deliver you 24/7 – without any manual efforts.

For reason or another, some people still associate manual testing as the only way to test games. Sure, there are business reason for many companies out there to emphasize it as they are eager to sell their services (or crowdsource it to unknown quasi-testers) but again, it is the technology and test automation that can get in depth and make sure app works across the ecosystem of different devices, with different software setups, hardware configurations and so on.

We have discussed about the pros and cons of using different test automation frameworks for testing in our blogs and this time we’ll take a look at it with different framework – Appium – and new advanced features that test automation can bring into testing – image comparison and recognition.

Appium on real devices

UI and Functional Testing – with Appium

In a nutshell, Appium is a mobile UI testing framework supporting cross-platform testing of native, hybrid and mobile-web apps for iOS and Android. In fact, Appium is a pretty good choice for mobile games as in many cases those games tend to be identical – or at least very similar – on both platforms, Android and iOS – and the same test script can apply for both. But there are also some other reasons why Appium is a great choice for mobile game testing. For example, you can write tests using your favorite development tools/environment and programming languages, such as Java, Objective-C, Javascript, PHP, Ruby, Python, C# and so on.

Appium enables you to execute your tests on mobile device irrespective of the device OS. This is because the framework is basically a wrapper that translates Selenium Webdriver commands into UIAutomation (iOS) or UIAutomator (Android, API level>=17) or Selendroid (Android, API level <=16) commands depending on the device type. For example, in the context of Android, this is how Appium compares to other test automation frameworks:

Android Family Tree

EXAMPLE: Using Appium to test Clash of Clans (game by Supercell)

In this example we are using Supercell’s Clash of Clans game. A fantastic game and I bet many of you have played it so you should be pretty familiar how the game looks and so on. We’re also going to use Appium as a selected test automation framework to basic clicking-through of Clash of Clans tutorial.

 

##
## Example script that tests Clash of Clans tutorial first steps
##
## Works on different resolutions, both iOS and Android
##
 
import unittest
from time import sleep
from TestdroidAppiumTest import TestdroidAppiumTest, log
from selenium.common.exceptions import WebDriverException
 
class ClashOfClansTest(TestdroidAppiumTest):
    def setUp(self):
        # TestdroidAppiumTest takes settings (local or cloud) from environment variables
        super(ClashOfClansTest, self).setUp()
 
    def test_tutorial(self):
        driver = self.get_driver() # Initialize Appium connection to device
 
        sleep(10) # Wait that game loads
 
        # Use this to get detected screen hierarchy
        # print self.driver.page_source
        # Dismiss the in-app purchases dialog if it shows
 
        okayButton = None
        if self.isAndroid():
            try:
                okayButton = driver.find_element_by_id('button3')
                okayButton.click()
                sleep(5)
            except WebDriverException:
                log("There was no in-app purchases dialog")
        else: # iOS
            self.driver.implicitly_wait(5)          # wait only 5 seconds to find it
            try:
                okayButton = driver.find_element_by_accessibility_id('Okay')
                okayButton.click()
                # No need to sleep here since for iOS we wait the Game Center to popup...
            except WebDriverException:
                log("There was no in-app purchases dialog")
            self.driver.implicitly_wait(30)
 
        # Cancel iOS Game Center login
        if self.isIOS():
            #print self.driver.page_source
            try:
                self.driver.implicitly_wait(5)
                cancelButton = driver.find_element_by_accessibility_id('Cancel')
                log("Canceling iOS Game Center login...")
                cancelButton.click()
                sleep(2)
            except WebDriverException:
                log("The Game Center login was not displayed")
            self.driver.implicitly_wait(30)
 
        self.screenshot("welcome-chief")
 
        # Check that there is a goldmine on screen
        rect = self.find_image("queryimages/tutorial_goldmine.png", screenshot_match="screenshots/goldmine_match.png")
        self.assertIsNotNone(rect, "There should be a goldmine on screen in beginning of tutorial")
        log('Gold mine found at %s %s! Tapping tutorial forward...' % (rect[0], rect[1]))
 
        # Dismiss the bubbles
        self.tap_middle()
        sleep(2) # second blabla
        self.tap_middle()
        sleep(2) # Goblin appears
        self.tap_middle()
        sleep(1)
 
        # Go to shop
        # NOTE: tap_image does also assert, fails test if target not recognized
        self.tap_image("queryimages/shopbutton.png")
        sleep(1)
 
        # Buy cannon
        self.screenshot('cannon')
        self.tap_image("queryimages/cannon.png")
        sleep(2)
 
        # Place the cannon
        self.screenshot('place_the_cannon')
        self.tap_image("queryimages/place_the_cannon.png", width_modifier=0.75)
        sleep(2)
        self.screenshot('finish_now')
        # Use gem to finish right away
        self.tap_image("queryimages/finish_now.png")
        sleep(3)
        # Bring it on!
        self.screenshot('bring_it_on')
        self.tap_image("queryimages/bring_it_on.png", height_modifier=0.75)
        sleep(10)
        self.screenshot('battle')
        sleep(10)
        self.screenshot('end_of_battle')
 
        # To be continued...
 
if __name__ == '__main__':
    unittest.main()
 

Let’s look some stages in this script. The test_tutorial contains the following steps:

1. It first figures out if test is executed either on Android (self.isAndroid()) or iOS. As you can see, it looks content different, on Android it is trying to find by element ID and on iOS by accessibility ID with description (‘Okay’). The same check happens for iOS Game Center login.

2. Screenshots are taken in various steps and stored in files entered as a parameter in a function call.

3. We do a check if “goldmine” exists on screen by comparing two pngs using self.find_image call. If these pictures match (=goldmine exists on screen), we’ll go forward with tutorial.

4. We proceed with the tutorial with the following steps: 1) Go to shop, 2) Buy cannon, 3) Place the cannon. The information about all these three items is stored in those pngs: shopbutton.png, cannon.png, place_the_cannon.png.

5. Finally, we finish the tutorial and start the battle! After battle, application is brought down.

Okay. That’s about the programming and scripting for now and let’s see how that script looks on real devices. We used one iOS (iPhone 4S) and two Android phones (Samsung Galaxy S3 Mini and HTC One X) for this script. The video footage is taken at our office and is not a high-quality but you should see something going there on devices:

 

How we are using Image Recognition?

What we just went through in Appium+Clash of Clans example was the basic image recognition flow for enabling mobile game to be tested on real devices, regardless OS was different (Android and iOS). There are different ways to recognize content from images. For example, template matching is a technique for finding small parts of an image which match a template image. It is actually very handy even for recognizing UI elements and graphics resized and/or rotated in different form. This is one way how some Testdroid game developers use image recognition today.

 

Image recognition

Let’s say the template image has some strong features – e.g. text easy to be abstracted from the background content – a feature-based approach can be used. In this example “Button 1” text was resized and rotated (or if it is otherwise transformed) this can be quickly and easily identified and further actions can be taken.

We’re planning to host a webinar regarding this feature in Testdroid later this autumn so please check our webinar offerings and join us to learn more! Also, make sure you check all the blogs in our Best Practices in Mobile Game Testing blog series.

PS. In case you are interested in full source code of this example, please contact me directly at ville-veikko [dot] helppi [at] bitbar [dot] com.


The Fundamentals of Mobile Game Development and Testing

Learn the importance of infrastructure and architecture aspects, as well as the improved agile process and the value of testing in mobile game development and testing

Download

  • jenny howard

    Hello i want the full source code for automate testing clash of clans..

  • sandeep

    Could you let me know from where does find_image come from ? . It’s not part of Appium driver .

  • jenny howard

    Very nice information

  • reg008

    i can’t find tap_image analog in node.js binding and documentation (wd).
    So I can’t test games in javascript right ?

  • Pavel Strongin

    Which library are you using for image recognition?

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close