My Background: Appium and Mobile Game Test Automation
Before going into details, I’d like to share a bit about my own context and the struggles I’ve had in mobile game test automation and the story of AltUnityTester.
A few years ago, I started working on a few projects with Bitbar developing automated scripts for mobile gaming companies. Most of these companies were looking to get fast feedback on their games by running basic checks on many different devices using Bitbar Cloud. Most of them were also interested in running checks on both iOS and Android and they mostly wanted to see that the game starts and loads correctly in a reasonable amount of time on as many devices as possible. Some companies were also looking into adding tests to verify the first steps of the game, finishing the first few levels, navigating through the main menus and trying out the most important features.
Appium seemed to be the best choice for this, since it offers great interaction with the device on both iOS and Android, supports many programming languages (so each company could choose their favorite) and is a favorite of many testers that are familiar with UI Automation in general, as it’s similar to Selenium WebDriver when it comes to mobile web testing.
Get started with successful Appium mobile app testing with the help of this free guide
But for games developed using game engines like Unity or Unreal but not using the native UI frameworks, Appium couldn’t detect any of the objects in the game. Clicking at specific coordinates was not an option considering the device fragmentation problem that most companies were trying to solve and the diversity of resolutions available out there (Here is a presentation I did at ETC 2016 about our approach). We decided to use image recognition to detect “objects” (pictures) on the screen. We had a lot of success with that, but, as expected, image recognition is slow and this approach was not suitable for games like endless runners or games where reaction time is essential.
Unity for App Development
A couple of years after that, I started working with a different company as a tester/app developer in their New Concepts team. We were trying out a lot of new ideas, developing small apps and prototyping a lot of solutions in a very fast-paced environment. Along with new ideas, we also needed to try new graphical designs and we needed to create apps that were fast to develop and could be deployed to a multitude of platforms if needed. We focused mostly on iOS and Android, but we needed to know that the apps could also run on Windows, MacOS, etc. After trying out a few options, including some web frameworks, we decided to develop all our apps in Unity. It has support for any platforms that we could ever need and it’s excellent for fast prototyping, with incredible graphics on every platform and with an abundance of ready-made artworks, scripts, plugins and solution available on their Asset Store.
While Unity allowed us to be very productive on the development side, I was again struggling on the testing side. I had to deal with a lot of apps that still required a lot of testing. Most of these apps also had to work together as part of a system, and we often found ourselves trying to figure out ways to deploy and run some automated tests on our environment automatically. While the apps were relatively simple and would have been the perfect candidate for simple automated checks with Appium (which works on Android, iOS and even Windows nowadays), Unity didn’t have a solution for running these types of system level / UI driven tests on real devices.
Looking for a solution
While searching for a solution for these two problems, Rasmus Selsmark, QA Lead who works at Unity Helsinki (a customer of Bitbar Cloud), pointed out the hack-day project that provides one possible option for running Appium tests against Unity games and getting feedback from Unity about the objects on the screen via the Debug.Log() functionality – so essentially by injecting information about specific objects in the device logs (logcat on Android or syslog on iOS), from where they can then be parsed and used inside the Appium tests.
I decided to give it a shot by implementing something similar using a TCP socket running from the Unity game that can listen to commands sent from the tests. And so AltUnityTester came to be.
AltUnityTester
AltUnityTester – now also available on the Unity Asset Store – will let you find objects in your game and get their coordinates on the screen so you can interact with them from Appium.
Currently, AltUnityTester has bindings implemented for Python, allowing you to
- create an AltUnityDriver,
- connect to the AltUnityTester component running on the device,
- ask for information on objects from the Unity Game,
- find elements and get their coordinate
Once you get the coordinates for an element (like “Bed” in the example below) you can use Appium to tap() on screen at those coordinates.
Source Code and Documentation
We decided to have AltUnityTester open sourced and we have posted it on our GitLab account here:
Getting Started with AltUnityTester
To help you set up your game and get started with AltUnityTester, we’ve created a video tutorial:
Running AltUnityTester in the Cloud
The Getting Started tutorial uses Unity’s Adventure example to show how a game like that could be tested using AltUnityTester. I wanted to give it a try on Bitbar Cloud as well since you can easily run Cloud-Side Appium tests in their cloud. I also used their Python server-side Appium example scripts as a starting point and made the following modifications:
- I added the python binding installation in the requirements.txt script:altunityrunner==0.1.3
- I added the adb forward command in “run-scripts.sh”, just before starting Appium:
adb forward tcp:13001 tcp:13000
I then used the following Python script to run the tests using AltUnityTester in the cloud (click on the photo for the link to the entire code snippet):
Here is the result of these tests on a Nexus 5 from Bitbar Cloud:
If you’d like to know more about AltUnityTester or about any of the aspects covered in this blog post, leave a comment below!
Ru Cindrea is a Senior Test Consultant and Managing Partner at Altom Consulting. With over ten years of experience in software testing, she is particularly interested in mobile testing and test automation with a special interest in mobile games. Ru is an instructor for the Black Box Software Testing series offered by Altom in collaboration with KFA. She is an organizer of the Test Lab, an open-space concept organized at testing conferences where participants practice testing and learn from their peers.