Using OpenCV and Akaze for Mobile App and Game Testing

Bitbar, the mobile devops company. Logo, large

Dear Testdroiders,

Image recognition technique has been found extremely handy for testing mobile games and also some graphics-intensive apps. The fundamental issue that certain graphical or even native components cannot be identified by those test scripts and frameworks can be easily tackled when using image recognition using template matching technique. Also, image recognition works well for native mobile apps where fragments have been used within the app. This basically eliminates the need to identify UI component details (e.g. ID, visual size, description) and instead straightforward activities can be performed.

Artboard 1

We’ve been discussing mobile game testing with image recognition few times here at Testdroid blog. To get started, full source code example can be found at Github repository so just download it and let’s take a look what is included there. If you have troubles of setting up your Appium environment, download our free Appium beginner’s guide to figure out the proper way of doing it.

Updated on the 3rd of February 2016: If you want to build OpenCV and AKAZE for the specific platform using the latest libraries Building the Latest OpenCV and AKAZE Libraries for Mobile Game Testing after reading this one.

Introduction and Examples with AKAZE and OpenCV

We’ve been using Accelerated-KAZE (AKAZE) algorithm to find the matching keypoints between two images and then save them to a json file. Then, we use the OpenCV Java bindings to process the json file with keypoints and find the homography of the wanted image in a screenshot. If you are not familiar with OpenCV, it’s basically an open-source BSD-licensed library that includes hundreds of computer vision algorithms – and many of are easy to adopt and use for mobile game testing.

Let’s go back to our code example. The AkazeImageFinder.java class will run akaze_match, then use the keypoints from json to try to identify a given image inside a scene and return the corners of the image found. In this case, we look for specific images from ./queryimages/ folder and try to find them on a screenshot taken from device.

Once the image is located, we can:

  • tap it
  • drag it
  • swipe the screen with it
  • etc.

All these methods are implemented in the TestdroidImageRecognition.java class. This class has different functions for working with the images found on the screen and for looking for them. These are then used inside the test classes for the game tests.

Some of the functions we’ll cover here in this blog:

// Returns the corners of the image if found, or null if no image was found
public Point[] findImage(String image, String scene)

// Takes a screenshots, then tries to find the image on it and return the corners
public Point[] findImageOnScreen(String image)

// Takes a screenshot, tries to find the image on the screen, then tap the middle of it
public void tapImageOnScreen(String image)

// Takes a screenshot, finds the image on the screen, taps and holds it, 
// then swipes the screen while still holding it
public void swipeScreenWithImage(String image, int repeats)

// Takes a screenshot, tries to find the image, taps it and drags it to a certain position 
// given as x and y offsets from middle of the screen
public void dragImage(String image, double x_offset, double y_offset)

Building and Installing Dependencies

Okay, you’ve forked the project for yourself and everything is on your harddisk. Let’s go through few installations first. We provided Java example with these source code but naturally you can use any other language and adopt same methods we’ll cover here.

OpenCV

NOTE! When using Mac OS X the OpenCV 2.4.9 installation via Homebrew works only with Yosemite.

If you are using Homebrew to get OpenCV installed, follow these steps:

brew install homebrew/science/opencv

In case you have the latest version of Homebrew installed, you need to manually pull version 2.4.9 – and it can be done as follows:

cd /usr/local/Library/Taps/homebrew/homebrew-science

Checkout the version 2.4.9 manually:

git checkout 05ab591 opencv.rb

Then, unlink the older version:

brew unlink opencv

And finally, install version 2.4.9 as follows:

brew install opencv --with-java
mvn install:install-file -Dfile=/usr/local/Cellar/opencv/2.4.9/share/OpenCV/java/opencv-249.jar -DgroupId=opencv -DartifactId=opencv -Dversion=2.4.9 -Dpackaging=jar

NOTE! If you want to build OpenCV locally (and for any reason that Homebrew installation doesn’t work), follow these steps:

Download OpenCV 2.4.9 from http://opencv.org/downloads.html and extract the package on your hard disk. Then build the files:

cd opencv-2.4.9
mkdir build
cd build/
cmake -G "Unix Makefiles" ..
make -j8

Once you have the OpenCV .jar file (you can use the one in this repo under opencv directory), install it using Maven:

mvn install:install-file -Dfile=path/to/opencv/build/bin/opencv-249.jar -DgroupId=opencv -DartifactId=opencv -Dversion=2.4.9 -Dpackaging=jar

Now, all should be set up with OpenCV.

Akaze – Accelerated-KAZE

The Akaze C++ implementation can be found and built from the same project. The current project also contains the ./akaze/ folder with built binaries found under ./akaze/bin/akaze_match. Please note that these are Mac OS X binaries and will work only on Mac.

If you want to rebuild binaries, please take a look at these instructions. Regardless of which platform you end up using, you should have the following binaries under bin folder:

  • akaze_features
  • akaze_match
  • akaze_compare

testdroid-appium-driver

Testdroid driver for Appium can be pulled from here. To install it simply run this:

git clone https://github.com/bitbar/testdroid-appium-driver.git
git checkout v1.1.3

mvn install -DskipTests

Maven will take care of installing all other dependencies.

Configure, Build and Run the Java Image Recognition Sample

We’ve been discussing about Java and image recognition in few of our blogs, but if you are interested to set up an environment with Appium check this blog out.

Run from command line with Maven

Create a testdroid.properties file in the project root folder, containing this info:

testdroid.username=<ADD USERNAME HERE>
testdroid.password=<ADD PASSWORD HERE>
appium.automationName=Appium 
testdroid.project=Sample Project 
appium.appFile=src/test/resources/BitbarSampleApp.apk

NOTE! You shouldn’t use quotes around username and password property!

Run the tests from Maven using:

mvn test

Access Any Device from Cloud Service

You can specify in your command line what device will be used for your test run. The device name can be fetched from here. Device name should be something like – “Acer Iconia Tab 8 A1-840FHD EU” – “Samsung Galaxy Note 5 SM N920R4” – “Apple iPad Mini 3 A1599 9.0.2” and so on.

First, make sure you’ve set all the required parameters in the testdroid.properties file, and then run the following command with device info:

mvn -Dtestdroid.device="testdroid device name"  -Djava.library.path=<java-lib-path> test  

where java-lib-path is the directory where opencv 2.4.9 library can be found (for example: /usr/local/username/opencv/2.4.9/share/OpenCV/java if you used brew to install).

Reports

The reports, screenshots and everything else will be found under: ./target/reports/deviceName/

How to Create Your Own Image Recognition Tests

We’ve used Hill Climb Racing as an example due its popularity and nice game-play. If you are interested to see how this example works on real devices at Testdroid Cloud, please fire me an email and I’ll share the project with you. We’ve used the same function calls in that example, and with the source code from repository you should be able to build your own easily.

Screen Shot 2015-05-19 at 1.11.17 PM

Okay. Let’s look at the mainTest() function first at repository example. This is very basic example of how to call findImageOnScreen().

    @Test
    public void mainTest() throws Exception {
        driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
        logger.info("Image Recognition sample script started.");
        
        driver.hideKeyboard();
        findImageOnScreen("bitbar_logo");
    }
    public Point[] findImageOnScreen(String image) throws Exception {
        int retries = 5;
        Point[] imgRect = null;
        while ((retries > 0) && (imgRect == null)) {
            if (retries < 5) {
                log("Find image failed, retries left: " + retries);
            }
            takeScreenshot(image + "_screenshot");
            imgRect = findImage(image, image + "_screenshot");
            retries = retries - 1;
        }
        assertNotNull(imgRect);
        return imgRect;
    }

Taking a screenshot can be done using the following type of implementation:

    public void takeScreenshot(String screenshotName) throws Exception {
        counter = counter + 1;
        String fullFileName = System.getProperty("user.dir") + "/" + getFolder() + screenshotsCounter() + "_" + screenshotName + ".png";
        File folderPath = new File(System.getProperty("user.dir") + "/" + getFolder());
        String folderName = System.getProperty("user.dir") + "/" + "target/reports/" + getDeviceName();
        driver.takeScreenshot(fullFileName);
    }

Got any questions or need more information about this example? Weigh in with a comment below and let’s further improve this example!

Happy Testing!


How to Build a Million-Download Mobile Game

Learn tips & tricks to build better games and make users’ lives more enjoyable.

Download

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close