Automate Detox Mobile Tests on Real Devices in the Cloud

Running Detox Mobile Tests against Real Devices on Bitbar Cloud

Every now and then a new mobile test automation framework emerges, it quickly gains attention from mobile developers. This is happening right now with the Detox framework.

A New Framework

Detox is a JavaScript mobile testing framework that is built into the application and the test execution starts with app launch. This makes test execution really fast and robust as no external additional tools are needed to orchestrate and synchronize during the test execution.

As for the popularity of Detox, it can be easily summarized.

  • First, it comes to the number of users. There are plenty of seasoned JavaScript web developers using libraries such as ReactJS, while experienced mobile programmers are scarce.
  • Some existing testing frameworks that looked still really promising a year ago are now fading out and users are forced to move to other frameworks. This also happened to Calabash which is no longer under development. 
  • The current framework leader Appium originated from the web app testing and was later widely utilized in mobile app testing. It was great for testers as one could use the same concept in web testing with Selenium and in mobile testing against real devices or simulators. The problem with Appium is that launching tests is rather slow with installing and starting communication with the Webdriver. 

The Current State

Detox is running on emulators (Android), simulators (iOS) and real Android devices. But unlike Robotium and Calabash that are not under active development, the further development around Detox is progressing really fast. Running Detox tests on iOS real devices can be done with Bitbar Enterprise, either Private Device Cloud or On-Premise.

As for the activity, Detox has got more attention and followers (over 2000 stars) on Github compared to either iOS or Android Calabash Github framework!

Run Detox Mobile Tests Against Real Devices on Bitbar Cloud

Running Detox in Bitbar Device Cloud

Bitbar Cloud has always been about openness, providing great freedom for end users to choose how to test their application. This applies also to running Detox tests on Bitbar Device Cloud.

Of course, because of the current Detox limitations, you’ll need to run your iOS tests using simulators, until the support for real iOS devices is released by the Detox developers.

Follow these steps below to start an iOS simulator Detox test run on Bitbar Cloud. Before that, let’s check how your `run-tests.sh` should look like.

    1. You should already have locally a `package.json` file listing your needed npm packages to use with your Detox test. You will need to include this file in your test package zip and upload it to Bitbar Cloud.
    1. Install all npm packages required to run your Detox tests: `npm install`. This uses your package.json file to select all the required packages.
    2. Next you’ll need to install the Apple simulator with brew: `brew tap wix/brew` && `brew install applesimutils` as Detox is not supporting real iOS devices yet.
    3. Your script is now ready to start the app and test execution! `"${PWD}/node_modules/.bin/detox" test –configuration ios.sim.release –loglevel verbose > detox.log 2>&1`

Once everything is ready, we can move to Bitbar Cloud for test creation.

  1. On Bitbar Cloud, create a new Appium iOS Server Side project and name it as, for example, “Detox iOS”.
  2. Start a new test run:
    1. Upload a small dummy app. The testing framework requires you to upload some app, even if this is not the app that you’ll be testing. The app to be tested is uploaded to the test suite zip.
    2. The test zip package needs to include everything required for running your test. This includes your app with built-in Detox tests and your `run-tests.sh` that installs Node dependencies and launches the app under test.
  3. In test run’s advanced options increase the “Test time-out period” to 20 minutes. Installing the applesimutils takes a bit of time. We’ll include this as default in future releases.
  4. Start your test run!

Once your test is ready, you can find all the results in Bitbar Cloud, screenshots, video recording, logs and device performance to compare execution between devices. One thing worth noting, when executing tests on simulators, you won’t get correct performance or video recordings.

Here’s a working sample made by Jarno from our Customer Success team if you want to get your hands dirty: https://github.com/jarnie/testdroid-samples/tree/detox-sample/detox/demo-react-native-bitbar-cloud

Detox Tests Console Log The Console.log for Detox tests on Bitbar Cloud

Final Thoughts

Test automation has historically been thought through manual testers’ point of view. It should be possible to interact with the application as it would be used by a real tester. This is the reason for the success of current mainstream frameworks (eg. Appium). But is this really a valid point still today with DevOps and the “shift left” concepts? With developers ever more engaged with creating and maintaining test suites, why not make use of the internal workings of the application when testing it?

This is what Detox is doing, moving away from black box testing to gray box testing, where developers make use of the application’s internal state to drive tests. If you have Detox tests already and are in need of a mobile device cloud, you can already try to automate the tests through our free trial here.

 


How to Build a Million-Download Mobile Game

Learn tips & tricks to build better games and make users’ lives more enjoyable.

Download

  • Xamarin Srikanth

    This is a good read, but I do have some doubts if you could help me out it would be great
    1. Does this also come up with expo support for the react-native app?
    2. If the detox tests are in a different setup from the test-app will the cloud be able to pick them up?