Speed Up Mobile Testing Cycles with Test Sharding

Speed up mobile test execution with test sharding

It’s obvious that companies nowadays are investing more in automation solutions to streamline the workflow of build, test, deploy and monitor mobile apps. As developing high-quality apps requires lots of intensive testing, companies will most likely highlight Speed as one of the cornerstones when evaluating test automation tools and services.

With speed, you can either get the same amount of tests done in less time or cover more test cases within the same amount of time. It depends on your use case which one you want to achieve. When you parallelize your tests across hundreds of real devices, more devices and tests are covered in one single test run. But how to verify a lot of tests say 100 or even 1000 tests in less time? Test sharding is the answer.

Why test sharding?

Imagine that you have 1000 tests taking 100 minutes in total to complete on a single device, wouldn’t this take too much time that keeps every team on the pipeline waiting for testing feedback? Now what if we evenly divide these 1000 tests to get them run on 10 devices (100 tests for each device)? It will theoretically take 10 minutes rather than 100 minutes to finish the test session and get the test reports.

In the end, the purpose of doing test sharding is to spread tests across devices and ease testing workload so that you can execute automated tests faster, increase throughput and scale continuous delivery with a short feedback loop.

Shorten mobile testing cycles with test sharding

How to shard tests on Bitbar Testing?

Bitbar Testing comes with the ultimate scalability so that allows you to run tests across any number of real devices concurrently. As to test sharding, we have the out-of-the-box support for it. You can create a free account to try it out. 

With our Key/Value pair argument capability available from the Advanced Options when setting up a test run, you can use 2 new Key/Value pairs, numShards and shardIndex. And we will address handling these within the testdroid.properties -config file (available for editing by a system administrator).

Key/Value pair for test shardingThe test runner supports splitting a single test suite into multiple shards, so you can easily run tests belonging to the same shard together as a group, under the same instrumentation instance. Each shard is identified by an index number. When running tests, use the -e numShards option to specify the number of separate shards to create and the -e shardIndex option to specify which shard to run.

For example, to split the test suite into 10 shards and run only the tests grouped in the second shard, you would give the following value to key/value pairs in test options. This can be done either in our Cloud UI or through API.


P.S. If you are not in the scope of doing test sharding but would love to run a specific test case rather than the entire test case, we have the option as well. Simple define which test package or class you want to execute in the Advanced Option.

run certain test package or test class

But you got a question about device availability?

To get shorter testing cycles, there are two pillars. As you’ve seen, one is splitting tests and the other one comes to the devices. More specifically, device availability.

By the time you split tests across a few devices, you want to make sure every device is available so that all the shards can be executed simultaneously. If the tests are executed at a different time, then you are basically doing it sequentially and your so-called ‘test sharding’ will bring minimal benefits to you.

Bitbar provides a public cloud that offers the highest number of unique device models with different SW and HW combinations. Beyond that, we are well prepared for the increasing demand for a single device model by providing several copies in our shared testing environment.

That being said, setting up a private infrastructure with a few dedicated devices is the best solution to maximize the device availability in terms of doing split testing. With access to this kind of private testing environment, you can distribute your tests to all of the connected devices to ensure the fastest possible test execution time and allow you to forward feedback to developers instantly.

To sum

The nature of automation is to reduce the manual work that needs to be repetitively done and decrease lead time with expected outcomes. By using advanced techniques like test sharding, you can attain a more tightly closed feedback loop among DevOps teams.

5 Things to Consider When Adopting Mobile App Test Automation

Learn these aspects to improve test efficiency and effectiveness.


  • pradeep h.k

    Are the keys limited to only these two values “numShards” and “shardIndex” or we can pass any custom keys ?

    It will be really helpful if we can pass custom keys/values and Instrumentation java can use those custom keys to filter tests tests based on those values. Example : filtering tests based on Tags in Cucumber Android framework with Instrumentation.This is one of widely used approach across different companies

    • Dave H

      From Docs: http://docs.bitbar.com/testing/user-manuals/projects/index.html?highlight=key%20value

      Public cloud supports a number of Shell environment variables that are made available to each test run. These can be used for test case sharding or selecting an execution logic for Calabash runs.

      Espresso test sharding, the variables numShard and shardIndex are supported out of the box. Use these the same way as you would locally.

      Calabash references environment variables to control its runtime behavior. There are two pre-defined environment variables CALABASH_TAGS and CALABASH_PROFILE that can be defined. These can be used to better orchestrate the test execution during the test run.

      Xcode based test suites can be controlled with XCODE_SKIP_TESTING and XCODE_ONLY_TESTING keys.

      XCODE_SKIP_TESTING should take the value of -skip-testing command line flag. This allows one to skip some named test case or class.
      XCODE_ONLY_TESTING takes the value of command line flag -only-testing. This flag allows one to define of running a single test method or all tests from a test class.
      On-premise and private cloud setups can allow users to create their own key-value pairs. For customers with advanced plans it is possible to create keys for specific tasks to be done before, during or after a test run also on public cloud.