Device Diversity in Mobile Testing

Bitbar, the mobile devops company. Logo, large

Every now and then, you find one or two pieces of negative comments like ‘Crashes on my Nexus 4’ or ‘It totally doesn’t work on my HTC EVO’. For the first comment, all you need to do is test more on Nexus 4, while the best suggestion for the second comment or alike is do not skip testing your app on that single model. Put it in another way, you need to ensure device diversity in your mobile app testing.

So how important is large device diversity in mobile testing? Well, if you look around in your office and figure out the total number of different devices or variations that your colleagues have, you will immediately understand it. As a matter of fact, much of the quality in a mobile app is wrapped up at the end user experiences. Suppose your app does not work on a single model, the end user experiences for that model is totally missing. As a result, more ‘thumbs down’ will appear and your app ratings and revenue generation will suffer.

Let’s see how large the device diversity should be. With the evolvement of generation and OS version, the iOS ecosystem is growing rapidly. To cover the most used iOS devices, you would need the following devices (see table). For most of these devices, you should make sure your apps have a good performance with various OS versions – iOS 7, iOS 6, iOS 5. In addition, it’s rumored that iPhone 6 equipped with iOS 8 will be released soon alongside iPad Mini 3, iPad Air 2, and 12.9-inches iPad Pro.

iOS device list

If the picture of iOS diversity looks still ok to you, however, then you must know this is not the case in the Android world. Assuming your target market is only North America, we can use Double Encore’s data as a good reference. The list of top 25 of most used Android devices should look like this.

Android device list

On top of these Android devices, the number of various Android OS versions also plays an important role in determining its diversity. Early this month Google published data about the distribution of Android OS versions. As you can see, each popular Android device equipped with a unique OS version forms a unique model. Once again, if you miss one of them, chances are that you will lose a large group of customers.

Android version

What gets even worse, this is not the end. When your mobile apps/games expand to new markets, ensuring the quality of your mobile apps/games gets increasingly difficult and you need to include local Android devices into your testing list as well.

According to the latest report from Flurry, taking iPhone users in China as a baseline, Xiaomi smartphone owners spent 7% more time, while Samsung users spent 14% less time and HTC users spent 27% less time. When China becomes one of your target markets, undoubtedly, you probably need to give high testing priority to local OEMs’ like Xiaomi, even though Samsung and HTC are still better performing than Xiaomi in terms of the global shipments. This requires a better understanding of the users in the target markets and the models of predominant devices when your apps start being active in nascent markets (Check out what Chinese devices and Japanese devices we have in Testdroid Cloud).

Believe it or not, some companies still hold the opinion that testing mobile apps on a handful of devices like 20-30 will suffice. But the truth is only ensuring a good user experience on a large number of devices will lead your app to success. While buying that amount of devices is not viable for many organizations, especially SMEs, relying on online cloud mobile testing services like Testdroid Cloud for a large device coverage is the best way to secure your mobile strategy.