Building In-House Test Labs #2: What Devices Do You Need for Testing, How Many and How to Select Them

Bitbar, the mobile devops company. Logo, large

The key principle in real device testing is, naturally, to use the same real devices that the users in your target market are using. This second part of our blog series on Building In-House Test Labs focuses on selecting the right number of right devices to get a relevant portion of your target market covered. After you have selected a good set of devices, you need to refresh your device pool frequently enough so that your device list remains relevant and actually covers your requirements.

How Many Devices is enough

Long gone are the days when selecting handful of devices (small, medium and large resolution on few OS versions) was enough to get any real coverage of the devices in users’ hands. Today the number of different device variants, OS versions, resolutions and form factors is really wide and to make matters worse a lot of these different devices have shipped tens of millions units making each of them significant in their own right. Also this problem is not limited to only Android anymore but also on iOS you have already over 10 unique hardware and resolution combinations and 3-4 active OS versions making the iOS device matrix fairly sizeable as well. Download our free ebook to learn everything about setting up a mobile device testing lab.

Let’s first focus on how many devices you need to provide wide enough market coverage. For  Android you take a great data published by OpenSignal and from here you can see that for instance if you want to cover 50% of the global market you will need about 60 devices, about 20% global market coverage can be reached with 12 devices and if you want to go over 50% you will need significantly more devices.

device-frag

If you focus mainly on US market you can find great and fresh information published by app development company Double Encore. According to this data with well selected 25 Android devices you can cover two-thirds of the Active users and for 90% US market coverage you will need 128 devices.

device-pie

For estimating the number of needed iOS different iOS devices we can use data published by Apple. Now in May 2014 there are 8 unique iPhone models, 7 unique iPad models and 5 generations of iPod Touches you already get 20 unique devices, however given Apple’s user base you can potentially safely leave out iPhone 2G, 3G and 3GS as well as the first  three generations of iPod Touch as the number of these devices in active use is most likely fairly small especially as these have been introduced 5-7 years ago and have not even been in production for several years. This leaves 15 unique and still signicantly represented iOS devices in the market. On iOS you also have to remember that while most of iOS users do always update to the latest version there are always some users who don’t. For instance now about nine months after the introduction of iOS 7 there are still 12% of users who have not updated to the latest version. While this is nothing close to the situation in Android land it is advisable that you keep some iOS devices with the second latest iOS version as well.

For a complete picture, you can watch our newly recorded webinar about building large scale in-house test lab.

Watch this webinar

Which devices are the right ones for maximum coverage

Now that we know how many devices you need for both Android and iOS let’s take a bit more detailed look at which ones are the correct ones. Assuming your target market is North America we can again use Double Encore’s data as a good yard stick so the list of 25 most used Android devices looks like this:

 

  1. Samsung Galaxy S3 (17%)
  2. Samsung Galaxy S4 (12.3%)
  3. Motorola Droid Razr (3.7%)
  4. Samsung Galaxy Note 3 (3.7%)
  5. Samsung Galaxy Note 2 (3.3%)
  6. Asus Nexus 7 (3.2%)
  7. HTC One (1.9%)
  8. Samsung Galaxy S2 (1.8%)
  9. Samsung Galaxy Tab2 10.1 (1.6%)
  10. Droid Razr HD (1.4%)
  11. Droid Razr M (1.4%)
  12. Samsung  Galaxy S2 Epic (1.3%)
  13. Samsung Galaxy Tab 2 7.0 (1.3%)
  14. Samsung Galaxy Note 10.1 (1.2%)
  15. LG G2 (1.1%)
  16. Samsung Galaxy S (<1%)
  17. Motorola Droid X (<1%)
  18. Motorola Droid Bionic (<1%)
  19. Samsung Galaxy S5 (<1%)
  20. Samsung Galaxy Nexus (<1%)
  21. LG Nexus 5 (<1%)
  22. Motorola Moto X (<1%)
  23. Samsung Galaxy Tab 3 10.1 (<1%)
  24. LG Nexus 4 (<1%)
  25. HTC Thunderbolt

Respectively if you want to cover the most used iOS devices the list looks like this:

1. iPhone 5S
2. iPhone 5C
3. iPhone 5
4. iPhone 4S
5. iPhone 4
6. iPad Mini
7. iPad Mini 2nd generation
8. iPad Air
9. iPad 4th generation
10. iPad 3th generation
11. iPad 2nd generation
12. iPad 1st generation
13. iPod Touch 5th generation
14. iPod Touch 4th generation

 

 

 

 

 

 

How often do I need to update my device list

Once you have your lab up and running you need to think how often you need to add new devices to keep your market coverage. For iOS devices it is simple as you only need to add new devices when Apple announces new models (twice a year) but on Android it a bit more complex than that. A good rule of thumb is to check every quarter what are the latest block busters (especially any new Galaxy devices) and add those to your lab. You can’t really substitute the older ones because, like you see from the TOP25 device list, even 2-3 year old number one devices (think Samsung Galaxy S2 and Samsung Galaxy S) have such a huge installed base that they will hang on the list of most used devices for years.

This blog post covered how many and which devices you need to select to your in-house device lab if you aim for maximum market coverage. Next week we will look into the required infrastructure i.e what other hardware and software you need for your enterprise grade device lab.

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close