The key principle in real device testing is, naturally, to use the same real devices that the users in your target market are using. This second part of our blog series on Building In-House Test Labs focuses on selecting the right number of right devices to get a relevant portion of your target market covered. After you have selected a good set of devices, you need to refresh your device pool frequently enough so that your device list remains relevant and actually covers your requirements.
How Many Devices is enough
Long gone are the days when selecting a handful of devices (small, medium and large resolution on few OS versions) was enough to get any real coverage of the devices in users’ hands. Today the number of different device variants, OS versions, resolutions and form factors is really wide and to make matters worse a lot of these different devices have shipped tens of millions of units making each of them significant in their own right. Also this problem is not limited to only Android anymore but also on iOS you have already over 10 unique hardware and resolution combinations and 3-4 active OS versions making the iOS device matrix fairly sizeable as well. Download our free ebook to learn everything about setting up a mobile device testing lab.
Let’s first focus on how many devices you need to provide wide enough market coverage. For Android you take a great data published by OpenSignal and from here you can see that for instance if you want to cover 50% of the global market you will need about 60 devices, about 20% global market coverage can be reached with 12 devices and if you want to go over 50% you will need significantly more devices.
If you focus mainly on US market you can find great and fresh information published by app development company Double Encore. According to this data with well selected 25 Android devices you can cover two-thirds of the Active users and for 90% US market coverage you will need 128 devices.
For estimating the number of needed iOS different iOS devices we can use data published by Apple. Now in May 2014 there are 8 unique iPhone models, 7 unique iPad models and 5 generations of iPod Touches you already get 20 unique devices, however, given Apple’s user base you can potentially safely leave out iPhone 2G, 3G and 3GS, as well as the first three generations of iPod Touch as the number of these devices in active use, is most likely fairly small especially as these have been introduced 5-7 years ago and have not even been in production for several years. This leaves 15 unique and still significantly represented iOS devices in the market. On iOS, you also have to remember that while most of the OS users do always update to the latest version there are always some users who don’t. For instance now about nine months after the introduction of iOS 7, there are still 12% of users who have not updated to the latest version. While this is nothing close to the situation in Android land it is advisable that you keep some iOS devices with the second latest iOS version as well.
For a complete picture, you can watch our newly recorded webinar about building large scale in-house test lab.
Which devices are the right ones for maximum coverage
Now that we know how many devices you need for both Android and iOS let’s take a bit more detailed look at which ones are the correct ones. Assuming your target market is North America we can again use Double Encore’s data as a good yardstick so the list of 25 most used Android devices looks like this:
- Samsung Galaxy S3 (17%)
- Samsung Galaxy S4 (12.3%)
- Motorola Droid Razr (3.7%)
- Samsung Galaxy Note 3 (3.7%)
- Samsung Galaxy Note 2 (3.3%)
- Asus Nexus 7 (3.2%)
- HTC One (1.9%)
- Samsung Galaxy S2 (1.8%)
- Samsung Galaxy Tab2 10.1 (1.6%)
- Droid Razr HD (1.4%)
- Droid Razr M (1.4%)
- Samsung Galaxy S2 Epic (1.3%)
- Samsung Galaxy Tab 2 7.0 (1.3%)
- Samsung Galaxy Note 10.1 (1.2%)
- LG G2 (1.1%)
- Samsung Galaxy S (<1%)
- Motorola Droid X (<1%)
- Motorola Droid Bionic (<1%)
- Samsung Galaxy S5 (<1%)
- Samsung Galaxy Nexus (<1%)
- LG Nexus 5 (<1%)
- Motorola Moto X (<1%)
- Samsung Galaxy Tab 3 10.1 (<1%)
- LG Nexus 4 (<1%)
- HTC Thunderbolt
Respectively if you want to cover the most used iOS devices the list looks like this:
- iPhone 5S
- iPhone 5C
- iPhone 5
- iPhone 4S
- iPhone 4
- iPad Mini
- iPad Mini 2nd generation
- iPad Air
- iPad 4th generation
- iPad 3th generation
- iPad 2nd generation
- iPad 1st generation
- iPod Touch 5th generation
- iPod Touch 4th generation
How often do I need to update my device list
Once you have your lab up and running you need to think about how often you need to add new devices to keep your market coverage. For iOS devices, it is simple as you only need to add new devices when Apple announces new models (twice a year) but on Android it a bit more complex than that. A good rule of thumb is to check every quarter what are the latest blockbusters (especially any new Galaxy devices) and add those to your lab. You can’t really substitute the older ones because, as you see from the TOP25 device list, even 2 to 3-year-old number one devices (think Samsung Galaxy S2 and Samsung Galaxy S) have such a huge installed base that they will hang on the list of most used devices for years.
This blog post covered how many and which devices you need to select to your in-house device lab if you aim for maximum market coverage. Next week we will look into the required infrastructure i.e what other hardware and software you need for your enterprise-grade device lab.
Learn best practices from this guide to maximize the ROI by building a flawless in-house test lab.Download