According to International Data Corporation (IDC), the worldwide smartphone market grew 13.0% YoY (year over year) in the second quarter of 2015. The total number of shipped smartphones was 341.4 million, and the leading vendors (in terms of shipped device volume) were the same as past quarters: Samsung (Android) as largest and Apple (iOS) as second, followed by Huawei (Android). While Android dominated the market with an 82.8% share, Apple got approx. 13.2% with its iOS devices, and others roughly 4.0%. Looking at these numbers there raises undoubtedly a question how relevant is iOS app/game testing and what devices are the must for your test roster? Let’s take a look at it.
When it comes to test automation, the entire iOS app testing ecosystem is still very much emulator/simulator-driven with a strong focus on using beta-testing services, but automating testing on real devices has quickly climbed up to be considered as the top priority for many of the most popular mobile app and game developers. And there are plenty of good reasons why iOS app and game developers should be looking into this. Download our free guide to get started with XCTest and create reliable tests for iOS 10 app testing.
Looking at the data by IDC, Apple is the second largest manufacturer of mobile devices, and has several popular smartphones and tablets in this dataset. According to Unity3D report, iPhones comprise 60% of all iOS installs, compared to 36% for the iPad and 4% for the iPad Touch. When it comes to generic apps – and not just mobile games powered by Unity – this is pretty much inline of iOS device volumes used for purchasing, installing and using iOS apps.
What OS Versions Are A Must for iOS App Testing?
According to Mixpanel the iOS9 adoption has grown to 79.85% (data fetched on the 10th of December 2015). This data includes all versions of iOS 9, including iOS9 9.2, iOS 9.1, iOS 9.0.2, iOS 9.01 and 9.0 beta. Despite close to 80% is a significant majority of all iOS devices, the remaining users
A good rule of thumb with iOS app testing is to focus on the latest iOS version but keep the prior ones part of the testing roster until adoption is less that 5% among total devices. Furthermore, Apple has a pretty good track record of delivering new iOS versions (upgrades) that fix issues for the prior versions. For example, the latest data from Testdroid Cloud test runs on iOS 9 devices yields that the trend of failures has gone significantly down when the same app has been tested on several versions of iOS 9.
Again, the good news is that Apple reacts quickly to these sort of bugs by releasing a new upgrade version. Here is what happened with the tests on iOS 9 variants, and how things started getting better after new minor versions were rolled out and used by app developers:
If we map this is in timeline, the first official iOS 9 version was published on September 16th (however, the first preview/beta was available after WWDC on June 2015) and in next two weeks Apple pushed out two versions – the iOS 9.0.1 on September 23rd and iOS 9.0.2 on September 30th. Next versions came out October 21st (9.1) and earlier this week we saw 9.2. The data from iOS 9.2 test runs is yet not comparable with the prior versions, so we’ll get back to that later when more testing and data can be extracted from those test runs.
Top 10 Devices for iOS App Testing
In order to cover the user volume, different types of hardware specifications (display sizes, chipsets etc.) and different iOS versions, you should consider the following devices for your testing. This can potentially provide you over 90% coverage of all iOS devices (in volume), plus the best testbed to spot issues in your iOS apps or games.
Here is the list of the most important iOS devices to be used for your app testing:
- iPhone 6S (Global / A1688) with iOS 9.x
- iPhone 6 Plus (Global / A1524) with both iOS 9.x and iOS 8.x
- iPhone 6 (Global / A1586) with both iOS 9.x and iOS 8.x
- iPhone 6S Plus (Global / A1687) with iOS 9.x
- iPad Mini 3 (Global / A1599) with both iOS 9.x and iOS 8.x
- iPhone 6 (GSM / North America / A1549) with iOS 9.x
- iPad Air 2 (WiFi / Cellular / A1567) with both iOS 9.x and iOS 8.x
- iPad Mini 2 (WiFi only / A1489) with both iOS 9.x and iOS 8.x
- iPhone 5C (GSM / North America / A1532) with both iOS 9.x and iOS 8.x
- iPhone 5S (GSM / North America / A1533) with both iOS 9.x and iOS 8.x
Sure, there are lots of other variants that can be as effective as these listed ones. For example, the new iPhone 6S are shipped as A1633, A1688, A1700, A1634, A1699 as well as mentioned A1687 in the top list. Many of these variants are geographically targeted/shipped in different countries but may contain very similar testbed. However, we recommend using these listed devices to get the maximal impact and use those devices that are used by masses. More importantly, these devices are gathered from regions (commonly ‘Global’ or ‘North American’ version) as these people do purchase apps and do in-app purchases that can improve your revenue generation.
How to Test Your iOS App and Game on These Devices?
The iOS app testing is typically done either manually, using beta-testing services with real people doing the testing (still manual testing) or test automation. When automating your iOS app testing and for any iOS app with some degree of complexity, you will have to write test scripts and automate runs of those on – preferably – these real devices. Simulators do provide some help, but there a bunch of good reasons why testing should be always done on real devices.
When it comes to test automation on iOS devices, there three main types of automated tests: end-to-end tests, unit tests and strictly UI tests. The end-to-end testing is the type of full system test where a real user is simulated and app is driven by the test script. The test script drives the app and checks how all different components of it work together. This type of testing involves launching the app on real devices (simultaneously on number of different device variants) and automatically perform certain user actions (e.g. clicks, swipes, even gestures) to drive the app like it would be done by a real person. This may include logging into app/service, purchasing some inside the app (in-app purchases) and generally testing how app reacts to user interactions. All interactions and progress is always captured as logs, screenshots, performance stats and results, and even live video of the test session.
Test automation is also used despite of different display and hardware differences. For example, test script knows the size of the resolution and can work its way to identify certain elements based on IDs, descriptions, classes etc. to identify that specific button/UI element. Furthermore, the image recognition used for mobile game testing is gaining popularity and lots of game developers built their test automation to include image recognition/OCR functionality. This makes the testing super easy on any of these device variants and variety of different devices can be executed simultaneously. If you are interested in automating your game testing, check this out.
How to Automate Website Testing on iOS Devices?
One popular test automation framework for mobile website testing is Selenium. The open-source Selenium project has become the de-facto standard for scripting and running end-to-end tests in the non-mobile automated testing world, but it used more and more also on mobile test automation. For example, Appium – that relies on Selenium WebDriver – is quickly becoming popular and is used for mobile test automation for website and web-related things.
Automating mobile web tests on different iOS devices using Safari will be a topic of its own, and this adds another dimension to generic test automation discussion. We’ll be covering that in our step-by-step blog series about Appium so please check it out and stay tuned for more tips and tricks!
In case you are interested to start right away, do check this one and start testing your websites on real devices and real browsers, just in matter of minutes.
Happy iOS App Testing!
Get all essentials about XCTest framework and learn how to get started with it for cloud testingDownload