Skip to content

The operating system has a crucial role in all development going on Android and iOS. The applications – whether considered as more traditional applications, gaming apps, hybrid or web apps – are the beef for end-users to get everything out of their devices. All those mobile apps play an essential role in determining if devices, OEMs, the platform itself and the whole ecosystem can flourish.

In this blog, we’ll take a look at Bitbar’s data and analyze it a bit to understand why operating system versions are an extremely important factor, and why app devs should take every update seriously. Harshly, you know, the OS update can make or break the revenue generation from your app.

The War of Mobile Apps and Ecosystems

In the early 2010s, there was a war of mobile ecosystems. From the retrospective point of view, how did all that play and where are we today? In 2010, Android has quickly got traction with OEMs, Apple pushed its iOS forward and entrenched its app ecosystem with new applications and allured more app devs to jump on bandwagon. However, the number of Android devices by a variety of different OEMs made Android also sexy among app devs, and more devs jumped on their (or both) bandwagons.

We all also remember the famous “burning platform” memo by Stephen Elop back in early 2011 – and as many of us mobile insiders thought about, there was a truth in that metaphor, however, the resolution and pick for a new platform was completely wrong (and yes, everyone who was seriously in to mobile business knew it). What would have happened if the giant was set to go Android waters? That’s speculation and we’ll skip that part for now.

Beside of Windows, there were few others that were poised to be the next significant mobile OS platforms for apps, app developers, OEMs and the whole mobile ecosystem. However, Android and iOS are the ones that remained popular – and for good reasons. App developers took both of them well and both Google and Apple got traction around the ecosystem and good incentives for developers to build their stuff on these platforms.

Going back to the comparison of these two, in general, the number of available Android apps exceed the number of iOS apps around autumn 2012 – for example, as shown in this statistic by XYLOGIC:

xyologic report about mobile apps

Since 2013, we’ve seen two major platforms become a standard (Android and iOS) for mobile apps that are very different from the app developers point of view.

What Role Does the OS Version Play in Fragmentation?

There are lots of things that every app developer should know about Android. For example, the “F” word – fragmentation – is widely used to describe how different apps work and are compatible with a variety of different devices. In the case of Android, there are more contributing factors for fragmentation (than with iOS): OEM branded devices (hardware), different adoption of the OS, different hardware specification and few others.

In the past, Android OS has been blamed for the platform fragmentation, making things very difficult for developers and users to keep up with upgrades, new tools and so on. This is still the case and nowadays the major contributing fragmentation factor is the OS version. Yes, few other factors such as OEM customizations, integration and compatibility with other software (e.g. backends, services) cause issues for apps.

Android fragmentation in 2017

On the hardware side of things, many things actually cause issues for developers and users. The display, its resolution, quality factors and portrait/landscape use with apps are the most common display-related problems that both app devs and users see. The other notable root cause for fragmentation or simply not getting apps working properly on devices is the memory. Devices that do not have enough memory (RAM) may work badly, flaky or simply can’t perform well enough.

The above picture illustrates various different characteristics that cause issues for developers and users. Let’s take a look at how operating systems work differently when the same hardware set up is used. These figures and generally this data contains the test run data from our Bitbar Public Cloud (aka Testdroid Cloud) from the last year and thousands of apps have been used to validate the data.

Android OS Versions Comparison

Android OS versions compared

Comparing Android OS versions against each other is straightforward when the same application is used. The devices under test should be identical with their hardware factors and because of this, people doing testing of their apps should always use real mobile devices for testing. The emulated/simulated environment doesn’t produce comparable or trustworthy results how applications actually work in hands of end-users.

iOS Versions Comparison

iOS fragmentation and OS version comparison

Comparing iOS versions is no different from Android versions. Applications are executed on the same device with different OS version and basically with the same test script or smoke test run.

The Importance of Mobile OS Version for App Developer

When looking at these statistics it’s quite obvious that the difference of average failure rate is pretty close with both of these platforms. With Android, the average failure rate is 17.8% and with iOS it’s only slightly lower, 15.5%. And despite there is dispersion between different OS versions the trend is typically this: more bugs have been fixed on new sub-version of that specific OS version. For example, every minor release of Android and iOS OS reduces the number of issues that app developers see with their app.

When the new major OS version is released it typically increases the failure rate and there are various reasons why this actually happens. New APIs, features, even OEM customizations and upgrades provide flakiness to test foundation. As this is not “vanilla-Android” OS versions that we compared with Android devices, OEMs typically provide those OS versions (with their own stuff).

If you are looking for trends in mobile app development and testing remember to keep yourself updated with the latest news on test automation field. There are constantly lots of things going on and new versions coming out to support your test automation effort with these great mobile platforms.

We’ll deep-dive more in details with these failure rates, statistics and comparison – so stay tuned!

Happy Testing Folks!

Ville-Veikko Helppi

Mobile Testing Product Expert