App Stability on Android vs. iOS (before iOS8)

Android vs. iOS

Dear Testdroiders,

As iOS8 just got out and Kitkat adoption is getting more popular, let’s look back into some statistics about older variants on both platforms. In this summary, we gathered failures from Testdroid Cloud test runs and categorized those under each OS version (by codename on Android and generally all on iOS). The data presents fair comparison of both platforms and decent number of test runs on each OS version. Take a look at this summary!


During August, we gathered a significant amount of data from Testdroid Cloud test runs executed by a myriad of different mobile app and game developers. In case you are not familiar with Testdroid Cloud, this online cloud-based service is a platform for mobile app/game developers and testers to make sure their apps and games work perfectly on those real devices, hosting the very same devices that those app consumers (end-users) use as well.

The primary goal for this research was to understand what are the most typical problems, issues and challenges that Android application and game developers confront with those devices they must build their stuff on. Also, this research represents a remarkable coverage of Android and iOS usage globally and shows the most obvious problems as well as the status of Android hardware and software from the application developer’s point of view.

How to interpret/read those graphs?

The findings from this study are presented as a box plot figures. In those figures darker black line is the median (50%) of failure percentage from different devices in that group. The upper line of box describes upper quartile (75%) and lower line describes lower quartile (25%). Lines connected with dashed line are maximum (on top or right) and minimum (on bottom or left). Circle describes an outlier.

Android vs. iOS – Head-to-Head

When looking at the overall average (iOS: 4.1% vs. Android: 4.7%) the difference isn’t significant between these two platforms. However, the difference is significant when comparing quartiles (25% and 75%) as it represents all possible test cases and test runs done on those devices. In the overall comparison, we had over 5,7 million (5,772,491) test runs for Android on 309 different Android devices and 0,7 million (728,560) test runs for iOS on 17 different devices.

android vs. ios

Naturally, there are lots of different reasons why apps fail when executed on different devices, configurations etc. As we want to give as neutral and correct insight to these tests, it is worth noticing that the device includes 100% of iOS usage and over 95% of Android volumes as well.

Android variants vs. iOS

Based on the first graph, Android seems more problematic than iOS, however, that is not the case with all different Android OS variants. If we look comparison of overall iOS and Android results and compare those to different Android variants we see that for example Jelly Bean and Icecream Sandwich have been more stabile than iOS in average. Those older variants – starting from Froyo – are definitely more problematic but they are not widely used anymore.

Android vs. iOS

The most surprising finding is that Kitkat has more problems than those prior versions. We’ve covered some things about this here.

Differences between Android and iOS variants

Then, let’s look at comparison between different Android and iOS variants:

OS variant comparisons



Apple vs. Android OEMs

And then, a holistic view to OEMs – Apple vs. other Android OEMs.