KitKat vs. iOS8 – Quick Glance at App Robustness

Bitbar, the mobile devops company. Logo, large

Dear Testdroiders,

While waiting mass adoption of Android 5.0 Lollipop on all possible Android devices out there, we’ve gathered some statistics about iOS8 and KitKat on our Testdroid Cloud devices. This summary presents a quick comparison of all test runs done at Testdroid Cloud on these two platforms.

KitKat vs. iOS8

During October, we gathered this data from Testdroid Cloud test runs that was executed by a myriad of different mobile app and game developers. In case you are not familiar with Testdroid Cloud, this online cloud-based service is a platform for mobile app/game developers and testers to make sure their apps and games work perfectly on those real devices, hosting the very same devices that those app consumers (end-users) use as well. As always, we’ll keep the data confidential – except summarizing the results!

The primary goal for these studies is to get better understanding of what are the most typical problems, issues and challenges that Android/iOS application and game developers confront with those devices they must build their stuff on. Again, this research represents a remarkable coverage of Android and iOS usage globally and shows the most obvious problems as well as the status of Android hardware and software from the application developer’s point of view.

How to interpret/read those graphs?
The findings from this study are presented as a box plot figures. In those figures darker black line is the median (50%) of failure percentage from different devices in that group. The upper line of box describes upper quartile (75%) and lower line describes lower quartile (25%). Lines connected with dashed line are maximum (on top or right) and minimum (on bottom or left). Outliers were excluded in this report.

KitKat vs. iOS8 – Comparison of App Robustness

When looking at the overall average (iOS8: 9.2% vs. Android KitKat: 7.8%), there is some difference between average and quarters. In the overall comparison, we had over 0.7 million (722,667) test runs for Android on 36 unique, different Android devices and 108,054 test runs for iOS on 6 different devices.

KitKat vs. iOS8


Done test runs represent good coverage of good coverage of different devices, specific OS versions and number of test runs (Android=722K+ vs. iOS=108K+).

It it worth of noticing that the application robustness on both platforms improved when new version was taken in use. For example, iOS 8.0 had lots of issues but version by version it got smaller and show the robustness of the platform. The same happened with Android KitKat variants: Quite quickly after 4.4, Google rolled out 4.4.2 (4.4.1 was for Nexus only).

Stay tuned! More details about app robustness on KitKat, iOS8 and Lollipop coming out soon.