Android 5.0 Lollipop – First Results on App Robustness

Bitbar, the mobile devops company. Logo, large

Dear Testdroiders,

We’ve had now nearly two weeks some Android devices with Lollipop 5.0 at Testdroid Cloud. During this time, over five thousand sessions/device runs have been done on these devices. As this doesn’t yet represent the best and the most robust statistics for comparison (against KitKat, iOS variants etc.) we wanted to share the first results and summary with you. Please notice that these are the first results, and we all know what tend to happen with new OS variants with existing apps.

Lollipop vs. KitKat vs. iOS8

It is worth noting that each and every time when a new OS version is rolled out – and this applies for both Android and iOS – existing apps tend be perform a bit more flaky on a new OS version. Typically, it requires small (or at least some) changes to keep the existing apps robust on new OS version. We’ve concluded here some statistics about prior versions and some reasons why things can go wrong here.

How to interpret/read those graphs?
The findings from this study are presented as a box plot figures. In those figures darker black line is the median (50%) of failure percentage from different devices in that group. The upper line of box describes upper quartile (75%) and lower line describes lower quartile (25%). Lines connected with dashed line are maximum (on top or right) and minimum (on bottom or left). Outliers were excluded in this report.

Three Lollipop Devices

Android Lollipop 5.0 vs. KitKat 4.4.x

In these statistics, we have included three different device models with Lollipop 5.0, including Nexus 7 and two Nexus 5 variants.

Failed test runs - KitKat ALL and Lollipop 5.0

From this figure, the difference is significant. However, the number of test runs (5,088) on Android 5.0 Lollipop devices doesn’t necessarily describe the actual robustness of the platform as all those were done on few devices (3 different models vs. 41 device models with KitKat variants). In fact, more test run data we got it become obvious that the overall average is getting smaller and more consistent.

Comparison of all variants – Lollipop, KitKat and iOS8

Just few weeks ago, we concluded some statistics about test runs on both versions – KitKat and iOS8. It seems that stats (and robustness) have not changed much with the latest test runs and differences between iOS8 and KitKat pretty much remains the same. However, as pointed out, the major difference is how Lollipop compares against these two.
Lollipop vs. KitKat vs. iOS8
We’ll get new statistics about Lollipop and comparison against KitKat and iOS8 once more test runs are done, more Lollipop updates are rolled out (other than Nexus devices) so stay tuned! In next few months we should have more complete picture of how robust Lollipop really is in the context of existing apps and games.


What do these charts tell us? Android 5.0 Lollipop is a brand new OS version, with lots of great features, new APIs and other stuff that obviously expose existing apps to some issues, and app/games devs should be aware of these changes. Naturally, it is highly recommended that existing apps and games are tested against Android 5.0 running on real devices, especially when more devices will support the new OS version.

Why is this important? Android OEMs will roll out their patches/upgrades soon for array of devices. For consumers to enjoy these upgrades on their devices, it is important app devs to see how their stuff works on all these possible variants. And that’s what we want to make easy, effortless and straightforward here at Bitbar Testing.

Happy Testing!

How to Build a Million-Download Mobile Game

Learn tips & tricks to build better games and make users’ lives more enjoyable.