All mobile enthusiasts are getting (again) lots of things to be excited about – but also things to think about. New versions of Android (6.0 Marshmallow) and iOS (9.0) are about to take off on some new devices and also on a huge set of existing devices out there in pockets of hundreds of millions of users. Naturally, all this excitement of having all those great new features, APIs, services, and devices among new OS versions will also bring some challenges for Android and iOS app developers. How will the testing evolve, what things you have to tweak in your existing tests and app itself?
Background and New ‘3 Months Trial’ for Developers
What has been fundamentally different from prior releases of these OS platforms is that both of them have had several ‘Developer Preview‘ or ‘Beta‘ versions before the official release. Those official releases may still take some time until we can call those being in majority of gadgets out there, but nevertheless, these pre-official versions have provided a great intro session for developers to get familiar with all those great new features and APIs (as well as tool and dependency changes). Looking back, Android 6.0 Marshmallow early variant was unveiled at Google I/O at the end of May, and after 2.5 months it was confirmed as ‘6.0 Marshmallow’, soon to be available for everyone.
Pretty much the same happened with the new variant of iOS. The iOS 9 debuted at the Apple WWDC on early June offering an early version of beta for developers right after the conference keynote. Again, after 3 months, the whole Apple community will finally have an access to iOS 9 with their devices.
This approach by both Google and Apple to launch their new major OS versions yields that application developers should be really taking a better look at their existing apps when upgrading support of those for these new platforms. It simply doesn’t work out-of-the-box in all cases – and that’s where agile approach with continuous development and testing will provide you the best results.
What to Expect from New Major OS Versions
Every now and then, we’ve been generating and publishing data out of test runs conducted in our Testdroid Cloud service. For example, during the past month (August) there were over 10 million test runs, with nearly million unique test cases, and average test execution time was 3 minutes and 22 seconds. The pass vs. failure is reported as if the test case found something or something went wrong with the execution, and the test returned a failure status. There are lots of different reasons why test case fails, but let’s look at the statistics first.
The Android ‘M’ has got quite a lot data during the past two weeks and the very same conclusion can be drawn from this data what typically happens with the new major OS versions: the average failure rate goes up. If you think about it – you are running the very same APK on different OS versions and you’ve probably have run it on those older variants for some time, and then you start seeing issues on the new version. As said, there are lots of different reason why test fails.
Furthermore, the very same trend of minor versions getting more stable and the new, major versions introducing new things that can make existing implementations behave flaky is something we’ve found out in these test runs. It’s very likely that once developers will adopt some new APIs or will change and fix their apps to work on Android M – and also iOS 9 – this failed test run ratio will come down. However, it doesn’t happen itself but developers have to jump on those apps to fix those issues.
What Are People Using Today?
According to Android Dashboard approx. 21 % of Android users are now on Lollipop (5.x) and the largest user-base still remains on KitKat (39.2%). Also, the use of Jelly Bean is also still significant (31.8% in total) so it is very unlikely that majority of users will jump on Marshmallow bandwagon on its first days/weeks. Our testing recommendation is to include all new testing patterns for Marshmallow and keep the existing tests ready with these older variants, on Jelly Bean, KitKat and Lollipop.
Comparing Android to iOS, things are way more different on Apple devices. According to App Store Support pages majority of iOS users already use iOS 8 and – we believe – are likely to upgrade to the latest and greatest as soon as it comes publicly available. Naturally, our testing recommendation for testing strategy is try to adopt all the necessities to get iOS 9 devices thoroughly tested, as soon as those are available. Things may require a bit more attention (than on Android side) as you may need to update your Xcode, plus some other tools.
As always, we’ll stay tuned with the official versions coming out and will make the environment ready for you to develop and test. Preliminary schedule for official Android M is now at the end of September (29th) and for iOS 9 it should happen as soon as next week (16th).
PS. Let us know what you think about these new versions or if you have any concerns of how to make your apps work perfectly on these. We’re looking forward for your comments in section below.