On last week’s blog post on Getting to Holiday Readiness we introduced three important steps to ensure your mobile apps and mobile web presence are really ready for the crunch time of the Holiday Season. We also laid out the timeline for going through these stages to really maximize your revenue potential.
This week’s blog post will cover the first and the most important stage of preparedness that will lay out the groundwork for all other stages as the outputs of this stage (the actual test scripts) will be used in all other stages as well. Therefore cutting corners here will significantly limit the benefits you reap from the following stage. Timeline wise the first stage: Preparing your functional and device compatibility test suite must be started about 10 weeks before the peak days at the end of November. This will leave enough time for several iterations to iron out all critical defects found in this stage and time to implement the next stages: Performance & Load testing and Proactive end-to-end monitoring.
Preparing functional test automation suite
The very first stage of preparedness is to ensure that any new version of your app or web service is thoroughly tested for all basic functionality. We recommend building a comprehensive test automation suite built for both native apps and your web channel with mobile access in mind. Design the suite so that it can be successfully executed against a wide variety of real target devices and verify this by using real devices. There is really nothing that wastes more of the value of test engineering than a test suite that produces different results on different devices. This stage can take anywhere from 2 to 6 weeks depending on desired functional and device coverage, the number of issues uncovered during this testing cycle and the release velocity to fix uncovered issues.
What needs to be covered by your test automation suite
The starting point is that your functional test suite must cover the most common use cases for your app or mobile site. Typically these include:
1. Installation of the app
2. Uninstallation of the app
3. Upgrading from previous version of the app to latest
4. Creating new account
5. Making any payment transactions in the app
6. Upgrading and downgrading the subscription level (if you operate on subscription levels)
7. Ensuring that the integrations to other services work (Facebook, Twitter, Google, etc)
8. Checking that user can navigate through the most common navigation paths
9. Checking visual correctness of each view/page
10. Ensuring that all reporting that app does is working and produces right kind of data to the server side
When implementing the test suites it is important to make each test atomic i.e. so that they can be run individually. This will be very valuable in Stage 2 and Stage 3 when the same suites can be used in Load & Performance testing and Proactive end-to-end monitoring without the need to run all test cases in same sequence every time.
Selecting top target devices for 2015 Holiday Season
Once you have your functional test suite in place and producing robust and repeatable results on wide variety of real devices for your native apps and mobile web sites on main platforms (at least Android and iOS) it is time to define a set of target devices where everything must work 100%. This list varies from year to year and it should contain:
1. Most popular Android devices from this and previous year
This is a good starting point for building up your target device list as device compatibility issues are more prevalent on Android due to number of different device manufacturers, form factors, OEM specific customizations and different OS versions in the market. The core of this list can be selected from Testdroid’s Top 25 Android Devices to Test On In 2015. On top of the list you see devices like Samsung Galaxy S5, Samsung Galaxy S4,Samsung Galaxy S3, LG G3, LG Nexus 5, Samsung Galaxy Note 2, Samsung Galaxy Note 3 and Samsung Galaxy Note 4.
2. The latest iOS devices plus some older ones that are still in use with hundreds of millions of users
On iOS land it is equally important to cover the latests popular models as well as the block busters from the past few years. The lifetime of each iOS device is much longer than that of an Android device so there are still large number of 2 or even 3 year old iOS devices in daily use. The most popular iOS devices of the past year are iPhone 6, iPhone 6 Plus, iPad Air 2 and iPad Mini 3 while the most common iOS devices out there are still iPhone 4S, iPhone 5S, iPad Mini (different versions) and iPad (different versions).
3. Good mix of phones and tablets
A lot of the content during the holiday season is used from tablets and therefore it is important to ensure that yours work on these as well. The most popular tablets in second half of 2015 are: iPad Air, iPad Mini 2, Samsung Galaxy Tab 4, Samsung Galaxy Tab S, HTC Nexus 9 and Asus Nexus 7.
3. All expected Holiday Sales blockbuster devices
As the Holiday season is very very important time for the sales of new mobile devices remember to include the expected blockbusters for 2015 Holiday season as these will be the devices that will run your latest app and access your web site. The list of this year’s must have devices include:Samsung Galaxy S6 Edge Plus, Samsung Galaxy S6, Samsung Galaxy Note 5, iPhone 6S, iPhone 6 and iPad Air 2.
4. Good mix of different OS versions
Make an effort to check that your target list covers at least 3 latest major OS versions for both Android and iOS. For iOS that means iOS 9 (expected launch mid-September), some version of iOS 8 and some version of iOS 7 and for Android that means Android 6 (release in Q3), some version of Android 5 and some version of Android 4.
5. Different browsers on each device
One often overlooked characteristic when testing for mobile web compatibility is that users access web sites from variety of browsers. Naturally this is more common in Android platform than in iOS but from August 2015 Mobile browser market shares you can see that you need to cover at least Chrome (35% of total) and stock Android browser (12% of total) for all Android devices while on iOS side testing only on Safari should be enough.
Recommended target device list for 2015 Holiday Season is:
Samsung Galaxy S6 Edge Plus
Samsung Galaxy S6
Samsung Galaxy S5
Samsung Galaxy S4
Samsung Galaxy S3
LG Nexus 5
Samsung Galaxy Note 2
Samsung Galaxy Note 3
Samsung Galaxy Note 4
Samsung Galaxy Note 5
HTC Nexus 9
Samsung Galaxy Tab 4
Samsung Galaxy Tab S
Asus Nexus 7
iPhone 6 Plus
iPad Air 2
iPad Mini 3
iPad Mini 2
iPad 4th Gen
Next week: Stage 2: Load and Performance Testing
On next week’s blog post on 2015 Holiday readiness preparations we will cover Stage 2: Load and Performance Testing for your mobile apps and web sites.