There are three important steps to ensure your mobile apps and mobile web presence are really ready for the crunch time of the Holiday Season. We also laid out the timeline for going through these stages to really maximize your revenue potential.
This week’s blog post will cover the first and the most important stage of preparedness that will lay out the groundwork for all other stages as the outputs of this stage (the actual test scripts) will be used in all other stages as well. Therefore cutting corners here will significantly limit the benefits you reap from the following stage. Timeline wise the first stage: Preparing your functional and device compatibility test suite must be started about 10 weeks before the peak days at the end of November. This will leave enough time for several iterations to iron out all critical defects found in this stage and time to implement the next stages: Performance & Load testing and Proactive end-to-end monitoring.
Preparing functional test automation suite
The very first stage of preparedness is to ensure that any new version of your app or web service is thoroughly tested for all basic functionality. We recommend building a comprehensive test automation suite built for both native apps and your web channel with mobile access in mind. Design the suite so that it can be successfully executed against a wide variety of real target devices and verify this by using real devices. There is really nothing that wastes more of the value of test engineering than a test suite that produces different results on different devices. This stage can take anywhere from 2 to 6 weeks depending on desired functional and device coverage, the number of issues uncovered during this testing cycle and the release velocity to fix uncovered issues.
What needs to be covered by your test automation suite
The starting point is that your functional test suite must cover the most common use cases for your app or mobile site. Typically these include:
1. Installation of the app
2. Uninstallation of the app
3. Upgrading from the previous version of the app to the latest
4. Creating a new account
5. Making any payment transactions in the app
6. Upgrading and downgrading the subscription level (if you operate on subscription levels)
7. Ensuring that the integrations to other services work (Facebook, Twitter, Google, etc)
8. Checking that user can navigate through the most common navigation paths
9. Checking visual correctness of each view/page
10. Ensuring that all reporting that app does is working and produces right kind of data to the server side
When implementing the test suites it is important to make each test atomic i.e. so that they can be run individually. This will be very valuable in Stage 2 and Stage 3 when the same suites can be used in Load & Performance testing and Proactive end-to-end monitoring without the need to run all test cases in the same sequence every time.
Selecting Top Target Devices
Once you have your functional test suite in place and producing robust and repeatable results on a wide variety of real devices for your native apps and mobile websites on main platforms (at least Android and iOS) it is time to define a set of target devices where everything must work 100%. This list varies from year to year and it should contain:
1. Most popular Android & iOS devices from this and previous year
This is a good starting point for building up your target device list as device compatibility issues are more prevalent on Android due to a number of different device manufacturers, form factors, OEM specific customizations and different OS versions in the market. For iOS, it is equally important to cover the latest popular models as well as the blockbusters from the past few years. The lifetime of each iOS device is much longer than that of an Android device so there is still a large number of 2- or even 3-year-old iOS devices in daily use.
2. Good mix of hardware and software
A lot of the content during the holiday season is used from tablets and therefore it is important to ensure that your work on these devices with different screen sizes. In terms of OS versions, it’s recommended to ensure a good mix covering at least 3 latest major OS versions for both Android and iOS.
3. All expected Holiday Sales blockbuster devices
As the Holiday season is a very important time for the sales of new mobile devices remember to include the expected blockbusters for each holiday season as these will be the devices that will run your latest app and access your web site.
4. Different browsers on each device
One often overlooked characteristic when testing for mobile web compatibility is that users access websites from a variety of browsers. Naturally this is more common in Android platform than in iOS but from the mobile browser market shares, you can see that you need to cover at least Chrome and stock Android browser for all Android devices while on iOS side testing only on Safari should be enough.