On last week’s blog post we covered the first stage of getting to holiday readiness – Functional and Compatibility Testing. This week we will cover how you can take the test automation assets created in Stage 1 and utilize those in Stage 2 to ensure that your apps and server back-ends can really handle the extreme usage conditions of those Holiday Season peak days.
Very few characteristics have as such significant impact on customer behavior as your app’s performance has. There are numerous studies that show a high correlation between app performance and expected Customer Lifetime Value. For instance, a study conducted by Compuware states that 62% of app users remove the app if it froze up. In this light it clear that testing your app’s performance and responsiveness on the most popular devices out there is a must. Performance testing covers various areas but at least these basic metrics should be covered in your performance testing:
1. Application installation time
This is a very important part of the first impression that your user will get from using your app. If installation takes several minutes the chances are that you have lost the user already. Common ways to reduce installation times are splitting the application to several installation packages that are then installed on-demand and moving large resource files such as graphics and video to be loaded from the server side during app execution.
2. Application launch time
Also the App launch time is a very important metric in conveying a feeling of fast and smooth end user experience. On the majority of the devices, your app should not take more than few seconds to launch and even on the slowest ones the app launch should never take tens of seconds.
3. Load times for each of the main views in your app
You can use the functional test scripts created in Stage 1 to also measure the load times of each main views of your app and to get this metric from a wide variety of devices. Naturally if your app uses a lot of content loaded from server side your tests need to take this into account as well because different network conditions give different results.
4. Loading content from server side
There are several tools that can help you analyzing how your app handles network connections, data caching etc. Your application’s network usage profile has also a big impact on the battery consumption of your app, which also is one common reason for users uninstalling the app from their phones. Good tools for analyzing your app’s network usage profile are AT&T ARO and Qualcomm Trepn profiler. For analyzing your mobile web site’s network usage you can use Chrome developer console and Safari Web Inspector.
5. Performance in different network conditions
It is also important to verify how your application performs under different network conditions and dropping connections to ensure that it can fail gracefully if loading of content takes longer than on typical conditions or if the network connection is lost during any transaction. Again, there are a variety of helpful tools and proxy servers that can be used to simulate different network conditions for this kind of testing.
It is not enough to just ensure your app’s performance on ideal lab conditions against your development or staging server but you also need to run the same performance tests against your production server that has a significant amount of concurrent load. Only this will give a realistic view of how well your app performs when there are tens of thousands or hundreds of thousands of concurrent users connected to your back end downloading content and doing transactions. Load testing is well-established practice in web development and there are a lot of useful tools for generating server-side user loads such as JMeter and Blazemeter. The trick here is to use these to generate server-side load under different load scenarios and then at the same time run your client-side performance test suite to see if the client side performance tests produce different results under different server-side load.
On next week we will cover the third and final stage of getting to Holiday Readiness – Proactive monitoring.