On last week’s blog post we covered the first stage of getting to holiday readiness – Functional and Compatibility Testing. This week we will cover how you can take the test automation assets created in Stage 1 and utilize those in Stage 2 to ensure that your apps and server back-ends can really handle the extreme usage conditions of those Holiday Season peak days.
If you have missed the first one, you can also quickly read through the whole blog series starting from here.
Very few characteristics have as such significant impact on customer behavior as your app’s performance has. There are numerous studies that show high correlation between application’s performance and expected Customer Lifetime Value. For instance a study conducted by Compuware states that 62% of app users remove the app if it froze up. In this light it clear that testing your app’s performance and responsiveness on the most popular devices out there is a must. Performance testing covers various areas but at least these basic metrics should be covered in your performance testing:
1. Application installation time
This is very important part of the first impression that your user will get from using your app. If installation takes several minutes the chances are that you have lost the user already. Common ways to reduce installation times are splitting the application to several installation packages that are then installed on-demand and moving large resource files such as graphics and video to be loaded from the server side during app execution.
2. Application launch time
Also App launch time is very important measure in conveying a feeling of fast and smooth end user experience. On majority of the device your app should not take more than few seconds to launch and even on the slowest ones the app launch should never take tens of seconds.
3. Load times for each of the main views in your app
You can use the functional test scripts created in Stage 1 to also measure the load times of each main views of your app and to get this metric from wide variety of devices. Naturally if your app uses a lot of content loaded from server side your tests need to take this into account as well because different network conditions give different results.
4. Loading content from server side
There are several tools that can help you analyzing how your app handles network connections, data caching etc. Your application’s network usage profile has also big impact on the battery consumption of your app, which also is one common reason for users uninstalling the app from their phones. Good tools for analyzing your app’s network usage profile are AT&T ARO and Qualcomm Trepn profiler. For analyzing your mobile web site’s network usage you can use Chrome developer console and Safari Web Inspector.
5. Performance in different network conditions
It is also important to verify how your application performs under different network conditions and dropping connections to ensure that it can fail gracefully if loading of content takes longer that on typical conditions or if the network connection is lost during any transaction. Again, there are variety helpful tools and proxy servers that can can be used to simulate different network conditions for this kind of testing.
It is not enough to just ensure your app’s performance on ideal lab conditions against your development or staging server but you also need to run the same performance tests against your production server that has significant amount of concurrent load. Only this will give a realistic view on how well your app performs when there are tens of thousands or hundreds of thousands of concurrent users connected to your back end downloading content and doing transactions. Load testing is well established practice in web development and there are a lot of useful tools for generating server side user load such as JMeter and Blazemeter. The trick here is to use these to generate server side load under different load scenarios and then at the same time run your client side performance test suite to see if the client side performance tests produce different results under different server side load.
On next week we will cover the third and final stage of getting to Holiday Readiness – Proactive monitoring.
Are you – and your apps and web channels – ready for the Holiday Season peak days? Stick with us, and you’ll learn how to get it right!