Skip to content

We have already covered the first stage of getting to holiday readiness. And so far setting up functional and compatibility testing procedures should be no longer your headache. Still, there are plenty of other QA tasks. At stage 2 – performance and load testing. We’ll be using test automation assets from stage 1 to be sure that apps and server back ends can handle extreme usage conditions.

stage two - performance and load testing

Performance Testing – Plus, User Concerns – Minus

Few characteristics affect customer behavior as much as the performance of your mobile app or website. According to Google’s research, 53% of mobile users leave a website that takes longer than three seconds to load. Moreover, 79% of customers said they would not return to a website with poor performance. The situation with mobile apps is very similar. Poor performance is one of the top three reasons people uninstall apps. No doubts, fighting user dissatisfaction, especially during the holiday seasons, has a direct impact on conversion rate.

53% of visits abandoned after if mobile site takes more than 3 seconds to load

The ultimate goal of any app testing is to satisfy users and keep them as happy customers/users. Discerning consumers won’t tolerate the slightest issues related to:

  • Speed. The app should load and run quickly in both favorable and less favorable scenarios.
  • Stability. The app should be stable and run as intended, regardless of user loads.
  • Scalability. The app should perform well, even if the number of users increases dramatically.

That is why performance testing should cover device, network, server, and API performance criteria. We’ve gathered a checklist of basic metrics to verify:

1. Application Installation Time

App installation is synonymous with the first impression users get of an app. At this stage, the task is to make sure that the app components are well-coordinated. Depending on the app’s complexity, there are various approaches to verify the app installation process. Consider looking at silent installation, attended/unattended mobile app installation, network mobile installation as well as clean or automated mobile installation. Other common ways to reduce installation times are splitting the application to several installation packages that are then installed on-demand and moving large resource files such as graphics and video to be loaded from the server-side during app execution.

2. Application Launch Time

Another critical performance parameter is the application set up time. The app launch time is a very important metric in conveying a feeling of fast and smooth end-user experience. It should take not more than 1-2 seconds for the first screen to appear. The longer it takes, the bigger the bounce rate.

3. Load Time for Each of the Main Views in Your App

Use the functional test scripts created at stage 1 to measure the load time of each of your app’s main views across a wide variety of devices. A lot of content loaded from the server-side as well as different network conditions will have a direct impact on these parameters.

4. Loading Content from the Server-Side

Handling data efficiently means it doesn’t take long to load data sent by a server. For instance, there are cases with certain types of apps when data is transmitted in a specific format. So, before displaying it in the app, it’s first converted to a suitable format. This takes extra time. Another possible issue is the server being down or unreachable. Make sure there are failover or backup servers synchronized with the main one.

5. Performance Under Different Network Conditions

It is also important to verify how your application performs on a variety of networks with variable speed and possibly dropping connections. Test the app against 2.5G, 3G and 4G networks, low-latency, and high-speed networks. QA specialists have to consider this fact when drafting tests and analyzing the results. They should also monitor the behavior of the app when both Wi-Fi and mobile networks are turned, especially when a user switches between them.

6. Working in the Background and Using Other Apps Alongside

Make sure there is no interference in-app performance when several apps run in parallel. When the app runs in the background and is retrieved, make sure no data is lost. Moreover, while it works in the background, it should not consume memory and heat the device.

7. Battery Time and Memory Consumption

You need to verify the extent to which the app consumes the amount of battery life and at which point it starts to heat the phone. Make sure that the app doesn’t use more resources than it’s required. Look for the features that drain the memory exceptionally and try to handle them before releasing the app in production.

Load Testing – Why, When and How

Load testing aims to find out how the server infrastructure will perform and if its performance is good enough while dealing with an avalanche of requests from multiple users. Believe us, multiple users are exactly what each app producer expects during the holiday season. On top of that, with the help of mobile load testing, QA specialists investigate dependencies between user requests and the response time of an application.

Why is Load Testing Critical?

When was the last time your development team asking questions like “Are we ready to handle a heavy number of concurrent visitors?” or “Will my system be able to stay up and running under peak hours and seasons?” Yes, you hear me right. The goal of load testing is to ensure that your website or mobile application can perform the way it should under some extreme conditions.

Today’s businesses have so many differences in how they operate, but there’s only one same result when downtime happens – revenue loss or cost increase. Gartner pointed out in 2014 that a certain IT downtime creates an average cost of $5,600 per minute to businesses. And depending on different industries, IT setup, infrastructure, etc. companies with a downtime may end up suffering an average cost between $140,000 per hour and a whopping %540,000 per hour.

Load testing can unveil the bottlenecks in how your business can serve a varying number of users at any given time. It helps to mitigate business risks, be it of speed, stability or scalability, saves money and indirectly generate revenue to the businesses. There is no doubt, in general, the earlier you think of load tests for your apps, the better performance you’ll get during holiday traffic. Running load tests ensures that you already know your limits, understand your target audience and the complexity of your app. But there’re some specific scenarios in which you should determine how many users your system can handle.

We suggest integrating load testing as part of your deployment strategy and running smaller load tests before or during deployment. Then, it’s better to schedule larger load tests to better understand how your app is performing, its bottlenecks and breaking points under peak loads. Choosing some of the load testing tools, e.g. LoadNinja, can help with that. Modern tools offer enough customization and a wide selection of parameters to design extensive and personalized load tests. The outcome of such load testing can be thoroughly analyzed in terms of native and mobile web apps performance.

So far we hope you’ve got a sense of why, how and when to do performance and load testing. In addition to what we have created and learned in Stage 1 for functional and compatibility testing, we’ve touched upon two major topics. Don’t stop halfway; get ready for holidays properly. Go to Season’s Readiness Stage 3 – proactive end-to-end monitoring!

Jouko Kaasila

Bitbar COO & Co-founder