Guest Blog: The Best of Both Worlds – Combining Performance and Functional Mobile App Testing

Bitbar partners with Neotys to combine performance and functional testing for mobile apps

We are excited to announce the partnership with Neotys, the maker of Neoload. The partnership will combine Neotys’s expertise in mobile performance and load testing with Bitbar’s market leading mobile functional testing capability and real device farm to deliver a solution to today’s growing complexity of challenges in mobile performance and functional testing.

Today, we have Henrik Rexed, Mobile Performance Engineer at Neotys shed some lights on the challenges in modern mobile performance testing and how to overcome them to bring the best mobile UX to customers. Before today’s guest blog, we would like to invite you to join our co-hosted webinar with Neotys on Wednesday, August 30th. In this webinar, we will go through the value of the joint force and demonstrate how you can actually combine performance and functional mobile app testing as a daily practice. Watch this webinar now!

What are the most critical factors of mobile app success, given that you are building an app your customers want? User Experience.

While everyone knows the importance of delivering a pleasant user experience to strive for mobile success, have you really thought about what happens when your app performance is sub par? The key findings of a research showed that

  • Users are impatient
    • 61% of mobile app users expected apps to start in 4 seconds or less
    • And 49% of them expected apps to respond in 2 seconds or less
  • Users are intolerant of issues within mobile apps
    • 80% of users will only try a problematic app three times or less
    • 53% of them have ever uninstalled a mobile app with crashes or errors

Tough facts, right? Over the past years, I’ve noticed that many companies or teams have realized the cost of poor mobile UX and been putting more emphasis on improving mobile user experience by continuously testing mobile app performance and improving app quality. But many of mobile engineers are still confronting a couple of challenges in mobile performance testing. Let’s break them down.

Challenge #1: Mobile devices and apps

Since each mobile device is differently connected to the server, the app content and data can be presented in a different way on devices with varying hardware settings and software configurations. On top of that, different types of mobile apps (native, web & hybrid) interact with back-end in a distinct way due to the fact that they are built and processed differently. The combo of growing mobile devices and different app types create a huge challenge for companies to optimize app performance for every device model, let alone parallelizing automated tests on the majority of real devices to maximize device coverage.

Challenge #2: Network access

Network characteristics have significant impacts on app performance and user experience. Even with the same network connectivity technology, e.g. 4G, the response time varies a lot because of different locations, network carriers, user’s surroundings (high population density vs. low population density).

While the bandwidth of networks is commonly deemed as the main challenge for mobile performance testing, Ilya Grigorik made an excellent study showing that the main challenge is the latency.

“Yes, the equipment can be improved to shave off a few milliseconds, but if you want significant improvements, then the answer is simple: you need new, shorter cables to reduce the propagation delay. As you may have guessed, this is an expensive proposition:”

On the other hand, the network also affects your entire architecture. Typically with a slower network connection, it takes a longer time for a user to achieve business transactions. In that case, if all of your users are on slow internet connections, then it means that you will have more concurrent sessions on the architecture. So testing without the network constraints will hide problems to the tester. That’s why you need to consider the real constraint of the end-user when you are trying to validate the behavior of your application

Challenge #3: Backend service

Many companies nowadays have both web app and mobile app to tackle different sources of traffic and users. There is no problem with that unless if your mobile app and web app share the same back-end. Imagine that when your web app gets a heavy load with many users at one time, your mobile app users most likely will experience a slow speed or bad app performance in general.

For example, if you have an eCommerce website and app, the best practice is to create a load test to check if the backend service correctly handles purchasing the product in your shopping cart for mobile app users when there are 100,000 users on your web app or website.

Load testing for backend service

Challenge #4: UX measurement

While it is a common practice to use emulators and simulators for load generation in app performance testing, one cannot rely on them to measure the actual end-user experience. When optimizing app performance, you should take into account metrics like CPU, memory, battery, rendering times, etc. which can only be directly pulled from real devices.

In addition, native apps or hybrid apps utilize API or services from the backend to collect data and allow the rendering on mobile devices. As the rendering starts on a mobile device from the moment the device is getting a response from an API call, limiting the measurement of UX through API calls would not be realistic. That’s why it is crucial to run load tests for performance testing and validate UX by executing tests against several real devices at the same time.

Metrics from Appium script and Neoload

Solution?

Our customer feedback showed that the need of validating user experience on real devices has been tremendously growing. And we think it will be beneficial to bring every mobile team a simplified way of validating mobile user experience by running performance and functional tests simultaneously with one test execution.

When we were looking for a partner with a cloud-based mobile device farm, Bitbar stands out with a similar company culture and business model as ours. More importantly, its competitive advantages on mobile testing scalability and the number of unique devices will be a great value for our customers that have a growing need for getting access to different real devices.
This is why we think partnering with Bitbar will provide you with the capability of constantly assuring mobile end user experience through continuous performance and functional app testing.

Bonus:

To give you a sneak peak of the integration, we are co-hosting a webinar with Bitbar on Wednesday, August 30th to go through the importance of mobile performance testing and functional testing and showcase how the integration between Neoload and Bitbar Testing will help you accelerate the delivery of high performance and quality apps to end users. Watch this webinar now!

Henrik RexedHenrik is Performance Engineer at Neotys, a leading provider of load testing software. He has been orchestrating and conducting performance tests for over 10 years, delivering projects in all contexts including extremely large Cloud testing on the most demanding business areas such as trading applications, Video on Demand, sports websites, etc.

Prior to Neotys, Henrik worked as .NET architect for Logica and Performance testing expert on large accounts in a variety of industries including insurance, car industry, retail and energy. Amongst the numerous initiatives he worked on, Henrik has built the performance Center of Excellence (CoE) handling load testing of all the European branches of a major insurance company.

About Neotys

Neotys develops NeoLoad, the fastest and most automated load testing tool on the market. Design and maintain load tests 10 times faster than with any other tool so performance testing smoothly fits in Agile and DevOps.Neotys is a leading innovator in load and performance testing solutions for Web and Mobile applications. In pioneering Continuous Performance Testing, Neotys enables Development, QA and IT Operations to quickly and efficiently test the quality and speed of their applications, from application component testing to system-wide testing. 

Neotys is a leading innovator in load and performance testing solutions for Web and Mobile applications. In pioneering Continuous Performance Testing, Neotys enables Development, QA and IT Operations to quickly and efficiently test the quality and speed of their applications, from application component testing to system-wide testing. 

Learn more at www.neotys.com


How to Build a Million-Download Mobile Game

Learn tips & tricks to build better games and make users’ lives more enjoyable.

Download

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close