Best Practice #3: Rely only on Real – Emulators vs. Devices

Bitbar, the mobile devops company. Logo, large

The mobile market continues to evolve and grow. This is evidenced by the large number of handset manufacturers that continue to churn out devices, the choice of different mobile operating systems, even the type of design that ranges from tablets to minis to standard sizes.

Given the rise in mobile use, companies and brands are turning more and more towards developing mobile applications and launching them into the market for consumers. However, the sheer number of possible mobile configurations can present a challenge for developers of these applications. Ensuring that an app works well on a large number of devices is crucial to avoid bugs and frustrated users.

This blog looks differences in testing your mobile application or game on emulators and real devices – in the context of test automation. As this is one of the most asked question regarding mobile app testing many things should be included in comparison. We’ll take a look what things are important for end-users, how those things are seen by developers, and what eventually strikes a balance between testing on emulator and real devices.

Percentage

To get a comprehensive comparison of using emulators and real devices in your app testing we separate different aspects of app testing to four categorizes: User Experience, Hardware, Software and Infrastructure. Whereas the user experience focus on overall usability and testing the entity, hardware and software categories are separated to different pieces, and infrastructure takes surrounding things into consideration.

Testing Categories

CATEGORY #1 – User Experience and Usability

Testing usability and what user experience your app delivers to its end-users simply cannot be done on a desktop with a mouse and keyboard. Therefore, we picked few things you should think about when selecting platform for testing:

User Interactions – Frankly, not all types of user interactions – or stimulis – can be tested on an emulator. Using a mouse or keyboard to click on simulator is different that using a finger on a screen of mobile device. Real devices provide a real user environment. Certain interactions like pinching, zooming and scrolling are considerably different on touchscreen.

Real occurring events – Interrupts, battery consumption, how charger effects to overall performance and usage. These sorts of issues can be simulated emulators but that isn’t real. Seriously, the only option for testing real occurring events during execution of your mobile app can be only done on an actual device.

Overall performance – It is important to test the app for its performance. Using an emulated or simulated environment for performance testing is not a realistic and results are meaningless. Performance lags are easy to expose using actual devices. In addition, the current Android emulator – which is based on QEMU – isn’t really fast or agile in any scale. The frustration of developers and testers using slow emulator can be avoided with help of real devices – physical ones or ones resided on cloud. For comparison, uploading your APK from Android SDK to Testdroid Cloud using run-in-cloud plugin gives you results in a minute. You barely see emulator getting up and running at that time. And that is yet emulator.

Consistency in results – You would think when running your app on emulator that results would be always the same. Actually, due the non-real environment results are not always guaranteed to be the right as many things are taken in account, starting from your desktop setup, internet connection, file system, capability to illustrate graphics and so on. Only real devices provide a platform to test real user experiences.

CATEGORY #2 – Spectrum of device configurations (Hardware)

Hardware is naturally the biggest difference in testing between emulators and real devices. However, many things make also hardware perform differently and here are some of the most common ones:

Chipset – It’s actually amazing how differently different silicon (CPU, GPU etc.) perform. Imagine yourself executing something that is targeted for high-end devices on low-end hardware. That simply doesn’t work. The user experience is the most obvious hit as performance goes down and targeted apps/games running on low-end silicon with a low clock-frequency rate is delivering a bad experiences to users. In case of many apps/games, it is actually severe as activities are synced with the clock and UI refresh can’t keep up with it. Users see this sort of behavior as badly implemented graphics, blinking of the screen, and general slowness.

Display – It is not always about the resolution that is causing a headache for developers, but more and more problems are related the quality of a display: it is also about the density, colors and overall quality of the display used in device. For example, many games suffer because of bad quality displays in mobile devices. For instance, developer wants to have a blue button on certain part of UI but it gets shown as different shade of blue – and even in some cases totally different color. This is a very common problem and not only related to display but also drivers. The graphics driver can break consistent-looking UI as we’ve experienced on Android world in the past. In addition, for some apps the graphical content has been missing because of color brightness and low density of display.

Devices-with-same-software

Memory – Developers have to eye-witness this problem quite often. In many cases their app or game doesn’t run on certain Android devices due to too big memory consumption. Typically the most popular apps and games – as rated with 4 and 5 stars in app markets – do not have this problem as memory handling is implemented right. But, too many of today’s apps and games are still originally developed for high-end devices and low-end devices can’t run those. However, what is clear is that this type of problem can be easily tackled when app/game is properly tested across the array of Android devices.

Sensors – Misbehaving sensors – and we’re not talking about badly calibrated or non-calibrable sensors – are causing various issues on games that require inputs from device handling. For GPS, the known issues are difficulties of navigation in indoors and some locations that satellites cannot be reached. One typical issue also is related to media streaming – for example, video that is meant to be shown in landscape mode worked well in landscape but you would have to rotate the device 180 degrees to see it right. Frankly, there is no way to test orientation properly with emulator and things related to accelerometers, geo-location, and push notification cannot be emulated or may provide inaccurate results.

CATEGORY #3 – Platform + Customizations (Software)

Your app is software but there are tons of other software involved in mobile device too. And this ‘other’ software makes your software perform differently on array of devices. Let’s look at some of the most common things experienced by app developers:

Platform/OS version – In fact, the operating system version combined with rest of the software – typically brought in by OEM – have an impact on developers and how their app/game performs on any device: The well-known root cause for Android problems is the OS platform version. Something that was meaningful for developers and end-users in the prior version may not play that important role in the new version. For example, the version of Android executed on QEMU is not the same that you’ll find on any of OEMs Android devices – no matter what the version number or API level tells you. Typically OEMs put in customizations for middleware, drivers, and other infrastructure software. How could any developer access these on emulator?

OEM and Carrier customizations – What is fragmentation for developers is a differentiation for OEMs. OEMs have been keenly built their own user interface layers, skins and other middleware on top of the vanilla Android, and this is significant source of fragmentation for developers. Many OEMs also bring their legacy – or just Android tailored software – to their devices and it is also breaking the compatibility of developers building their stuff to work identically on different brand models. Drivers also cause major problems, and many of those are related to graphics. As mentioned, developer may see whole different color scheme on different Android device and nothing close for what it was meant to be.

Dependencies to other software/app – Sometimes apps or games depend access to some other application. For example, many apps/games include social media and in worst implementations it is taken for granted that some of the social media applications are preinstalled in devices before user can actually use those shortcut SoMe buttons. No need to mention but this causes already situation that not only affects to design but also implementation of the app.

CATEGORY #4 – Infrastructure (Network)

Network/WiFi – Devices experience network issues and slow network is not possible to be tested on emulation. In terms of network configuration, emulators run on the PC, connect to the LAN and access the internet via your corporate firewall. Using real handsets, the network is connected to the radio interface and from there to the internet.

Conclusion

Every developer is yet more carefully considering those pros and cons between emulators and real devices to ensure the right decision in their path to create their mobile app testing strategy. Typically emulators are used in initial stages of the app development and later on real devices are brought into game. However, using the right platform to foster your next big thing should be as authentic as possible – from the day one. From our experience that has been one of the most important cornerstone of creating those successful apps – with hundreds of millions of downloads.

Some thoughts how to fully take advantage of real devices and test automation you should read this and that.

Happy testing!

  • John

    Top four criteria for comparison between emulators and real devices sounds very valid to me. How much does carriers or their networks affect nowadays on apps? My understanding is that very little but do you have any data about it?

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close