In the real world, many developers and testers have migrated to using real devices for app testing from emulators. Yet, mythes like ‘We use emulators only’ or ‘Emulators are also good for testing’ still persists. But the truth is that no one would use your application or play your game on an emulator for any reason. Mobile apps and games that get tested on emulators cannot deliver the optimal quality to your users. Without a high quality, your application will eventually get removed from users’ mobile devices.
Today we will cover 4 myths and truths about whether to use emulators or real devices:
- User experience
- Hardware – All about devices
- Software – Platform and Customization
- Infrastructure and Network
Myth #1: Emulators brings great user experience
Testing usability and what user experience your app delivers to its end-users simply cannot be done on a desktop with a mouse and keyboard. Here are few things that you should think about when selecting platform for testing:
User Interactions – Frankly, not all types of user interactions – or stimulus – can be tested on an emulator. Using a mouse or keyboard to click on simulator is different than using a finger on a screen of mobile device. Real devices provide a real user environment. Certain interactions like pinching, zooming and scrolling are considerably different on touchscreen.
Real occurring events – Interrupts, battery consumption, how charger effects to overall performance and usage. These sorts of issues can be simulated emulators but that isn’t real. Seriously, the only option for testing real occurring events during execution of your mobile app can be only done on an actual device.
Overall performance – It is important to test the app for its performance. Using an emulated or simulated environment for performance testing is not a realistic and results are meaningless. Performance lags are easy to expose using actual devices. In addition, the current Android emulator – which is based on QEMU – isn’t really fast or agile in any scale. The frustration of developers and testers using slow emulator can be avoided with help of real devices – physical ones or ones resided on cloud. For comparison, uploading your APK from Android SDK to Testdroid Cloud using run-in-cloud plugin gives you results in a minute. You barely see emulator getting up and running at that time. And that is yet emulator.
Consistency in results – You would think when running your app on emulator that results would be always the same. Actually, due the non-real environment results are not always guaranteed to be the right as many things are taken in account, starting from your desktop setup, internet connection, file system, capability to illustrate graphics and so on. Only real devices provide a platform to test real user experiences.
Myth #2 – Emulators cover a wide spectrum of devices
Hardware is naturally the biggest difference in testing between emulators and real devices. However, many things make also hardware perform differently and here are some of the most common ones:
Chipset – It’s actually amazing how differently different silicon (CPU, GPU etc.) perform. Imagine yourself executing something that is targeted for high-end devices on low-end hardware. That simply doesn’t work. The user experience is the most obvious hit as performance goes down and targeted apps/games running on low-end silicon with a low clock-frequency rate is delivering a bad experiences to users. In case of many apps/games, it is actually severe as activities are synced with the clock and UI refresh can’t keep up with it. Users see this sort of behavior as badly implemented graphics, blinking of the screen, and general slowness.
Display – It is not always about the resolution that is causing a headache for developers, but more and more problems are related the quality of a display: it is also about the density, colors and overall quality of the display used in device. For example, many games suffer because of bad quality displays in mobile devices. For instance, developer wants to have a blue button on certain part of UI but it gets shown as different shade of blue – and even in some cases totally different color. This is a very common problem and not only related to display but also drivers. The graphics driver can break consistent-looking UI as we’ve experienced on Android world in the past. In addition, for some apps the graphical content has been missing because of color brightness and low density of display.
Memory – Developers have to eye-witness this problem quite often. In many cases their app or game doesn’t run on certain Android devices due to too big memory consumption. Typically the most popular apps and games – as rated with 4 and 5 stars in app markets – do not have this problem as memory handling is implemented right. But, too many of today’s apps and games are still originally developed for high-end devices and low-end devices can’t run those. However, what is clear is that this type of problem can be easily tackled when app/game is properly tested across the array of Android devices.
Sensors – Misbehaving sensors – and we’re not talking about badly calibrated or non-calibratable sensors – are causing various issues on games that require inputs from device handling. For GPS, the known issues are difficulties of navigation in indoors and some locations that satellites cannot be reached. One typical issue also is related to media streaming – for example, video that is meant to be shown in landscape mode worked well in landscape but you would have to rotate the device 180 degrees to see it right. Frankly, there is no way to test orientation properly with emulator and things related to accelerometers, geolocation, and push notification cannot be emulated or may provide inaccurate results.
Myth #3 – Platform and customization are not important for app testing
Your app is software but there are tons of other software involved in mobile device too. And this ‘other’ software makes your software perform differently on array of devices. Let’s look at some of the most common things experienced by app developers:
Platform/OS version – In fact, the operating system version combined with rest of the software – typically brought in by OEM – have an impact on developers and how their app/game performs on any device: The well-known root cause for Android problems is the OS platform version. Something that was meaningful for developers and end-users in the prior version may not play that important role in the new version. For example, the version of Android executed on QEMU is not the same that you’ll find on any of OEMs Android devices – no matter what the version number or API level tells you. Typically OEMs put in customizations for middleware, drivers, and other infrastructure software. How could any developer access these on emulator?
OEM and carrier customizations – What is fragmentation for developers is a differentiation for OEMs. OEMs have been keenly built their own user interface layers, skins and other middleware on top of the vanilla Android, and this is significant source of fragmentation for developers. Many OEMs also bring their legacy – or just Android tailored software – to their devices and it is also breaking the compatibility of developers building their stuff to work identically on different brand models. Drivers also cause major problems, and many of those are related to graphics. As mentioned, developer may see whole different color scheme on different Android device and nothing close for what it was meant to be.
Dependencies to other software/app – Sometimes apps or games depend access to some other application. For example, many apps/games include social media and in worst implementations it is taken for granted that some of the social media applications are preinstalled in devices before user can actually use those shortcut SoMe buttons. No need to mention but this causes already situation that not only affects to design but also implementation of the app.
Myth #4 – App testing doesn’t require infrastructure and network
Network/WiFi – Devices experience network issues and slow network is not possible to be tested on emulation. In terms of network configuration, emulators run on the PC, connect to the LAN and access the internet via your corporate firewall. Using real handsets, the network is connected to the radio interface and from there to the internet.