Recently, we published a summary of our research about application stability on Android and iOS devices. In this summary, we focused solely on operating system, different versions of those and how different OEMs compare against each other (Apple vs. different Android OEMs). Naturally, one important aspect in mobile devices is the chipset, the power of the system that enables all those great games, graphics, connectivity and so on. Let’s take a look at another summary of how chipsets varied in this same study.
It’s actually amazing how differently different silicon (CPU, GPU etc.) perform. This doesn’t always come through and is not necessarily obvious for end-users. Sometimes it feels that people pay too much attention of CPU clock rate than actual chipset and other combination that makes the actual device.
For our research, we’ve gathered a significant amount of data from Testdroid Cloud test runs executed by a myriad of different mobile app and game developers. As said, the primary goal for this research was to understand the most typical problems, issues and challenges that Android application and game developers confront with those devices they must build their stuff on. By doing this research we got enormous amount of data from these devices – that represent a remarkable coverage of Android and iOS volumes globally – and we will now take a look at the hardware side of things.
How to interpret/read those graphs?
The findings from this study are presented as a box plot figures. In those figures darker black line is the median (50%) of failure percentage from different devices in that group. The upper line of box describes upper quartile (75%) and lower line describes lower quartile (25%). Lines connected with dashed line are maximum (on top or right) and minimum (on bottom or left). Circle describes an outlier.
Okay, these devices running single-core chipsets are still big in volumes when it comes to Android use. In this category we got two Apple devices (iPhone 4 A1332 with iOS 6.0 and iPod touch A1367 with iOS 6.0.1) and the following chart shows the failure rate of apps compared to various other chipsets. The blue line splitting the screen shows the average failure percentage of those Apple devices running Samsung-Intrinsity Apple A4 APL0398.
In general, all semis seem to have more and less robust chipsets. Furthermore, the number of different OEMs using these chipsets with Android is much greater (compared to Apple) so that also explains why there is more dispersion with those chipsets.
In dual-core category, we got more Apple devices to compare with myriad of different Android devices with different chipsets. The most interesting finding with Apple devices was that A7 and A6X were quite robust and failure rate didn’t change much. However, with A5R2 and A5 the dispersion was much greater. Devices with Apple A5R2 (iPad mini A1432 with iOS 6.1, 6.0.1 and 7.0) had more issues running apps. On the other hand, Intel Atom based devices were the most robust and also the most challenging (Atom X2580 vs. Atom X2560).
The most interesting finding with quad-core chipsets is that clock rate doesn’t directly correlate with performance. Some benchmarks show significant improvement in performance, even when the user experience and the app performance on those chipsets and devices don’t improve significantly. Furthermore, power consumption is a bigger problem with these devices running high-end clock-rate chipsets, and some devices run out of battery quickly. However, there is a tight relation to the quality and age of the battery, and cannot be solely attributed to the chipset.
What do these charts tell us? Well, as said, there are lots of differences in chipsets and it naturally have to take into consideration in which devices are these chipsets included. No OEM is running vanilla Android but they have customized those with their custom UIs/skins, put their own legacy software in these, and some update those versions more frequently than others. Also, how big is the memory (RAM) included in devices is also tightly related to the robustness as well as other hardware in these devices (WiFi, sensors, etc.).
Why is this important? Again, it is very critical for app developers to understand that a chipset has a significant impact on how well apps/games perform. A good rule of thumb is that you should also thoroughly test your applications on all kinds of devices — low end, mid-range and high end — e.g. if your app has heavy graphics (such as apps with video streaming) or uses the GPU heavily, to ensure maximal performance across all possible devices.
Do not assume that your app works across different chipsets. Chipsets have a lot of differences between them!
Learn major cost drivers in mobile testing and how to leverage test automation to improve the ROI.Download