Good performance is very closely related to good user experience. Users want to see constant progress with the game and do the smooth gameplay. To get the optimal result, graphics performance needs to be up to snuff and work across all different mobile devices (phones, tablets etc.) the very same. This can be achieved by optimizing graphics performance with test automation on real mobile devices.
Mobile games require performance testing before they hit the markets. The type of performance testing varies on the nature of game, graphics, gameplay, multiplayer functionalities, and many other things. The most typical performance tests done for mobile games are to determine how the game performs in terms of responsiveness, stability and refresh rate under certain workload or user interaction pattern.
The resource usage is an important metric with mobile games as the power of CPU and GPU and the size of memory varies device by device. High-end devices are more capable to run anything whereas low-end devices can’t even get the game up and running. This is not an exceptional case that game is built for a high-end device, published and then gets complained by thousands of users that it can even get in the first loading state.
Here, by using test automation you’ll get your game’s performance tuned to its best. Conversely, you can’t get any sort of understanding what happens underneath the UI/front-end/control, if done by manually.
Different Types of Performance Tests
Performance testing – this is the basic type of performance tests that provides information how quickly something is executed, how much it can be scaled (in terms of more graphics assets, or more connections/integrations) or how stable the performance is in general. Typical metrics from performance tests are CPU load, memory consumption and frames per second (FPS). Based on this data game developers should further improve the game to either scale down certain functionalities, heavy graphics assets or something else to ensure game works on lower-end devices as well. With performance tests, it is easy to fine-tune, further plan improvements on graphics, gameplay and connections, and optimize game accordingly.
Load testing – is somewhat the most simple form of performance testing. When certain load is exposed to the game and infrastructure the developer gets understanding of where possible bottlenecks are and what parts of game/infrastructure need optimization. Load testing can vary from basic server upload/download function to database connection and queries, and different types of inputs given to the game/infrastructure. During load testing, developers can get a better understanding of concurrency issues related to the game and how those could be optimized. For example, if your game needs to pull something from back-end server and suddenly 300+ devices do that in the same WiFi network, you can get very invaluable information about your game’s capability under this kind of scenario. With load testing, furthermore, you can understand how a game needs to ‘compromise’ when certain operations cannot be handled. Also, it typically gives great insights to hardware capabilities and how much real resources (CPU, Memory etc.) are needed for your game and what happens when those resources are exceeded.
Stress testing – determines if the system is able to handle everything, how it handles with data when the game infrastructure is overstressed, and how far stress testing can go before the game stops responding and how errors are displayed to users. Stress testing is important in the case of games especially if your game requires connections to servers or needs to be connected to some other device. Typically not so many mobile games communicate everything instantly online but rather in parts. This limits the recurring use of networks, draining less battery and limits error scenarios as your game doesn’t need to send/receive something all the time. The stress testing helps you understand what are the most valuable failures you should plan and what are the most common hardware failures or issues that you need to anticipate.
One of the subcategories for stress testing is soak testing. It’s also known as endurance testing and is used to understand if the game and infrastructure can sustain certain continuous load set on it. When soak testing is done, typically CPU, Memory, as well as FPS and other important metrics, are monitored to detect any potential leaks and issues. The goal of soak testing is to discover how the system behaves under sustainable use.
The other type of stress test is called spike testing which is done by suddenly increasing the load generated by a very large number of users, data, inputs, or any other type of stimuli. This is not continuous but periodical and the thing is to find out if a dramatic change in load will cause any errors or issues for the game and its infrastructure. The goal with spike testing is to determine if performance will suffer, the infrastructure will fail or how those error states can be handled in case of dramatic changes of inputs.
Simple Steps to Start Doing Performance Testing
1) Write and conduct performance tests in whatever types of tests make the best use for your game. Make sure you stress the game itself but all infrastructure related to it as well. This means back-end servers, integrations with social media, and any other third party or external entity. All these things have an impact on the performance of your game and through the performance to its user experience.
2) Log everything. This is very important and will help you further investigate what happens in your game – even you would be monitoring it in real-time during the testing. Logs will maintain all that information and you can easily compare those later – even build-by-build – to see how things have regressed while building a new version of the game.
3) Use only real mobile devices – and as many as possible. You’ll get quickly understanding of how your game performs when you start doing basic smoke testing on hundreds of different devices. While running your game on those real devices, check at least these things:
- The game’s graphics and UI – does the game display/screen show properly (and similarly) on all these devices?
- Do the UI and layout and other graphical elements (toolbars, buttons, icons etc.) display correctly (and the same way) on all devices?
- Is the usability of controls (buttons, menus, boxes, what) identical in all devices? Some OS versions have different GUI look and feel and controls so this important to check across all possible devices out there.
- Is navigational / game progress flowing smoothly enough? How many delays are there between “game stages”? Again, try with a high-end and low-end device to understand the difference/gap between top and not performing device.
- Animations – are those flowing well? This is about to change with Android L for native apps but doesn’t have an impact on mobile games. Remember! Animations require some performance capability from the device. It is important to test this in the context of low-end devices.
- Fonts – do the device where your game will be executed have those fonts you meant there to be – or not?
- Screen Resolution – one old root cause for fragmentation. Try to support all possible screen resolutions and have those tested on real devices. Resolutions “are not identical” when OS version is different as for example KitKat handles overlapping UI notifications differently than Jelly Bean, Ice Cream Sandwich or Gingerbread.
- Screen Orientation – what happens – if anything – when users rotate the screen? Is your game meant to be rotatable?