In mobile games, great performance is closely related to great user experience. Gamers are hungry to see constant progress with the game and have the great gameplay experience. For this, graphics performance needs to be up to snuff and the game must work well – and provide the very same experience – across all different devices (phones, tablets etc.). We’ll take a look at here how to easily automate the performance profiling and monitoring for mobile games. Even you are still starting the development, there are lots of things you can test in the performance context – and that’s where test automation brings you some real benefits.
In fact, mobile games require quite a lot of performance testing before those can be published in markets. What type of performance testing is needed varies on the nature of the game, graphics/animations/assets, gameplay, what sort of multiplayer functionalities are enabled and many other things. The most typical performance tests done for a mobile game are just generic stress tests where the game is automatically played through, the frames-per-second (FPS) measured in the context of responsiveness and refresh rate – or even used some additional workload to ensure the game can run fine on lower end of device spectrum, as well.
Automatic Performance Profiling with Test Automation
Performance needs to be consistent across all different device variants that your gamers have. Because of this, it is highly recommended that you use as many real devices for testing as possible. In order to determine how well your game responds to various levels of usage, you should consider creating tests which last for hours to observe how well the game as well as how much of battery is used during long usage. To determine if your game will be able to run successfully for a sustained duration under heavy load, Load/Stress tests can be used. The performance tests will measure how responsive your game can be on a real device.
Here is one example of an Android game executed on one randomly picked mobile device (Samsung GT-i8190N Galaxy S III Mini). In this case, the test was very simple, enabled only the basic click-throughs over the menus and end up doing those game configurations and selections, and just did nothing else but automatically played the game’s tutorial phase through. The device contains 1024MB RAM, runs on ST-Ericsson NovaThor U8420 1 GHz processor with ARM Mali-400MP1 GPU while the resolution is 800×480. Not significant stress for CPU or GPU.
The used example game first loads graphics assets, few smaller animations and initializes the gameplay. The high peak with FPS comes from animations and introduction type of show which lasts about 3 seconds. After this, we see test automation script going through some main menus, selecting gameplay and do the basic configs until the actual tutorials start. During the tutorial, there are more and less GPU-intensive periods but as we can see from the picture above, the game will work smoothly on this test device.
This type of testing is extremely easy to automate and do during all phases of the development. The more devices that are used for testing, the better – as you’ll get plenty of useful data on how to improve the gameplay performance, fix those bottlenecks and fine-tune everything. You know this but sluggish and badly performing games can ensure that the game won’t be successful, gets bad ratings and won’t make its creator (positive) name.
These sort of features can help mobile game developers to understand potential bottlenecks in performance, reveal unused potential with CPU and GPU resources, and help to push the boundaries of Android gaming and visuals, without forgetting the most important thing – a need to test the game on real devices.
Automatic Resource Monitoring with Test Automation
In addition to performance profiling, you can also use resource monitoring to see how much of memory in the device is allocated (and when deallocated) at a certain point of the execution. The same applies also for CPU monitoring and you can fetch all this data from each and every device run to yourself. This is really a great way to spot those problems and get a better understanding of how to optimize your game to work with different devices with different memory sizes.
The resource usage is critical and provides all important metrics for mobile games as the power of CPU and the size of memory varies device by device. High-end devices are more capable to run anything whereas low-end devices can’t even get the game up and running. This is not an exceptional case that game is built for a high-end device, published and then gets complained by thousands of users that it can even get in the first loading state.
Few Steps to Start Profiling Your Mobile Games
First, create an account at Bitbar Testing – if you don’t already have one. Use our App Crawler – if you don’t have dedicated tests. However, building specific tests especially for games is always recommended as OpenGL ES graphics content might be overkilling for App Crawler.
Simple Steps to Start Doing Performance Testing:
1) Write and conduct performance tests in whatever types of tests make the best use for your game. Make sure you stress the game itself but all infrastructure related to it as well. This means back-end servers, integrations with social media, and any other third party or external entity. All these things have an impact on the performance of your game and through the performance to its user experience.
2) Input everything to logs! This is very important and will help you to further investigate what happens in your game – even you would be monitoring it in real-time during the testing. Logs will maintain all that information and you can easily compare those later – even build-by-build – to see how things have regressed while building a new version of the game.
3) Use only real mobile devices – and as many as possible. You’ll get quickly understanding of how your game performs when you start doing basic smoke testing on hundreds of different devices. While running your game on those real devices, check at least these things:
- The game’s graphics and UI – does the game display/screen show properly (and similarly) on all these devices?
- Do the UI and layout and other graphical elements (toolbars, buttons, icons etc.) display correctly (and the same way) on all devices?
- Is the usability of controls (buttons, menus, boxes, what) identical in all devices? Some OS versions have different GUI look & feel and controls so this important to check across all possible devices out there.
- Is navigational / game progress flowing smoothly enough? How many delays are there between “game stages”? Again, try with a high-end and low-end device to understand the difference/gap between top and not performing device.
- Animations – are those flowing well? Animations require some performance capability from the device. It is important to test this in the context of low-end devices.
- Fonts – do the device where your game will be executed have those fonts you meant there to be – or not?
- Screen Resolution – one old root cause for fragmentation. Try to support all possible screen resolutions and have those tested on real devices. Resolutions “are not identical” when OS version is different as for example KitKat handles overlapping UI notifications differently than Jelly Bean, Icecream Sandwich or Gingerbread.
- Screen Orientation – what happens – if anything – when the user rotates the screen? Is your game meant to be rotatable?
Happy Performance Profiling for Your Games!