The Value of Testing in Mobile Game Development Process

  February 04, 2015

We’ve been discussing various aspects of mobile game development and testing, different required architectures and infrastructure to build more robust mobile games. Again, the feedback we’ve got is awesome and we encourage you to get in touch with us as we’re highly dedicated to continuing sharing our insights, best practices, tips & tricks and lots of other relevant things when it comes to making better mobile games. Today, let’s look at how testing adds value to the mobile game development process itself – and what are the most important metrics you should be thinking about.

Time-to-Market with Mobile Games

Crossing the Mobile Game Development Chasm

Much of the mobile game testing is still done manually and that doesn’t provide an efficient approach to get everything properly covered. In fact, effective mobile game testing should derive from a well-structured and systematic approach, use of test automation framework(s) and seamless integration with your agile process.

Furthermore, many test automation frameworks have been originally built for native apps (or web apps/elements) and one of the biggest misconceptions among mobile game developers has been that test automation frameworks do not comply to game testing – where certain native UI elements, their IDs, characteristics etc. cannot be right away identified. Well, this isn’t really the case and many of those frameworks provide excellent ways to test mobile games. For example, by using image recognition features.

When it comes to testing, the simplistic view of it is that bugs are identified and documented so that developers can remove those bugs. Therefore some metrics traditionally used for software testing – such as the number of bugs found and the number of test cases created – are great but do not show the value-add of testing and because of this, QA is hardly seen as a productive and well-spent effort in organizations. However, due to instant feedback capabilities by gamers through App Markets and concerns about user retention has changed the way serious mobile game developers think about testing. Bad ratings and feedback leads to a low number of downloads and eventually causing the game being a rogue investment.

Does crash reporting help then? Yes, absolutely, but that only provides you the information that your mobile game doesn’t work, crashes or users experience something else bad – and that’s already too late. Your game has been exposed to users, negative feedback is publicly available and fixing an issue doesn’t always help. In some cases, bad ratings/reputation can even prevent getting next titles approved, or for sure not getting that title in front of hundreds of millions of gamers.

3 Factors for Improved Value in Mobile Game Testing

The value-add of testing brought into the development process can be also seen through effectiveness, speed up development (improved productivity) and the possibility to release your title sooner (time-to-market). All these ‘improvements’ can directly be measured in the terms of value:

The effectiveness of your QA on finding and development on fixing the defects. This is the no. #1 cost driver in today’s mobile app and game development. As said, direct costs of defects have a direct effect on your bottom line, like the actual cost of fixing the defects from testing process, cost of fixing and verifying crash reports coming from the market, and lost customer acquisition costs (CPI) and lost lifetime value of the customers (LTV, affecting top-line).

Too often, effectiveness is easy to neglect as a cost driver, but when you put your figures honestly on the table, you will find that late bug-fixing is consuming a lot of developers’ time as well result in delaying the release, which then equals to lost revenue and lost customers.

Integrate development and testing

Time-to-market costs directly affecting your top-line. Time is a crucial attribute not only against competition but also to gain revenue as soon as possible. When you launch your game, every day counts as you start generating revenue earlier. The important metrics are monthly/daily acquired users (MAU/DAU) as well as average revenue per (daily active) user (ARPU, ARPDAU). Remember that the delays with publishing aren’t coming because of testing – if your development process accommodates true agile development and testing.

Many game developers do not calculate the impact of delays, but frankly, it is quite easy to calculate the impact of each day of delay to your top-line. Manual testing – even with mobile games – will delay the launch even more. The solution is the agile, test automation driven development with continuous integration, continuous testing and quality assurance on real end-user devices, all the time during development.

The productivity of your QA. This is the most obvious one affecting your bottom line. The easiest way to further analyze the productivity is to compare the costs of manual testing efforts to cost on test automation tooling & automation costs. While this gives you only the financial cost estimates, it is also important to consider other aspects: the common approach with mobile game testing, unfortunately even still today, is that all of that important QA and testing work gets done at latest phases of the development process. We discussed the new idea of the agile process for mobile game development and how to fully gain the benefits that this agile approach can offer – and make your QA more productive.

How to Add Value to Game Development Process?

Effectiveness. The easiest way to tackle the challenges related to effectiveness is to integrate game development and testing process into a cohesive agile development process. This enables you to automate a huge number of different types of tests, using real mobile devices (all sorts of those) and check code after every build (daily/nightly/weekly builds). The effectiveness has the three following attributes and advantages:

1. Instant feedback on defects. Mobile development brings the added dimension with automating the process on real devices your end users are using. From developers point of view, the instant feedback on defects increases the productivity dramatically, since you still have the code you recently touched fresh in your mind. Who remembers what you had for lunch 2 weeks ago vs. today?

2. Effectiveness in defect finding prior the release will lead to fewer field failures. The gaming apps are the least robust and have the highest crash rate of any app types. Let’s take an example: a mobile game with 50,000 monthly active users make 2,200 crash reports every month. If you have 3 games on the market, that would be 6,600 crashes per month. If the analysis-fix-verification takes around 1h, the cost is huge. Not because of engineering costs, but the cost to the reputation, given-up users and potentially lost revenue.

Proactive testing on real mobile devices before the release would have improved and made the game more stable – and therefore the development effectiveness would have been better by a factor of 2-4. Post-launch monitoring is naturally still crucial, but you better not be in reactive mode what comes to defect finding, or it will cost you a lot.

3. Lost CPI and LTV. You know how much each install costs you, and you probably are aware of the LTV of your users. If your game fails after a release it is highly likely that you will lose both the CPI as well as LTV of that user/all users using the device model. It is highly recommended to plug in your CPI + LTV for each crash coming from the field to understand the impact of the failures.

Being proactive, and using true Agile development process with continuous game testing approach on real mobile devices, will lead to higher top-line and highly improved bottom line as well as happier developers: Every developer wants to work on the new features rather than fix existing features in fire-fighting mode based on the crashes coming from the market. Remember! Manual testing is not agile, and it is not an answer to those mentioned aspects. You do not get instant feedback. If you want to test your game manually on top 50-100 devices, for example, the cost of doing that bi-weekly or even monthly is high and practically impossible.

The Agile Process for Mobile Game Development

Time-to-Market. Shorter time-to-market drives up your top-line. The earlier you launch, the earlier you start generating revenues. When you release your new game – or old game with new monetized features – 2 weeks earlier than previously using agile combined with automated mobile testing, that is 14 days shaved off from every release cycle. If you do 6 releases per year (every 2 months), it is 84 days saved annually.

The maths for lost revenue is really simple:

  • Suppose 100,000 daily average users (DAU)
  • Suppose $0.05 average revenue per daily active user (ARPDAU)
  • 14 days (x) 6 release/year (x) 100 000 (x) $0.05 = $420 000/year/game more to your top line.
  • If you have a portfolio of 5 games, total would be around $2M from faster time-to-market.

You can achieve all this by shaving off 2 weeks in every release cycle. Not all games are direct revenue generating, so replace ARPDAU with your specific value for each mobile user.

Again, time-to-market is improved with integrated development and testing process. This is the case for all software development, but in mobile app dev, it is the only way to do development efficiently. There is a huge number of end-user platforms (e.g. different devices with different hardware and software setups), that your games should support an ever-increasing complexity in games that need to be tested on those devices in every release.

You can automate a huge amount of testing, and do it on real devices after every new code checking and/or nightly builds. This will shorten your development-testing cycle, improve defect-fixing times and result in an improved top line.

Faster time-to-market also keeps your competition at bay. Your competitors will not get a chance to promote their offerings over yours, which is always a danger when your releases start to slip. You can release more often with more confidence in the quality of your releases.

Productivity. The quality assurance (QA) is a critical part of the process when creating and maintaining successful mobile games for hundreds of millions of gamers. One of the metrics in this game creation process is naturally QA’s productivity – the success to efficiently find and filter out the problems in games before they land on hands of users. This type of productivity of quality assurance can be measured by various metrics, for example:

  1. Bugs found rate – how efficiently the QA team finds bugs during the testing process per time used on testing
  2. Post launch issue rate – how many issues are found after the go-live release
  3. Customer satisfaction rate – for example complaints to customer support, issues reported to app stores, overall ratings of the games
  4. Quality of deliverables and issue documentation – how easy it is for developers to understand the issue and make a fix based on the documentation
  5. The number of new test cases added per period of time

The challenges unique to mobile game development & QA are device diversity, different operating system versions, hardware configurations and OEM/carrier customizations. Those aspects increase the complexity of QA and result in longer development cycles (time-to-market), higher fields crash rates (lower customer satisfaction and uninstalls hurting your top line) and higher customer support costs (more customer complaints) and inefficient development team (dealing with customer issues vs. developing new features). In order to improve the QA productivity, teams need to have access to real devices – the very same ones that end-users use.

Productivity of QA

Developers and QA teams need access to those devices exactly when they face an issue during development, they are fixing post-launch issues or get customer complaints. Earlier the issues are found, lower is the cost of fixing them. To maximize bug-finding efficiency, you can automate testing, and do it on real devices after every new code checking and/or nightly builds. This will shorten your development-testing cycle, improve defect-fixing times and result in an improved bottom line.

When using a cloud-based service for mobile development and QA, the performance of your team will improve a lot: The cost is fraction of the manual testing, developers are happier (instant feedback on new code, no need to deal with purchasing when acquiring new devices), product owner is happier (higher customer loyalty, higher LTV, faster time-to-market), customer service is happier (fewer customer complaints) and most important of all, your customers are happier as they do not get frustrated by “not working on my device”.

Conclusion

There are several practical ways to add value to the existing mobile game development process. If you focus on mentioned three items and follow the agile process with continuous testing, you’ll very quickly gain benefits and be able to build more robust games. Here is a check-list to move forward with better quality:

  1. Consider automating most of the testing, choose any of the excellent open source testing frameworks available for the most popular mobile platforms
  2. Avoid totally unnecessary vendor lock-in by using standard languages (Java, dotnet, perl, php, python, ruby), there is a pool of millions of developers using these languages to tap into
  3. Automate as much as possible of the testing to improve the organizational agility (even make developers responsible for creating automated tests)
  4. Use continuous integration and continuous testing, testing is a continuous process in agile and will deliver you results 24/7
  5. Integrate with your test management systems to improve the collaboration and transparency throughout the organization. Tools must enable a free flow of collaboration
  6. Instant accessibility. The development teams do not have time to engage with long sales cycles, procurement processes for each tool and device. Choose a service with rapid provisioning without waiting for hardware, procurement or internal processes.
  7. Finally, the most important one: Tools matter, but how you use them matters more. Tools make a difference, but how you use them will ultimately determine your level of success.

Happy Mobile Game Testing!