The most successful companies do their mobile apps in agile and use fully integrated development and testing processes. The most successful companies also track their performance carefully. To achieve higher efficiency and lower costs, you need to understand the impact of various parameters to your top line and bottom line.
Effectiveness, Time-to-market and Productivity
The financial benefits of Agile fall into 3 main categories which each can be measured in $$:
- The effectiveness of your app development team on finding and fixing the defects. Direct costs of defects affect your bottom line, like the actual cost of fixing the defects from the testing process, cost of fixing and verifying crash reports coming from the market, and lost customer acquisition costs (CPI) and lost lifetime value of the customers (LTV, affecting top-line)
- Time-to-market cost directly affecting your top-line. When you launch, every day counts as you start generating revenues earlier. You know your MAU/DAU as well as your mobile KPIs, like average revenue/value per daily active user (ARPU, ARPPU, ARPDAU). It is very easy to calculate the impact of each day of delay of the release on your top-line. Late surfacing issues and defects found in the long manual testing process will delay the launch always.
- The productivity of your QA. This is the most obvious one affecting your bottom line. You can compare your costs of manual testing efforts to cost on test automation tooling & automation costs.
Naturally these 3 high-level aspects are also heavily correlated, and amplifying impacts of each other. I will discuss all these 3 aspects in the blog series, now let’s focus on #1, The Effectiveness of your app development on finding and fixing bugs.
Effectiveness tells how efficiently the development team finds defects, fix them and verify the fixes. The cost of finding and fixing bugs is the Number 1 cost driver in the app development. Effectiveness is easy to neglect as a cost driver, but when you put your figures brutally honestly on the table, you will find that it is consuming a lot of developers’ time as well result in lost revenue and lost customers.
Effectiveness is improved with integrated development and testing process. You can automate a huge amount of testing, and do it on real devices after every code checking and/or nightly builds. Effectiveness has the following factors:
- Instant feedback on defects. This is widely documented the benefit of agile practices like continuous integration and continuous testing. Mobile development brings the added dimension with automating the process on real devices your end users are using From a developer’s point of the instant feedback on defects increases the productivity dramatically, since you still have the code you touched fresh in your mind. Who remembers what you had for lunch 2 weeks ago vs. today.
- Effectiveness in defect finding prior the release will lead to fewer field failures. Stats show that average crash rates are as high as 4.4% for Games, 1.8% for Media, Photo and Video apps, 1.4% for News, Sports & Entertainment apps, 1.2% for Business apps, 0.6% for Social apps and 0.4% for e-commerce apps. For example, a mobile video app with 50000 monthly active users, will receive on average 900 crash reports monthly. If you have 3 main apps in the market, that would be 2700 crashes per month. If the analysis-fix-verification takes around 1h, the cost is huge. Proactive testing on real devices before the release will improve development effectiveness by a factor of 2-4. Post-launch monitoring is naturally still crucial, but you better not be in reactive mode what comes to defect finding, or it will cost you a lot.
- Lost CPI and LTV. You know how much each install costs you, and you probably are aware of the LTV of your users. If your app fails after a release it is highly likely that you will lose both the CPI as well as LTV of that user/all users using the device model. Plug in your CPI + LTV for each crash coming from the field to understand the impact of the failures coming from the field.
Being proactive, and transforming your development to Agile with a continuous testing approach on real devices, will lead to higher top-line and highly improved bottom line as well as happier developers: Every developer wants to work on the new features rather than fix existing features in a fire-fighting mode based on the crashes coming from the market.
Manual testing is not agile, and it is not an answer to these 3 aspects. You do not get instant feedback. If you want to test your app manually on top 50-100 devices for example, the cost of doing that bi-weekly or even monthly is high and practically impossible. You should do it daily or nightly.