CEO Blog Series #2: Effectiveness of Agile App Development

Bitbar, the mobile devops company. Logo, large

The most successful companies do their mobile apps in agile and use fully integrated development and testing processes. The most successful companies also track their performance carefully. To achieve higher efficiency and lower costs, you need to understand impact of various parameters to your top line and bottom line.

Effectiveness, Time-to-market and Productivity

The financial benefits of Agile fall in to 3 main categories which each can be measured in $$:

  1. Effectiveness of your app development team on finding and fixing the defects. Direct costs of defects affect your bottom line, like the actual cost of fixing the defects from testing process, cost of fixing and verifying crash reports coming from the market, and lost customer acquisition costs (CPI) and lost lifetime value of the customers (LTV, affecting top-line)
  2. Time-to-market cost directly affecting your top-line. When you launch, every day counts as you start generating revenues earlier. You know your MAU/DAU as well as your mobile KPIs, like average revenue/value per daily active user (ARPU, ARPPU, ARPDAU). It is very easy to calculate impact of each day of delay of the release on your top-line. Late surfacing issues and defects found in the long manual testing process will delay the launch always.
  3. Productivity of your QA. This is the most obvious one affecting your bottom line. You can compare your costs of manual testing efforts to cost on test automation tooling & automation costs.

Naturally these 3 high level aspects are also heavily correlated, and amplifying impacts of each other. I will discuss all these 3 aspects in the blog series, now let’s focus on #1, The Effectiveness of your app development on finding and fixing bugs.


Effectiveness tells how efficiently the development team finds defects, fix them and verify the fixes. The cost of finding and fixing bugs is the Number 1 cost driver in the app development. Effectiveness is easy to neglect as a cost driver, but when you put your figures brutally honestly on the table, you will find that it is consuming a lot of developers’ time as well result in lost revenue and lost customers.

Effectiveness is improved with integrated development and testing process. You can automate huge amount of testing, and do it on real devices after every code checking and/or nightly builds. Effectiveness has the following factors:

  1. Instant feedback on defects. This is widely documented benefit of agile practices like continuous integration and continuous testing. Mobile development brings the additional dimension with automating the process on real devices your end users are using From developers point of the instant feedback on defects increases the productivity dramatically, since you still have the code you touched fresh in your mind. Who remembers what you had for lunch 2 weeks ago vs. today.
  2. Effectiveness in defect finding prior the release will lead to fewer field failures. Crittercism’s (the mobile app performance company) research shows that average crash rates is as high as 4.4% for Games, 1.8% for Media, Photo and Video apps, 1.4% for News, Sports & Entertainment apps, 1.2% for Business apps, 0.6% for Social apps and 0.4% for e-commerce apps. For example a mobile video app with 50000 monthly active users, will receive on average 900 crash reports monthly. If you have 3 main apps in the market, that would be 2700 crashes per month. If the analysis-fix-verification takes around 1h, the cost is huge. Proactive testing on real devices before the release will improve development effectiveness by factor of 2-4. Post launch monitoring is naturally still crucial, but you better not be in reactive mode what come to defect finding, or it will cost you a lot.
  3. Lost CPI and LTV. You know how much each install costs you, and you probably are aware of the LTV of your users. If your app fails after a release it is highly likely that you will lose both the CPI as well as LTV of that user/all users using the device model. Plug in your CPI + LTV for each crash coming from the field to understand the impact of the failures coming from the field.

Being proactive, and transforming your development to Agile with continuous testing approach on real devices, will lead to higher top-line and highly improved bottom line as well as happier developers: Every developer wants to work on the new features rather than fix existing features in fire-fighting mode based on the crashes coming from the market.

Manual testing is not agile, and it is not an answer for these 3 aspects. You do not get instant feedback. If you want to test your app manually on top 50-100 devices for example, the cost of doing that bi-weekly or even monthly is high, and practically impossible. You should do it daily or nightly.

Improve the ROI with the Help of Mobile Test Automation

Learn major cost drivers in mobile testing and how to leverage test automation to improve the ROI.