Transforming a company from Agile Development to Continuous Delivery by deploying a high degree of automation into its delivery pipeline, is called DevOps nowadays. DevOps is short for Development and Operations – the two departments that generally operate with two very different goals. Software Development teams are continuously writing new and better code to achieve fast iteration cycles. They need an environment with improved and instant feedback loops on new innovations. Conversely, Operations teams are all about keeping production software stable and working, no matter what. They need an environment that alerts of any deviations in production. In general, the fewer changes to the environments are introduced, the better.
At the speed of mobile, successful companies always strive for deployments to production at high frequency. For example, ride-sharing company Lyft has a goal of weekly mobile app updates. Also LinkedIn picks the last “known good” build and promotes it to Google Play and App Store. But still, how do you successfully manage two opposing goals from Development teams and Operations teams? Adopting DevOps processes is the key.
DevOps is not a tool, a team or just a practice. Instead, it is a mindset and a focus on SPEED and QUALITY through automation. DevOps is an automated assembly line for software delivery versus a handmade process. And to implement high-performance DevOps processes, you need a streamlined pipeline.
The biggest challenge for companies in transforming to DevOps is to get the scalability. When companies say they already have a process to automate the flow of code changes from commit to testing to deployment and finally to production at high frequency in small scale, that is just the first step. As Adam Auerbach (Sr. Director Continuous Delivery and DevOps at Capital One) explained in the very insightful PurePerformance Podcast – ‘The real challenge is when you scale the automation of code changes from a handful to 20, 50 or more developers or teams’. If you start to push new builds every couple of minutes, you will see the bottlenecks in your pipeline, especially when you try to run thousands of integration, functional and performance tests per commit that all take more time than it does for the next code commit to come in. Eventually, this slows down the flow and increases lead time.
In that regard, a LinkedIn engineering post – Speeding Up Mobile Releases written by Drew Hannay also explained that how their 3×3 philosophy (“Release three times per day, with no more than three hours between when code is committed and when that code is available to members”) demands a complete automation pipeline for every step, from code commit to production release. And it was emphasized that falling back into the pattern of manual verification of each release is not an option.
Product development done by over 250 engineers simply means that a high volume of builds is being continuously pushed through the pipeline. And to have a green light on all of the builds, you need to focus on the quality assurance or testing, which is the most time-consuming phase in the pipeline. Therefore, you need to have high scalability in testing to realize the goal of “fast feedback to engineers”. And the only way to mitigate this issue is to increase throughput: Run Tests in parallel.
How do you achieve faster process from code commit to build publish? The only effective way is to find and fix issues earlier in the pipeline and only allow “known good” build to make it all the way to the later stages of your pipeline. Only then Operations can assure that the production keeps stable.
The biggest bottlenecks in the pipeline are building production binaries and running UI tests. When 250 developers are pushing commits every 3h, it means that on average 83 commits per hour are being pushed through the pipeline. That is 1.4 commits every minute to be built and tested! That’s a lot of builds and tests. Builds and UI tests must be run in parallel at very large scale and these stages are very much tied together. The green build must be tested immediately, so in the ideal case, the two are executed in the same environment without any friction between the stages.
Scale and Parallelize
Our mindset at Bitbar has always been on speed and quality. We do not limit our customers by throttling the parallelism, which is very counter-productive practice when transforming the company into a DevOps mindset. It’s like intentionally introducing bottlenecks to your assembly line while you should actually focus on removing them.
We host a super scalable cloud-based testing infrastructure and a device farm with hundreds of real Android and iOS devices. Every one of our customers, regardless of Enterprises or SMEs, can run their builds and tests against all of our devices with unlimited concurrency, fast execution and reporting. We also deliver environments (private cloud or on-premise) with several hundreds of machines and devices with unlimited concurrency to companies who are determined to transform their practices from handmade process to an automated production line. And you can use the same infrastructure to automate your builds, not just UI testing!
Keep in mind, getting the pipeline working is not sufficient. You need to have a view on the health, metrics, reports, and analysis of your pipeline so that development, bottlenecks and the entire process can be properly managed and continuously improved. Like Lord Kelvin said: “If you can not measure it, you can not improve it.” We are working on very interesting things in our labs to bring the mobile delivery pipelines to the same level where the server-side development is.
We help our customers from Global 2000 companies to deliver mobile software at the pace of LinkedIn, CapitalOne, Google and Facebook without the need to have 50 people team in the development of the delivery pipeline.
If you want to talk more about transforming your company from handmade process to automated assembly line, please contact me at marko (dot) kaasila (at) bitbar (dot) com or you can sign up for a free trial of Bitbar Testing to see how we can help accelerate your app delivery.