Skip to content

As we advocate an agile development and practices for any kind of mobile development, we would like to share with you some fundamentals and our thoughts on how we’ve built our products, what kind of processes we use internally to get things done efficiently and what has increased our capability to faster builds and deploy new versions of Bitbar Cloud. This being a broad topic, let’s start first with a glance at how we changed our internal doing into yet more agile doing – enabling us to achieve a faster release schedule. The part of that is an agile process, agile regression testing and test-driven development approach. Let’s deep dive into details!

Agile Regression Testing for Mobile Apps, Games and Web

Testing online cloud service isn’t that much different from how you test your mobile apps. Basically, testing mobile app software is pretty much the same: front-end (UI) and back-end (service, OS integration, HW integration, etc.) can be tested with the same elements, frameworks and tools we use here at Bitbar team. Naturally, a big exception is the enormous farm of devices we host at Bitbar Testing. For this, we’ve built a sophisticated monitoring system that makes sure up-time for each device would be close to 100%. Our DevOps team will explain more about this during the year, but before that, let’s discuss a bit about our processes, our testing approach and how we’ve managed to utilize an agile process to get features delivered to you on a bi-weekly basis.

Whenever developers change or modify their app software or logic, and we’re talking about even tiny tweaks in there, some unexpected consequences can occur. Testing mobile apps can be done in so many different ways but the goal eventually is the same: to ensure that a change or addition to app/software hasn’t broken any existing functionality. That’s called agile regression testing and in these regressions, a new build or release candidate must go through a serious testing pattern.

By running these test scripts again, again and again against each release we make sure that any new changes to our products haven’t resulted in a regression. First, something about the release schedule.

What We Learned from Moving to a Faster Release Schedule

Bitbar Testing release process used to be rather slow – an almost one-month development cycle was followed by a lengthy testing phase before the release. Typically a release then was followed by several days of debugging and patching as bugs were found in production. This was shown in maintenance notifications – primarily in Bitbar Testing – but the service was usable all the time.

The pros of patching releases, of course, is that you keep the service online all the time, up-time is maximal and people can carry on with their mobile app development and testing tasks as usual. However, there are cons with this kind of release-patching-another-release approach as well. For instance, the service may not be complete and provide the full power of all features instantly. Some features and new things are even considered as bugs – and that’s actually where a faster release schedule can help.

faster release schedule

Currently, our product is released about every two weeks and release is over in a few hours before launch time. Changing to a faster release schedule has taught us a lot but there is still a lot to learn. Here are some of the findings in this shift and also some thoughts on what we’ll be trying out next.

How Test-Driven Development and Regression Testing Work for Us

Moving to a faster release schedule brought some expected challenges but also some unexpected obstacles that we are still working on currently to create new cool features to our customers even faster than now.

Our user interface is a very independent component allowing us to make major changes to the front-end without a need to make changes to our back-end. As a significant majority of the logic is in the back-end, front-end changes should fully expose all those changes, improvements and add-ons we’re pushing for each release. This is naturally challenging and ever-changing front-end isn’t really great from the user experience standpoint.

bitbar backend ui

How does the agile approach help us here? Well, again, it’s very important for mobile app developers but also for us that even small changes to the front-end or back-end code can break things in pieces. The agile regression testing is highly important and as everyone knows even a little thing can break functions that seem completely unrelated to the recent modifications/regressions. When we run our regression tests, we always check and ensure that our modifications not only behave as we want them to but haven’t inadvertently caused any issues to other parts of the product. Not a small thing, but test automation provides a fantastic foundation to spot these issues right after the staging build.

It was anticipated that releasing often would bring us challenges in keeping our test automation up with the new release schedule. To improve our automation process we now write our requirements using a test-driven development (or TDD) approach. The requirements feature files can be executed with minimal updating by the testing team using Python’s Behave test automation framework. As test automation improved, additional effort was put in getting testing environments (integration, staging & acceptance) also configured automatically.

With a slower release cycle, documentation was updated in separate mini-projects where the user documentation was updated to match the currently running version of the service. The documentation would stay valid for some time or would even be forgotten as it was not part of the release process.

Agile Regression Testing and How it Serves Our Customers

agile approach for a faster release schedule

Clearly, we’ve had some issues with our product documentation. So obviously something needed to be done to maintain our online documentation. This time we are trying out an approach where documentation is part of the definition of done of each feature. Everyone in the company, or outside the company for that matter, has the right to propose updates to the documentation. Currently, it is not yet online, but as soon as it has been used for some time we’ll blog about how it’s working.

The new release cycle has increased the communication needs from a product management perspective towards other parts of the company. There is a constant need to advertise the (smaller) features that are coming out. Big features get built from smaller stories, each of which is valuable to the end customer but also for sales and customer support too.

In retrospective, it seems obvious but we didn’t realize that having faster and smaller releases will slow down our internal development. We give customers’ requests greater priority compared to internal requests in our feature backlog. As a consequence, some of our important internal development tasks have been delayed. We are still learning how to balance internal and customer requests.

What is Next

Overall the benefits of a faster release cycle have been great. Stress levels have gone down at all levels of the company. Smaller stories are less risky and better understood by all. The number of mishits has dropped greatly. Automation at every stage is key to success and brings confidence and stability.

Our roadmap includes continued improvements in creating and testing releases such that every single feature can be released independently as it becomes ready.

Happy Testing!

Niko Cankar

Bitbar Product Owner