The mobile test automation genre is constantly evolving and new frameworks have been appearing for every specific need. As we’ve been supporting all mainstream test automation frameworks, we’ve learned quite a lot about how different frameworks work for different types of mobile apps. Appium has been in tremendous growth, but there is no reason to forget Calabash. Both of them provide a great foundation for cross-platform testing strategy and due to some concerns about Calabash and recent changes in the industry, we’d like to provide you a tutorial of best practices with Calabash, how to get started with it, how to combine Calabash approach across other frameworks, and some useful tips!
Calabash Framework Definition
In short, Calabash is a test automation framework that enables mobile developers and pretty much anyone without coding skills to create and execute automated acceptance tests for Android and iOS apps. Calabash works by enabling automatic UI interactions within an application such as pressing buttons, inputting text, validating responses, etc. While this is a great first step in automated UI acceptance test automation, the real benefits can be gained when Calabash tests are executed on real mobile devices. This is very easy and Calabash tests can be configured to run on hundreds of different Android and iOS devices, providing real-time feedback and validation across many different form factors, OS versions, OEM customizations and hardware specs.
Calabash, which is considered as Behavior-Driven Development (BDD) test automation framework, means basically a philosophy of outside-in development. It’s conceptually similar to Test-Driven Development (and is in fact based on it), but takes it one step further, in that instead of creating tests that describe the shape of APIs, application behaviors are specified.
The idea of Calabash isn’t much different from the practice of creating functional specifications for software and then building to the spec, but it takes functional specifications and makes them executable, by providing a means in which to test the actual specifications. And just as typical with specification creation, BDD is meant to be a process in which multiple stakeholders weigh-in to create a common understanding of what is to be built. The benefit of the BDD approach is that it helps ensure that the right software is built (as opposed to the software being built right). The intent is that the software is designed from the perspective of the business owner. That way, there is little interpretation required as to how the software should behave.
Ensure You Get The Best Out of Calabash
Use only real mobile devices with Calabash. First of all, it’s always recommended that you rely on real mobile devices in your development and testing. This enables you to get accurate information of how your app does on the same devices that end-users use. Having an authentic environment where you run your Calabash tests must include the various version of OS, OEM customizations (especially in case of Android), hardware with different setups, chipsets and amount of memory, and real-environment conditions (e.g. integration with the backend, network).
Test coverage needs to be adequate. Writing tests with Calabash is a very easy and actually fun job to do. The language syntax is human-readable and you don’t need coding experience or skills that are typically required for software developers. Writing, configuring and setting up tests can be also nicely scaled for different software versions (e.g. OS versions, and those OEM customizations) but you’ll likely confront more fragmentation when it comes to running Calabash tests on different hardware.
Real-user conditions. When you have those real devices in your testing roster you also want to make sure there are enough capacity and power with your surrounding testing infrastructure. This means having a capable network and if you want to use the mobile network, you should equip devices with SIM cards. We constantly enable these sorts of setups in our private cloud. There are any Android and iOS devices available, with whatever infrastructure required for the setup, and all runs smoothly with your Agile development tools and environment.
Integrated with CI/CD. Calabash, like other test automation frameworks, can be easily integrated with continuous integration / continuous delivery tools, such as Jenkins. In addition, there is a comprehensive API for app testing that can be hooked with any additional development tool, testing framework or tool, and test management software.
Calabash Tutorial and Schedule
During this and next week we’ll be thoroughly covering Calabash mobile app testing with the following themes:
5 April – How to Setup and Get Started with Calabash (both for Android and iOS)
7 April – How to Build The Right Tests – Steps, Features and Scenarios
14 April – Tips and Tricks for Calabash Steps and Step Definitions
21 April – How to Combine Cucumber with Other Test Frameworks
Stay tuned and make sure you check these tutorials during the next few days!
Happy Testing Folks!