We walked through the basic setup and installation of Calabash for Android and iOS in previous blog, and this time we’ll take a look how you can build Calabash tests for your mobile application. If you have followed the example showcased in How to Setup and Get Started with Calabash you have properly generated the file structure for your first Calabash tests on Android and iOS. There are lots of difference (best) practices of how to create your Calabash tests, and we’ll use an example of basis to illustrate how these can test templates can be generated and further developed.
This blog takes a look what is the Calabash architecture and how to create tests using that architecture, what cucumber means in the context of test creation, and basically how to create your first Calabash test. We have an example application available for Android and iOS, plus you’ll find a link below where the test script can be downloaded. Furthermore, we briefly discuss about running tests on a cloud service that provides comprehensive results, data, logs, performance metrics, and highly sophisticated reporting, even with the actual Calabash logs. Don’t forget to download a copy of Calabash 101 to access it anytime.
Calabash Architecture for Test Creation
There are differences when it comes to Calabash running on Android and iOS. The basic test and test creation flow, however, remains the same. The recommended process to get test scripts done varies but the good rule of thumb is that you first create your features. All this information can be found in example below and even example provides a basic example of feature.
Then you test your feature and execute it to see what it actually does. Then you can create those Step Definitions. Step definition is basically a Ruby code that gets executed when you run your test. The following picture illustrated relation between Feature, Scenario and Steps.
A scenario specifies a single behavior or use case within a given feature that is comprised of various Steps. For example, the following scenario describes the behavior of ensuring that a credit card input field has the correct length of digits:
Scenario: Run whole app Given my app is running And I touch the "Start" button Then I take picture Then I press "More Info"
Steps usually begin with one of the keywords Given, When, Then, And, and But, however, they don’t have to, they can use * in place of those keywords. In fact, Cucumber does not distinguish between them (or *). They are instead meant to provide a language hint based on cause and effect to the stakeholders as to what is being described.
As such, simply recognizing their language implications are enough to use them effectively. However for a detailed examination on these keywords, see the Cucumber Wiki entry on them.
Feature is rarely defined by a single behavior. For this very reason, Scenarios can be grouped together logically under a Feature Definition. Feature definitions are typically given a name and an optional, short description. For example:
Feature: Test the entire app Scenario: Log in to app Given my app is running Then I take picture Then I use the native keyboard to enter “firstname.lastname@example.org” into the “your name” text field Then I use the native keyboard to enter “myPassword123” into the “password” text field Then I press "Sign In" Scenario: As a valid user I can start using the app I wait for text "Hello" Then I wait for activity "HomeTabActivity" Then I press view with id "menu_compose_tweet" Then I enter text "Testdroid" into field with id "edit" Then I press view with id "composer_post"
Steps and Step Definitions
Step definitions are like code-behind for the scenarios defined in step definition scrips. They provide the glue that makes them runnable in the application. Their function is to translate readable texts into runnable actions. For example, step definitions for used example could be as follows:
require 'calabash-android/calabash_steps' Then /^I set screen to portrait$/ do perform_action('set_activity_orientation', 'portrait') end Then /^I set screen to landscape$/ do perform_action('set_activity_orientation', 'landscape') end Then /^I hide keyboard$/ do hide_soft_keyboard() end
We’ll take a look of steps “I set screen to portrait/landscape” and “I hide keyboard” examples below.
Calabash Architecture / Infrastructure – How Calabash Works on Real Devices
The great part of Calabash is that it is a cross-platform framework and works really well for both major platforms, Android and iOS.
With iOS, Calabash installs a HTTP server as an instrumentation package that then listens commands from Calabash server. Tests are also executed on server side and each test case is described in Cucumber. Slighly different than with Android, the Calabash iOS converts Cucumber commands to ‘FRANK’ method calls that then get executed. The webview support is done the same way than with Android.
Getting Started with Test Creation
To save time, we’ll use an example application and ready test package. The BitbarSampleAppTests.zip file contains a functional test script. You can easily create similar file set for using
calabash-android gen command. If you are looking for Calabash-iOS Tests you can basically use the same test package but you need to fetch the .IPA file.
As application is relatively simple, we’ll start by creating a Feature, written in Cucumber. The possible scenarios for user interaction in the above example would be that user 1) clicks through the radio buttons, 2) types a name in input field, 3) hides (or after hiding, asks to show) the keyboard, 4) changes the orientation of the screen, plus asking test to take screenshots at any given moment.
Feature: Test UI Components Scenario: Test Radio Buttons Then I press view with id "radio0" Then I take screenshot Then I press view with id "radio1" Then I take screenshot Then I press view with id "radio2" Then I take screenshot Scenario: Input Text and Change Orientation Then I set screen to portrait Then I take screenshot Then I set screen to landscape Then I wait Then I enter text "Hello Calabash" into field with id "editText1" Scenario: Test Keyboard Then I take screenshot Then I hide keyboard Then I take screenshot Then I wait Then I show keyboard Then I take screenshot Scenario: Click Button Then I press view with id "button1"
If you look at
my_first.feature it basically contains the same steps as shown break-down in the scenarios above.
Running Calabash Tests and Interpreting the Results
The reporting and test results are important to understand how application does under the testing. These reports and results should yield all fine-grained details about the test execution, performance, and provide users actionable and rich information about tests and test runs. Furthermore, it’s important to get that test result data as real-time as possible, and those are stored based on each application regression, test executed against the specific version of app, and that you have a way to compare results based on each app and test against the older ones.
First package your test zip file to include all files that has been either generated (e.g. calabash-android gen) and modified/added while you created the test. The folder structure (regardless if you have added new tests, steps, features etc.) should look like this:
After you have created a package of it, just log in to Testdroid Cloud and upload your APK or IPA, and this zip file. If you are not familiar with the interface, there is a full-blown guide of how to use real mobile devices on cloud for testing. This guide will provide you all necessary steps to access devices, get tests executed, and finally provides you a comprehensive reports of both, test details and Calabash report.
Naturally, when you Calabash test script is more complex you probably want to check also performance aspects of the app running on specific device. The same view provides you the details of CPU consumption, memory allocation/deallocations, and even FPS metrics on how apps does.
Looking for more information about Calabash, running tests or how test automation on real devices would work for your case? Contact us to discuss about your needs!
Happy Calabash Testing Folks!