Creating the Right Calabash Tests

How to create the right tests for calabash

Dear Testdroiders,

We walked through the basic setup and installation of Calabash for Android and iOS in previous blog, and this time we’ll take a look how you can build Calabash tests for your mobile application. If you have followed the example showcased in How to Setup and Get Started with Calabash you have properly generated the file structure for your first Calabash tests on Android and iOS. There are lots of difference (best) practices of how to create your Calabash tests, and we’ll use an example of basis to illustrate how these can test templates can be generated and further developed.

create-tests-with-calabash

This blog takes a look what is the Calabash architecture and how to create tests using that architecture, what cucumber means in the context of test creation, and basically how to create your first Calabash test. We have an example application available for Android and iOS, plus you’ll find a link below where the test script can be downloaded. Furthermore, we briefly discuss about running tests on a cloud service that provides comprehensive results, data, logs, performance metrics, and highly sophisticated reporting, even with the actual Calabash logs. Don’t forget to download a copy of Calabash 101 to access it anytime.

Calabash Architecture for Test Creation

There are differences when it comes to Calabash running on Android and iOS. The basic test and test creation flow, however, remains the same. The recommended process to get test scripts done varies but the good rule of thumb is that you first create your features. All this information can be found in example below and even example provides a basic example of feature.

Then you test your feature and execute it to see what it actually does. Then you can create those Step Definitions. Step definition is basically a Ruby code that gets executed when you run your test. The following picture illustrated relation between Feature, Scenario and Steps.

Screen Shot 2016-04-07 at 2.23.31 PM

Scenarios

A scenario specifies a single behavior or use case within a given feature that is comprised of various Steps. For example, the following scenario describes the behavior of ensuring that a credit card input field has the correct length of digits:

Scenario: Run whole app
  Given my app is running
  And I touch the "Start" button
  Then I take picture
  Then I press "More Info"

Steps usually begin with one of the keywords Given, When, Then, And, and But, however, they don’t have to, they can use * in place of those keywords. In fact, Cucumber does not distinguish between them (or *). They are instead meant to provide a language hint based on cause and effect to the stakeholders as to what is being described.

As such, simply recognizing their language implications are enough to use them effectively. However for a detailed examination on these keywords, see the Cucumber Wiki entry on them.

Features

Feature is rarely defined by a single behavior. For this very reason, Scenarios can be grouped together logically under a Feature Definition. Feature definitions are typically given a name and an optional, short description. For example:

Feature: Test the entire app

Scenario: Log in to app
  Given my app is running
  Then I take picture
  Then I use the native keyboard to enter “username@domain.com” into the “your name” text field
  Then I use the native keyboard to enter “myPassword123” into the “password” text field
  Then I press "Sign In"

Scenario: As a valid user I can start using the app
  I wait for text "Hello"
  Then I wait for activity "HomeTabActivity"
  Then I press view with id "menu_compose_tweet"
  Then I enter text "Testdroid" into field with id "edit"
  Then I press view with id "composer_post"

Steps and Step Definitions

Step definitions are like code-behind for the scenarios defined in step definition scrips. They provide the glue that makes them runnable in the application. Their function is to translate readable texts into runnable actions. For example, step definitions for used example could be as follows:

require 'calabash-android/calabash_steps'

Then /^I set screen to portrait$/ do
    perform_action('set_activity_orientation', 'portrait')
end

Then /^I set screen to landscape$/ do
    perform_action('set_activity_orientation', 'landscape')
end

Then /^I hide keyboard$/ do
    hide_soft_keyboard()
end

We’ll take a look of steps “I set screen to portrait/landscape” and “I hide keyboard” examples below.

Calabash Architecture / Infrastructure – How Calabash Works on Real Devices

The great part of Calabash is that it is a cross-platform framework and works really well for both major platforms, Android and iOS.

On Android, Calabash uses Android instrumentation test server (separated application) that is based on ActivityInstrumentationTestCase2. All Android tests are executed on the server side and users can flexibly use predefined or custom steps. Looking at the picture below, Ruby client library translates Features and commands defined for those features to instrumentation calls. If application uses Webview, the execution is injected through JavaScript.

With iOS, Calabash installs a HTTP server as an instrumentation package that then listens commands from Calabash server. Tests are also executed on server side and each test case is described in Cucumber. Slighly different than with Android, the Calabash iOS converts Cucumber commands to ‘FRANK’ method calls that then get executed. The webview support is done the same way than with Android.

android-ios-server

 

Getting Started with Test Creation

To save time, we’ll use an example application and ready test package. The BitbarSampleAppTests.zip file contains a functional test script. You can easily create similar file set for using calabash-android gen command. If you are looking for Calabash-iOS Tests you can basically use the same test package but you need to fetch the .IPA file.

Screen Shot 2016-04-07 at 4.46.22 PM

As application is relatively simple, we’ll start by creating a Feature, written in Cucumber. The possible scenarios for user interaction in the above example would be that user 1) clicks through the radio buttons, 2) types a name in input field, 3) hides (or after hiding, asks to show) the keyboard, 4) changes the orientation of the screen, plus asking test to take screenshots at any given moment.

Feature: Test UI Components

  Scenario: Test Radio Buttons
    Then I press view with id "radio0"
    Then I take screenshot
    Then I press view with id "radio1"
    Then I take screenshot
    Then I press view with id "radio2"
    Then I take screenshot

  Scenario: Input Text and Change Orientation
    Then I set screen to portrait
    Then I take screenshot
    Then I set screen to landscape
    Then I wait
    Then I enter text "Hello Calabash" into field with id "editText1"

  Scenario: Test Keyboard
    Then I take screenshot
    Then I hide keyboard
    Then I take screenshot
    Then I wait
    Then I show keyboard
    Then I take screenshot

  Scenario: Click Button
    Then I press view with id "button1"

If you look at my_first.feature it basically contains the same steps as shown break-down in the scenarios above.

Running Calabash Tests and Interpreting the Results

The reporting and test results are important to understand how application does under the testing. These reports and results should yield all fine-grained details about the test execution, performance, and provide users actionable and rich information about tests and test runs. Furthermore, it’s important to get that test result data as real-time as possible, and those are stored based on each application regression, test executed against the specific version of app, and that you have a way to compare results based on each app and test against the older ones.

First package your test zip file to include all files that has been either generated (e.g. calabash-android gen) and modified/added while you created the test. The folder structure (regardless if you have added new tests, steps, features etc.) should look like this:

Screen Shot 2016-04-05 at 11.56.16 AM

After you have created a package of it, just log in to Testdroid Cloud and upload your APK or IPA, and this zip file. If you are not familiar with the interface, there is a full-blown guide of how to use real mobile devices on cloud for testing. This guide will provide you all necessary steps to access devices, get tests executed, and finally provides you a comprehensive reports of both, test details and Calabash report.

Screen Shot 2016-02-09 at 14.06.37

Screen Shot 2016-02-09 at 14.06.59

Naturally, when you Calabash test script is more complex you probably want to check also performance aspects of the app running on specific device. The same view provides you the details of CPU consumption, memory allocation/deallocations, and even FPS metrics on how apps does.

Looking for more information about Calabash, running tests or how test automation on real devices would work for your case? Contact us to discuss about your needs!

Happy Calabash Testing Folks!

 


Calabash 101: Basics, Getting Started and Advanced Tips

Go through the basics of Calabash, how to create proper Calabash tests and how to make the most of them.

Download

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close