Getting Started with Robot Framework on Bitbar’s Mobile Device Cloud (with Example)

Robot Framework is a generic keyword-driven test automation framework for acceptance level testing and acceptance test-driven development (ATDD). Surprisingly, many of you have been using this framework (in the past) for a basic acceptance testing that extends the system level testing capabilities with specifications and test cases associated with the actual app testing. However, if you are not familiar with Robot Framework or if you look for a way how to use it in Bitbar Device Cloud, I’ll walk you through some basic things in this blog.

What is Robot Framework and How Does It Work

Robot Framework is extremely easy to set up, use and modify to get both Android and iOS apps tested. The test syntax that it uses is based on keywords and these keywords are quick to edit and further configure to make a match with the application under test. Furthermore, testing capabilities provided by Robot Framework can be easily extended with the test libraries which can be implemented using Python, Java and even some other languages are supported.

The nifty thing with Robot Framework is how those mentioned keywords work. Users can quickly create new keywords, either using the existing example ones or by writing everything from scratch.

If you look for more generic information about Robot Framework, there are plenty of great examples and online documentation in GitHub.

Get started with successful Appium mobile app testing with the help of this  free guide

The Basic Setup of Robot Framework

To get started with Robot Framework basically you need Python and pip installed. If you have the environment properly configured you can just execute the installation command lines:

pip install robotframework
pip install robotframework-appiumlibrary

After you have installed Robot Framework, you can take a look at the example where we use our Bitbar Sample Application with very basic Robot Framework test examples. For the cloud run you basically need a Python script and shell script that makes sure the test execution on cloud end works fine. For instance, you need to add the pip installation in your shell script to ensure everything gets properly installed on your cloud session as well.

In this example, we’re using Appium as a basis to run the test and make sure everything gets logged for further inspection.

## Cloud setup
echo "Starting Appium ..."

/opt/appium/appium/bin/appium.js --log-no-colors --log-timestamp --command-timeout 120 >appium.log 2>&1 &

# Wait for Appium to fully launch
sleep 10

ps -ef|grep appium

echo "Extracting tests.zip..."
unzip tests.zip

echo "Installing pip"
curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user

echo "Installing requirements"
~/.local/bin/pip install -r ./resources/requirements.txt --target .

## Start test execution
echo "Running test"
pybot -x TEST-all tests/android_example.robot

This should be all that you need for a shell script. To create a compatible test you need to add a few lines to ensure all paths are recognized and test cases can be found from the right fileset. The file structure for the test package can be something as follows:

Screen Shot 2016-08-03 at 6.39.26 AM

The following example includes both Android and iOS application in the test package (under resources ->app folder). The test definition can be found under ‘tests’ and testing libraries are included under libs->examplelib. In addition, you can easily and quickly tweak and create your own test libraries with the instructions shown on the Github documentation of Robot Framework.

Want to run your Robot Framework tests against real devices in the cloud?

We offer a mobile device cloud with real iOS devices for automated and manual testing. Sign up for a 30-day trial.

What Data Can Robot Framework Produce

When running a Robot Framework test in Bitbar Cloud users do not need to do any significant tweaks for their test runs (or job configs). The test automation and execution works the same way as with any other framework, and results are quickly available from any of those Android and iOS devices.

If you are starting out test automation withBitbar Cloud and planning to use Robot Framework, simply create a project (Android or iOS), upload your application (APK or IPA), upload the test package, and select whatever devices you want to use for a test run. Just in a few minutes the test run will be done (naturally this depends on the length of your test case) and the following data can be fetched from a test run:

Screen Shot 2016-08-02 at 10.04.31 PM

This example includes three basic test cases (we’ll come back to those in the next chapter of this blog). However, after the test run is finalized you have now all the data in one, compact view of Bitbar Cloud. If you click any of those files shown in the overview you’ll get full log opened in the view below:

Screen Shot 2016-08-02 at 10.04.10 PM

For instance, the shell script file that we introduced in the first chapter of this blog extracted the tests.zip, installed the PIP, setup other tools and finally executed the example tests. The output of the log can be also inspected in this view – and for example, it will look something like this:

Imports done
==============================================================================
Tests
==============================================================================
Tests.Example
==============================================================================
Simple Smoke Test - Correct Answer | PASS |
------------------------------------------------------------------------------
Simple Smoke Test - Wrong Answer | PASS |
------------------------------------------------------------------------------
Simple Smoke Test - Wrong Answer 2 | PASS |
------------------------------------------------------------------------------
Tests.Example | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==============================================================================
Tests | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==============================================================================
Output: /home/ubuntu/test/output.xml
XUnit: /home/ubuntu/test/TEST-all.xml
Log: /home/ubuntu/test/log.html
Report: /home/ubuntu/test/report.html

In addition, to get a comprehensive understanding of how the app (with the test) really does on the physical device, you’ll also get performance data about the CPU and memory consumption:

Screen Shot 2016-08-02 at 10.04.21 PM

Okay, that’s some fundamentals for creating test runs using Robot Framework and managing the data after tests are done. Let’s take a look of how easy those test cases are to build and what you need to know about Robot Framework to use it efficiently with Testdroid Cloud.

How to Create Robot Framework Compatible Test Cases

First, some generic settings are defined for application and test session. Resource links to a file where all common procedures are provided in the test package. Test Teardown provides instruction of what system should do when test session is done:

*** Settings ***
Resource                 ...${/}resources${/}common.robot
Test Teardown            Close Application

For the variables, creator of a test should define some basic things here. In the Bitbar sample application, there is a name field (editText) and one button (button1). IDs and actions can be defined in this section:

*** Variables ***
${NAME}                  John Doe
${INPUT_NAME_FIELD}      id=com.bitbar.testdroid:id/editText1
${SUBMIT_BUTTON}         id=com.bitbar.testdroid:id/button1

The example application looks like this:

Screen Shot 2016-08-02 at 10.23.28 PM

Now, let’s look at those test cases (using Android). To create three different test cases for the application, we start with a simple smoke test where the correctness of selection is validated. The correct answer in this scenario is when the option to use Testdroid Cloud is clicked and the name is entered in the text field. Two other test cases indicate the wrong answer with radio buttons (Option to buy 101 devices and ask mom to help you). Test cases are as simple as follows:

*** Test cases ***

Simple Smoke Test - Correct Answer
   [Tags] cloud
   Set Up And Open Android Application
   Input Name ${NAME}
   Select Option Use Testdroid Cloud
   Submit Selection
   Validate Correct Answer

Simple Smoke Test - Wrong Answer
   [Tags] 101
   Set Up And Open Android Application
   Input Name ${NAME}
   Select Option Buy 101 devices
   Submit Selection
   Validate Wrong Answer

Simple Smoke Test - Wrong Answer 2
   [Tags] mom
   Set Up And Open Android Application
   Input Name ${NAME}
   Select Option Ask mom for help
   Submit Selection
   Validate Wrong Answer

When test execution is started, Robot Framework first parses the test data. After that, it uses keywords provided with the test library/libraries to interact with the system.

In the keyword example, there is an input name, select of options (3 radio buttons) and submit of a selection (=a button press). In addition, the keywords section can include validation of correctness. In the example below, selecting the right radio button uses xPath to find a name included in the actual UI element:

*** Keywords ***

Input Name
   [Arguments]      ${name}
   Input Text       ${INPUT_NAME_FIELD} ${name}
   Hide Keyboard

Select Option
   [Arguments]      ${option}
   Click Element    xpath = //*[contains(@text, '${option}')]

Submit Selection
   Click Element    ${SUBMIT_BUTTON}

Validate Correct Answer
   Wait Until Page Contains      Congratulations ${NAME}       5s

Validate Wrong Answer
   Wait Until Page Contains      Wrong Answer                  5s

Conclusion and Download Link

Robot Framework is very handy and easy to use. Some of you have been using it in the past so in case you have your tests implemented, and you have your Android / iOS app ready for the test, give this framework a try on Bitbar Cloud. You’ll be surprised at how smoothly things work and how quickly you can do acceptance testing. Furthermore, here is the full test package available for a download (and tweaks).

Have fun with Robot Framework on Bitbar Cloud!


The Beginner’s Guide of Using Appium for Mobile App Testing

Learn all the basics about Appium, how to set it up and how to use it for mobile app testing.

Download