Skip to content

Like in all software projects, products and companies, the process of writing down documentation about requirements sparks never-ending debate of how it should be done, how much is enough, what and how certain feature requests and requirements should be documented. We are no exception here at Bitbar.

In this blog, I’d like to explain how we have approached this challenge, what have been the pit-falls for us, and how we have overcome of those issues. It’s just not about documenting requirements and specifications, but how to get those in our every-day doing and make these our strength – and build an incrementally better product and the requirements documentation.


The Challenge – Working Software vs. Comprehensive Documentation

We’ve adopted Agile methodology pretty much in all fine-grained of details here at Bitbar, but every now and then we come back to ourselves to ask whether Agile Manifesto holds the right approach: “Working Software over Comprehensive Documentation.”

Some claim building, maintaining and improving the documentation is a tremendous effort and in fact, one of the biggest hindrances to progress in a software project. Then, as an organization, you have to draw a line to somewhere and ask yourself “Why do we really need documentation” as the following things are still yet more important:

  1. be sure the system behaves as it should
  2. be sure our customers will be able to get the best result out of Bitbar products
  3. be sure that new feature are being implemented (and also adopted by users) in the best way

While point 1 and 3 have a smaller scope, typically inside the R&D department, point 2 has a much wider impact, not just to our users, but also to our support and sales team which are our direct contact with the customer.

When you have a complex system with components spread in different continents, ensuring that the system behaves as it should not be a simple task. In addition, our own software infrastructure includes methods and tools that are extremely targeted for certain components and can expose what’s going on those. In short, we make use of several tools such as:

  • JUnit Tests (triggered at every change in the code)
  • Automated Selenium Web UI Tests
  • and yes – we still do lots of manual testing during development as regression testing and exploratory testing
  • Service monitoring tools
  • Open discussion and open documentation for things that (people think) matter

At this point comes the challenge: how to have a common “understanding” of the system?

Screen Shot 2016-01-14 at 1.50.29 PM

In the past, we’ve tried some different approaches or tools like JIRA Plugins for Requirement Specification / Test Case definition, written documents, intranet sites where requirements and documentation were hosted in a lightweight manner. All these different forms of requirements documentation gave us some benefits – like traceability of the progress and healthiness of the product, possibility to share across all the company, less effort to maintain documentation (respect to typical doc oriented documents) and so on.

The Point of Failure

At a certain moment we started to fall into the classical R&D debates, and asking questions like “it is a bug or a new feature?”, “How this actually should behave?”, “Where are the specification for that old thing?” started to frustrate us and slow down the development just at the moment we wanted to go faster and be more agile!

Additionally as there were several tools or places to host the requirements documentation it was not clear what needs to be updated to end a sensible loss of information. A solution was needed and – luckily – we figured it out.

The Game Changer

When we started to develop automated web regression tests for our products, we opted in for a quite typical setup:

This last little guy provided us what we were missing: the Gherkin Language.

As many of our users already know (e.g. Calabash developers) the great thing about Gherkin language is that it is a human understandable language that also machine can understand and interpret in an intelligent manner. This was proven to be successful and gave life to our requirements that can be written in form of .feature files, which are simple text files with some keywords. For example, a Calabash Feature file could look like this:

Feature: Login
Scenario: Successful Login
     Given user open the login page at “”
     When username is set as “”
     And password is set as “mySecretPass”
     And the sign in button is pressed
     Then the user is logged in
     And the project page is shown
Scenario: Some other scenario

A simple file like that can be understood by everyone, the product owner, the developer, the tester and anyone from sales too, as well as the computer that runs automated tests if you use a framework like behave, jBehave, Calabash or one of the many others.

From the technical perspective, in order to make this file understandable by a machine you just need to implement the step definitions (in our case in a Python file) to interpret human-readable language for the machine, for instance:

from behave import *
@Given('user open the login page at  “{home_page}”')
def step_impl(context, home_page):
    # go to the cloud home_page
@Given('username is set as “{user_emai}”')
def step_impl(context, user_email):
    # set the username in the input field

Getting in Speed with The Right Approach

Actually, to start writing these types of feature files simply wasn’t enough, but the process used for development and testing needed some changes too. You know, as a tool is just a tool, it needs good hands to be properly used 🙂

The next step we did was a decision with the PO that all new user stories must be provided with one or more feature file written in Gherkin language. As the purpose of this language is to allow different parts to sing the same song we added a constraint: feature files should be “reviewed and accepted” from both PO, QA and developers, and everyone should provide their input in order to better describe what the feature means.


We started to implement lots of feature files and once the number of those files were increasing rapidly, we needed a way to organize those files in a proper manner to make all information accessible by all stakeholders – and this is where we adopted more structured documentation to maintain file information.

How We Organized Calabash Feature Files

From the beginning, we hosted these files in our GitLab repository so that we can check what and when is something was changed and eventually deal with requirements for different versions or flavors instantly. With Calabash, it’s actually important to get the folder structure – all Calabash Feature files – in decent order.

Then, we discussed a lot how to organize the feature files and finally we agreed to create one separate folder for each “Feature Group” like the “AccountsAndUsers” or “Subscriptions”. Inside each feature group folder, we have 2 subfolders, and the feature folder contains all feature files related to that group each one with a UID as filename suffix (e.g. ACC001_Login).

Under the “steps folder” there is all the Python step implementation, then there is an file with some logic on flow handling (you can define what has to be done before or after a feature, step or Scenario). We believe this approach is the best for us at the moment as it makes it easier to check which feature file you need to change or where to place a new one if you need to introduce a new behavior:

  • root_dir
    • AccountsAndUsers
      • feature
        • ACC001_Login.feature
        • ACC002_Registration.feature
      • steps
    • Subscriptions
      • feature
      • steps

Using this approach has shown that the Behave framework had a limitation as it does not support such a hierarchical folder structure. The Behave command can just run all the features contained in the specified folder or in a subfolder called feature; in order to solve this small problem we decided to use Python again, a small script that launches the behave command looping into all the Feature Folders, that way we keep the same behave context and we can detect issues like duplicated step definitions etc.

The code looks as simple as this:

from behave.__main__ import main as behave_main
for group in sorted(listdir(root_dir)):
	feature_group_folder = '%s/%s' % (root_dir, group)

Improving Documentation Accessibility

In fact, all this we did was not going to improve the accessibility of the requirements and the old (but yet unmaintainable) specification intranet site was still a step ahead.

The new challenge was taken and we developed a small Python tool (testdroid-documenter) that goes through all the feature files and generates a Sphinx set of documents out of which we generate HTML version deployed at every regression run on a web server in our intranet… et voila’!

Screen Shot 2016-01-14 at 2.04.34 PM

The last but really important decision we made was to completely switch to Gherkin language both for describing the behavior of legacy part of the system and to replace the JIRA plugin we used to define and trace the status of the manual regression. We found out that those text files were much easier to maintain than issues in JIRA.

As we do not develop automated tests for all feature files or all Scenarios we define, in order to do not break the automated regression execution we just used another really handy feature of the behave framework: the tags. In fact we just defined a tag @ManualTesting and configured in the behave.ini the property so that all features or scenarios with the manual testing tag can be ignored:

# Just skip the execution of Manually executed test cases
tags = ~@ManualTesting
(same can be obtained by using a cl parameter --tags)

Short Summary of Our Setup

So, to summarize our work for improved documentation, testing and internal process enhancement, we got the following things in place:

  • A GitLab repository of Feature files organized in a hierarchical folder structure that composes our requirements
  • A set of Python files with the actual implementation of the steps defined in the features
  • A small but handy Python tool that makes does magic generating nice HTML pages for the requirements documentation
  • Some Jenkins job which executes automated regression and triggers the documentation update on the intranet
  • … most important the whole TEAM that speaks the Gherkin language!

The most important achievement was that we are now working more efficiently than in the past – and according to the Agile Manifesto – we do have code describing our product behavior. This only leaves us to produce the User Manuals, other additional help pages and some examples that are shared at Github.

The Way Forward – What’s Next?

Step definition autocompletion

As feature files actually are “code” all coding rules applies to them – and especially “no-copy”. Once you start to write hundreds of steps you want to avoid the situation where you have similar steps, such as: “the user input his username as “xxxx” and “the user set his username as “xxx”.

In order to provide useful code as quickly as possible, we want to enable also non-programmers to easily write those feature files. There are plenty of great tools available but at the moment only Calabash offers a real step autocompletion.

JIRA integration

Currently we use a JIRA plugin that permits to define test cases and steps, and the nice side of this plugin is the possibility to trace the progress of the manual regression. Furthermore, we’ve decided to automatically generate JIRA issues (not bugs but something like “Manual Test” issue type) for tracing the manual regression test activities we do before each release as well for the automated tests. In that way we can easily check the progress and the healthiness of the product – and instantly.

Inspect and adapt

Finally we do our retrospectives to be more agile and efficient for sure!

Happy Testing (and documenting your regressions with Calabash Feature files!)


Article author