Although Bitbar Testing is our most visible service, our company has been building in-house test labs of different configurations since 2010. The size of these labs varies from 20 devices to several hundred devices and while setting up and even operating some of these we have learned a thing or two on how to build operate and maintain an enterprise-grade in-house test lab.
In this blog series, we are letting you into the secrets of how to build large scale device labs meaning more than hundreds of devices at the end of the spectrum. As nearly all of the tips apply to smaller setups as well, the focus will be on professional, robust and scalable setups. We have also combined this series into a free ebook for your convenience. Download it today.
Top Reasons for In-House Test Labs
Cloud-based test automation services (like Bitbar Testing) whose device catalog extends to hundreds of unique device models from across the world are great for most mobile developers because they offer the full breadth of both smartphones and tablets but there are several reasons why some companies cannot use such services. The most typical of these reasons are:
1. Pre-market devices
Surprisingly many top tier developers work with pre-market devices. Typically you are working on a significant enough app that will ship with a device and you have to make sure that your app works even when the actual device software changes from release to another. Typically you are working inside or together with a device manufacturer, carrier or one of the TOP100 app companies that do get access to pre-market devices from top manufacturers.
2. Corporate security policies
Many companies have strict security policies regarding uploading development versions of their unreleased applications outside of the corporate firewall. Quite often these companies are also very professional with their testing and setting up their own in-house device lab makes a lot of sense to get the scalability and time to market benefits of test automation on real devices while maintaining required security levels.
3. Availability and usage level
Sometimes the QA teams need to run very long soak tests that take all night or there are tens of development teams relying on the availability of the test results every day so the shared infrastructure of public cloud-based services does not fit well to the usage profile of these customers. If you are running a lot of tests (potentially from every code commit), your tests take a long time to execute or you are relying on having the test devices available when you need them an in-house device lab is a way to go.
4. Special setups
Some companies require specific hardware or environment setup to be able to fully test their software. For instance, some special Bluetooth or Wifi enabled accessory needs to be physically in the same location as the test devices or the devices need to be in some specific location or in some specific carrier network. Many of the environment-related characteristics can be simulated but not all and therefore this sort of special case needs to be run in specific in-house labs.
As you can see the reasons for setting up in-house device labs can be as varied as the companies that are setting those up. An in-house lab is by far the most flexible and secure setup and it accommodates for a wide spectrum of different hardware setups as well as wildly different usage profiles from 24/7 continuous testing to providing remote access to specific device prototypes. While the configurations and the user profile may differ the basic principles of building and operating the test labs are universal and we will be covering those in this blog series.
The Core Benefits of Continuous Integration & Testing
One essential tool for mobile application developers to use in order to automate their testing effort is continuous integration software – for example, Jenkins, which is currently the leading open-source continuous integration (CI) server monitoring executions of repeated jobs. The CI is the process of continuously integrating the code that developers create frequently – at least daily and often multiple times per day – into one consistent application package.
1. Version control and build revision
This rather old practice in the software industry is the same as mobile application development. The quicker you can ensure your additions and regressions to builds work properly, the more quickly you can move forward. On the other hand, if you face problems or found any issues, all attention to fixing those bugs can be drawn there right away. Fixing a bug today costs much less than fixing the very same bug tomorrow.
2. Build automation, instant execution, and testing
Build Automation refers to the ability to either automatically trigger or build a clean version of your application. As many mobile application developers take this further and allow their systems to be build from source code just committed to version control. This typically shortens the build time significantly but also makes the application ready for execution and testing instantly. This is one of the greatest benefits that continuous integration can bring to you.
3. All types of testing enabled
By automating mobile application testing developers can ensure that their application behaves correctly on every mobile OS version & build (30+ with Android, 10+ with iOS), which will reduce costs and increase the efficiency of the overall testing process. Not only the basic unit and GUI testing but testing in continuous integration environment provides a way to adopt any test frameworks, methods, complementary technologies and so on, for their continuous testing and development environment.
4. Frequent commits, code consolidation -> Fast builds
Time is money. And fast builds give you that time. Fast builds allow developers to receive test results (= bug report) faster.
5. Continuous Integration/Testing is the right way to go forward
By adopting continuous integration you are not only improving the quality of your code and process but also making your development and testing effort much faster, more consistent and easier to measure. These are significant benefits to be gained out from simple process change and adopting something that already exists in processes.
Which devices do you need for testing? How you select the right ones for your in-house test lab, how many of those you need and how to acquire and prepare them for your in-house test lab use.