After selecting your target device set you need to take a look at what other infrastructure you need for a robust, enterprise-grade test lab. Let’s start with the most important pieces:
Device Control Servers
Regardless of the technology, you use to build your device lab with you will need some sort of device control node servers that will take care of managing the devices and possibly executing the tests. Additionally, these servers can collect, process and store test results and do other test execution related tasks. At Testdroid we use either generic PC-hardware running Ubuntu Linux for controlling Android devices or Mac Minis for controlling iOS devices. Because we use the OS vendor provided methods to control the devices (Android Debug Bridge (ADB) on Android and Instruments on iOS) the control nodes/servers need to have a decent number of free USB ports as each device will be connected to the device servers with a USB cable. You can download this complete ebook to learn the whole things about building your own mobile device lab.
In order to have a stable test execution infrastructure you need to take the following things into account when selecting the hardware for device control servers:
- Hard Disks: You need to have enough disk space for all test artifacts (logs, screenshots, videos, performance data, memory dumps, network dumps etc) and make sure that there is a mechanism that either deletes the data created by previous test runs or transfers it to somewhere where there is enough storage space. Running out of disk space is one of the most common failure points in large scale test automation. You also may want to consider investing in SSD disks to make sure that writing the test results to storage does not become a bottleneck under heavy load.
- RAM: You need enough RAM to run several virtual machines in one physical device control server as that is the best way to ensure that each test run starts from an identical starting point. Also running virtual machines is required to run several iOS Instruments sessions on one Mac.
- Energy efficiency: If you are building a large device lab you will potentially running tens of device control servers as well so keep in mind that the more power you hardware draws the more heat those will generate and more cooling you will need.
As all devices will be connected to the device control servers over USB you need to pay attention to the quality and capabilities of your USB hardware. This is one of the most error-prone parts of the system and while there are no fixed hard limits on how many devices you can plug into one USB bus our experiments show that you should not connect much over 16 devices per each bus. If you connect more the devices may start disconnecting randomly especially when a lot of data is being transferred either at the beginning or end of each test.
The other important reason why good quality USB hubs are essential is that many devices (especially ones with large displays) have problems with charging if the charging current is not high enough. The USB specification mandates that the output current is limited to 500mA when the USB port is on data mode but the best USB hubs on the market allow you to programmatically switch between data and charging modes allowing you to implement automatic charging cycles for devices that cannot be charged in USB data mode.
To make sure that there are a few USB related problems as possible we prefer professional USB hub hardware by a company called Cambrionix. Those are sold under several names and at least ones sold under the Datamation brand are great. Just be careful when selecting those as some models are just for charging and some models are just for the iOS device.
Wifi infrastructure is another very important infrastructure area that is often overlooked when creating large scale mobile test automation. You can get to about 10 maybe even 15 devices with any Wifi infrastructure without any problems but as the number of devices in your Wifi network adds up, so do the problems. Quite often all devices are able to connect to the wifi without problems but the problems appear when all of those start transferring data at the same time. Most Wifi access points are not designed for this kind of throughput and you will start observing different kinds of timeouts on server responses etc. Network throughput on typical Wifi router is around 100-150 Mbit/s but and when you have several dozen wireless devices trying to download at tens of Mbits/s you can easily see where the bottleneck is.
There are several brands on the market selling enterprise-grade Wifi access points and we have tried at least Aerohive, Cirrus and Cisco Meraki. Our current favorite is Cisco Meraki which has three separate radios each capable of working on either 2.4Ghz or 5Ghz frequency band and each of them can transfer up to 300 Mbit/s giving whopping 900 Mbit/s from your devices to the back end servers. Another nice thing about Cisco Meraki Wifi infrastructure is their cloud-based management UI that has features like throttling the speed of individual Wifi connection which can be useful for instance when simulating different network conditions.
Integration to your build process
Another important consideration is how you integrate your in-house device lab to your build process. Most likely you have already a separate Continuous Integration server in place and you have to make sure that your test infrastructure can be controlled by your CI server without any manual interaction. This means that everything needs to be controllable over an API so that your CI jobs can seamlessly both trigger test runs and use the test results from your in-house test lab to trigger further actions such as deploying your app to production.
Access to your organization’s centralized access control system
Last but not least of infrastructure considerations is that if you are building your in-house device lab in an enterprise setting you need to be able to connect to your company’s centralized user right management system in order to validate, grant and revoke user rights for each user in your test automation system. This is just to ensure that any unauthorized users are not able to access data and binaries that they are not supposed to and to synchronize these access rights when people leave the organization, change roles or simply do not anymore need access to such data. Most common protocols for doing this are LDAP (Lightweight Directory Access Protocol) and OAuth and your infrastructure should validate each user login through such a centralized system.
Next, we will cover tips and tricks on how to make this setup so robust that you can have your whole development organization relying on the results your in-house lab is producing for every build.