Top 4 Characteristics When Considering Cloud Testing Technologies

Bitbar, the mobile devops company. Logo, large

Here at Bitbar we solve large problems with mobile development and quality assurance. From being able to tackle the device fragmentation issue, to providing a means for thorough agile continuous integration testing, Testdroid has become a great solution for companies tired of slow dev cycles and growing QA costs with inadequate returns. Companies need to scale their testing coverage, but at the same time accelerate feedback on bugs to development. That’s a big catch 22!


We believe in the power of modern test automation and QA infrastructure technology that scales effectively to meet requirements. Automation for mobile application testing has traditionally been riddled with challenges. With so many device platforms, screen sizes, etc. folks have had to customize scripts on a device to device basis using proprietary frameworks. Even if folks have all the devices they need in house, it’s been all but possible to orchestrate testing across the HW in a fully automated way.

We are coming into an age where the power of open source object driven testing frameworks allow folks to write scripts on fewer devices, then be able to execute against many. A good number of cloud based platforms have popped up over time to help with device access. We believe our testing infrastructure technology that allows tests to run against any number of devices at the same time is a game changer. Fundamentally being able to use any open framework with broad device coverage and swift throughput is the basis of value for Testdroid.

Top 4 Characteristics When Considering Cloud Testing Technologies

Cloud testing technologies have their own perks over the local ones. If you plan to use cloud-based testing technologies, here are few important things to consider:


Using Open Source

There are a good number of automation frameworks available now for mobile application testing. A few years ago there were only a few options and most of them were proprietary platforms. These have disadvantages in that you are dependent on the limited capabilities that a vendor can provide and if you need to do something outside of the box you don’t have many options other than begging for enhancements and having to wait. The maturity of open source testing frameworks such as Appium, Calabash, UI Automation and Espresso have come a long way. There are vibrant communities surrounding these platforms and lots of contributors. Most use cases are covered and you have the ability to branch for your own needs. There are a few cloud vendors supporting open source, but make sure that you’re not having to use custom libraries or methods in order to make things work, as this limits the ease of executing locally on your own devices and creates vendor lock in.


Test Parallelization

What good is a cloud if you are limited by the number of tests that can be running at one time? Increased device coverage is great, but we believe decreasing test cycles times should be a primary goal. We see the best automation practices integrate automated tests into their CI (continuous integration) process. In this case, throughput becomes a huge factor. With other cloud vendors, there are hidden costs of increasing concurrency generally charged through virtual user licenses or subscription plan restrictions. Some vendors just know that their systems can’t handle the work load. From the day one, Testdroid was built to execute tests concurrently at whatever scale is desired.


Device Control and Support

There are a few different ways that cloud vendors control and connect devices. Some use proprietary circuit level integration, cradle systems or SW agents. These approaches create blockers around being able to support new devices in a timely fashion. It could take weeks, if not months to support the newest devices. We believe in using the standards that are included from Apple and Google that use the USB port such as Xcode and ADB. These open, widely used connection methods allow efficient instrumentation control together with open source frameworks. The big value in this is that new devices can be brought online immediately when they hit the market. It also keeps the cost of device support down relative to proprietary approaches.


Environment Flexibility

Not everyone is going to be able to use a public cloud offering. Organizations within industries like financial services, insurance and healthcare might have strict security guidelines around using multitenant environments. Pre-production systems might need to be accessed behind the firewall. End to end tests could use a mix of tools beyond the basic frameworks or there might be additional HW required for test flows. Some organizations have so much testing to do that a usage based system wouldn’t make sense. In these cases, you should be able to use private cloud or on-premise system options.

8 Questions on Cloud Testing Technology and Vendor

Screen Shot 2016-02-04 at 12.42.21 PMIn my opinion, I believe folks should be asking the following questions when looking at mobile testing technologies and cloud offerings:

  1. Am I reducing test cycle times?
  2. Are there enough devices supported?
  3. Am I able to integrate with my process and CI objectives?
  4. Is there adequate support?
  5. Can the testing environment be customized?
  6. Am I bound to a public cloud or are there private and on premise deployment options?
  7. Is it cost effective?
  8. Is there vendor lock in?

Testdroid provides positive answers on all of these points.

Does this checklist provide you a handy way to compare cloud testing technologies? Weigh in with a comment below – or contact me directly at dave dot horner at bitbar dot com.

Happy Cloud Testing Folks!

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.