Testdroid Cloud has now been in closed beta for 6 weeks and we have had a great opportunity to iron out most of the obvious technical and operational challenges. We have been adding new devices to Testdroid Cloud on weekly basis and well will continue to do so. We have made several usability improvements and introduced several new features based on beta user requests. Please keep sending new feature suggestions – we will implement them as you can see from the list below. Now that the scalability is proven we are opening the beta for everyone. You can sign up by leaving your e-mail address here.
One of reasons for running a beta is to gather feedback from users to make the service more fit for purpose for real users. This is what we have done and we will continue to do this in the future as well. The features added during the beta include:
The screenshot feature of Testdroid Cloud seems to be one of the most popular ones. It is true that it is quite amazing to see your app exactly how it looks on each real Android device – pixel to pixel. Based on a user recommendation the screenshot view is now even better with Screenshot Baseline feature. Now for every screenshot on every device you can mark a screenshot to be a baseline (i.e. this screenshot looks exactly as it should) and then every time you run your tests the new screenshots will be compared to your baseline screenshot and if they are not exactly the same you will be notified. To make things even better, this feature can show you exactly how the screenshots differ from each other!
Running a specified subset of tests:
Sometimes you may may want run only certain subset of your tests or you want to run your tests in a specific order. With Testdroid Cloud you can now specify exactly which test classes or test packages will be run and you can even specify the execution order by using a comma between the test class or test package names. This feature was also suggested by an active Testdroid Cloud beta user and now everyone can benefit from it.
Recording timeline feature is actually an improvement in Testdroid Recorder but Testdroid Cloud users reap the most benefit out of this feature request. Now Testdroid Recorder has an option to capture the time between user actions during recording and then repeat the same user actions with exactly same time intervals during the playback. This feature is very useful for instance when your application accesses a web service to retrieve some data or does some other time consuming processing before the the user can continue. Especially when you are running the same test on very different kinds devices with wildly different performance this feature makes sure that the user inputs in tests are not entered too early.
How do we do all this?
A lot of people are asking how we are able to automatically control all these devices and what is the exact setup. Here are some answers: Yes, all the devices are real physical Android devices connected to clusters of Testdroid Servers. Before every test run Testdroid Server uninstalls all non system applications, formats the SD-card and reboots each phone. All this is done to ensure that each phone has a consistent starting state and to ensure that there are no data or processes running from previous test. Then when the test execution starts, Testdroid Server installs the application under test and starts executing the tests. (This in done in parallel in asynchronus way to optimise the capacity of Testdroid Cloud) Once tests are finished Testdroid Server copies the results from each phone and publishes them on Testdroid cloud. Below is a screenshot of the device management view of Testdroid Server to show how easy it is to manage tens of Android devices in one server.
And last but not least this is how our device racks look like. The sound of all these phones rebooting at the same time is amazing 🙂
Some users have been asking about the pricing of Testdroid Cloud and we can tell you at this point that there will be always some free user level and some features and higher test execution priority will cost something. We are currently working on the pricing model but we want to take our time to ensure it will be fair and that there will be options for developers of all shapes and sizes. If you like what you see on Testdroid Cloud but would prefer to run your tests on your own phones you can contact sales (at) bitbar.com for more details.