Day 2 of GTAC 2014 was at least as interesting as day one if not even more varied in terms of topics and different approcaches. Here are some of the highlights of the day. Check also the Day #1 recap here.
How Facebook uses bots to manage tests
Roy Williams from Facebook gave a great presentation how Facebook uses bots to identify and disable automatically flaky tests. Back in 2009 Facebook’s challenges were that they had too many bugs in production which led to too much developer time spent on firefighting the production issues. As a solution they introduced automated testing which got the initial problem under control but as the number of automated tests grew, the number of failing tests grew even faster leading to a vicious circle or developers losing trust on automated tests, ignoring the results and disabling tests very easily. As a solution they implemented project greenhouse to rebuild trust, keep tests in good state without human intervention and trade human time for machine time. The outcome was a sophisticated system that automatically identifies and disables flaky tests so that test results could be trusted again. Great use of technology and great success!
Server side mocking frameworks
One recurring theme on day two was the use of different frameworks/tools for mocking the server side responses to isolate potential sources of random failures like network delays or even to work around missing server side features. Michael Bailey from American express introduced WireMock.org for this purpose and then a bit later in the afternoon Bryan Robbins and Daniel Koo from FINRA introduced MSL (pronounced Missile) Mock Service Layer for the same thing. MSL can be found at https://github.com/FINRAOS/MSL. This server side mocking thing seems to be getting more and more popular and naturally that is needed for achieving hermetic tests.
TESTING USER EXPERIENCE!
Alex Eagle from Google presented very interesting angle to test automation. Most of the presentations and work in test automation is around frameworks to make tests run but very little work is about presenting, analysing and utilizing the test results. This is very important area especially when you get results from hundreds or even thousands of devices and you have to abstract and visualize the results somehow. We at Testdroid have spent quite much effort on thinking how you can visualize results from very large number of test runs and will be introducing more tools for that in the future. He introduced a very welcome initiative in this area called Rich Test Results, an open source project which aims to provide an interoperable format for representing all the results of a test execution, making it much easier to reason about failures and identify the root cause. The project is still in very early stage but think this is a very important area for improvement and will embrace it fully. The project can be found at https://github.com/Google/rich-test-results
The last presentation of the day was very entertaining presentation from the same team who have created Espresso framework for Android and are developing the Android emulator infrastructure at Google. It was interesting to see that they had developed many similar solutions for setting up their emulator infra as we have done to our real devices. For instance ensuring each device is in consistent state before every test, using Testdroid Device Service to provide features like changing device language, controlling device settings and so on. These guys do really have a big task ahead but it seems that they are very determined to crack it.
All GTAC2014 presentations can be viewed at https://developers.google.com/google-test-automation-conference/2014/stream