On last week’s blog post we covered the second stage of getting to holiday readiness – Performance and Load Testing. This week we will cover how you can take the test automation assets created in Stage 1 and Stage 2 and utilize those to create automatic synthetic monitoring to ensure that your end-to-end mobile service performs throughout the whole Holiday Season.
It’s not a sprint – it’s a marathon
If you look at the most important days of the Holiday Shopping season from the online/mobile commerce point of view, you notice that it is not really a short sprint but rather a marathon. The traffic starts to ramp up already mid November and by Black Friday (November 27) the traffic is already at its peak. This continues further to Cyber Monday (November 30th) and peaking every weekend until Christmas. The biggest peak after Thanksgiving comes in mid-December when its last time to order anything physical that would be delivered before Christmas eve. Also if you look traffic from online content usage point of view the peak traffic continues well into the new year when people have time off and are able to spend more time with their electronic content.
Image reference: http://wemakewebsites.com/blog/9-important-shopping-dates-for-your-2015-diary
Synthetic End-to-End Monitoring
With this in mind, it is very important that you maximize your revenue potential throughout the whole Holiday Shopping season by ensuring that a any given moment your end-to-end user experience performs as it should. You need to start from basics like:
1. Checking that your app can be found and downloaded from Google Play and Apple App Store without any issues on most common devices.
2. Checking that the latest version of your app you have on the app stores communicates flawlessly with the current version of your production back end.
3. Checking that the most typical use cases work always: Creating new account, logging in, purchasing and upgrade/downgrade of subscription levels.
4. Check the response times for most common use cases described above.
5. Do this automatic check routine often – ideally every 10-15 minutes.
It is fairly easy to set up this kind of monitoring routine on your CI system by using Testdroid Jenkins plugin and by utilizing the automation scripts created in Step 1: Functional and Compatibility testing and Step 2: Performance and Load testing. The point is that you just select the scripts that test the most common use cases, add some scripts for fetching the latest version of your app from app store and you just configure your CI system to run these in frequent intervals against your production back-end to ensure that everything is in sync and everything works. This way you will also get live feedback that all systems that you are integrating with (such as payment systems) work as well.
Making sense of the results
Naturally running the same scripts in loop every 10-15 minutes creates a lot of test results and there can be a lot of redundant data if everything works but there are several ways to make your monitoring really smart and capable of seeing beyond all this redundant info. You can integrate your monitoring tests with generic monitoring dashboards such as Nagios or Zabbix where you can create result based rules like “If this test fails 3 times in a row, create a level 1 alarm”. These alarms can then be routed to be sent out as e-mails, SMSes or even to your favorite instant messaging/group collaboration app. This way you are always on top of what the live status of your service is and you are able to act immediately if something breaks.
Next week’s blog post on getting your mobile apps and services to holiday readiness will cover the ultimate Holiday Readiness checklist.