Using public cloud for mobile app testing has been an exponentially growing trend. Amazing number of app developers have migrated from their local – and nearly in all cases – manual testing practices to test automation on cloud device farm where devices are available 24 hours per day, seven days per week. One of the top reasons have been the lower maintenance and operational costs, as well as device acquisition costs. While migration to device cloud farm has lots of benefits for variety of app, game and web developers, not all companies can use cloud-based services. However, there is always ‘private cloud’ option that gives even greater benefits and actually provides the most cost-efficient way to get mobile apps, games and web related stuff developed and tested.
There are lots of reasons why cloud-based services cannot be used by certain types of companies. Banks are great example. Due to their internal regulations, security and privacy concerns, the use of public cloud services isn’t an option within this segment. But could private cloud tackle these challenges – and if yes, how? Well, they actually can, and provide the most optimal solution for high security-concerned instances.
We can define “private cloud” as a cloud service that is operated by someone on data centers and locations outside of the actual site where development, source code, and all critical assets are resided. We can also define private cloud as an internal cloud setup that is on-premise but provides cloud-alike access and solution for these types of instances internally. Typically those are called ‘on-premise solutions’.
In this comparison we’ll deal ‘on-premise’ and ‘private cloud’ as one similar entity. Yes, they are different by many standards and measures, but from the top 10 most critical characteristics that companies consider when selecting the most suitable solution, the difference is actually pretty minimal. Especially in terms of benefits that private cloud setup can provide the similar way on-premise solutions do.
By gathering the right information app developers can make the decision whether public cloud is good enough for their needs – or if private cloud can still be better, more cost-efficient and more flexible solution to use. Let’s look at the top 10 things to consider when comparing between these two solutions.
Things to Consider When Choosing Between Public or Private Cloud
Security is the primary concern for many app developers that build safety-critical apps, and for example banks, insurance companies and many others that handle money transfers, personal information (that is critical by any level) and other critical data. Regardless of whether app is meant to be data, user and safety-critical it is naturally always important to rely on services that reliable, provide certifications and take security measures seriously.
When we compare public and private cloud implementations, it’s safe to say that private environment provides the more secure approach. Despite it is always highly recommended to be demanding on security issues, in too many cases the security topics has been seen as a blocking issue for the use of cloud. On-premise solutions naturally provide the most secure solution but they don’t run by themselves; they require constant attention, maintaining of the infrastructure, devices, tools, integrations and many other things.
This gap between on-premise solution and private cloud is actually pretty insignificant nowadays. There variety of ways to build private cloud environment to be highly secure, providing a lots of great security implementations and methods to get access between private cloud and local development environment. For example, certain data and user protection mechanisms, and other techniques and practices can be utilized in service provider’s data centers. In addition, firewalls, intrusion detections, SSL and app security methods, 24×7 security measurements and monitoring, and the third-party certifications for security practices can be also implemented on private cloud.
Data Ownership, Maintenance, and Retention
The ownership and the actual physical location of data, tests, app details, runs and many others things looks to be the same when public and private cloud implementations are compared. However, private cloud instance always means that it is entirely separated from the public setup, servers, databases, and networks. Private cloud setup means that it is an entity of its own with all these features included, but only dedicated and reserved for one user (customer).
Backing up hard disks, making sure no files, data or other (e.g. user) information is left on anywhere has been one the most important reason for so many companies to use on-premise solutions. However, this again doesn’t change much with private cloud if implemented the right way. You still do not give access to anyone outside of your organization, all the data, ownership of it and backing it up is taken care for you. Doing this locally isn’t always an easy and straightforward task and can mean some serious operational expenses. Both cloud options provide these kinds of tasks as-a-service.
Private cloud provides here very compelling offering as all data aspects, the maintenance of databases, logs, and user information is taken care for you – with security measures.
Performance and Availability
Performance – in nearly all of the cases – is based on the infrastructure and how things have been built. As private cloud is meant to be used only by one company, they do not get any traffic from outsiders, the infrastructure hardware is reserved and dedicated for just one company, they tend to provide the best performance in terms of computing power, fastest access to devices, test runs, results and all other assets.
For availability the private cloud naturally means those reserved and dedicated devices only for one customer’s use. Those can be shared across different development teams geographically so that the use is done on different times of the day. When comparing the availability of the device access and efficient use of the cloud service, it is essential to consider if the service and environment are critical, important or medium level importance to you.
The uptime and availability metrics should be always high (always over 99.5%) for any good and reliable cloud service. In case of sudden event, there should be redundant operations that can get users up and running as soon as possible (for example, using mobile devices from various data center locations). The same applies for backup services so that users can access files, results and data when something breaks (in cases of hardware failures etc.). Furthermore, a reliable private cloud service provides 24×7 monitoring for performance, uptime and accessibility for all assets needed by devs and testers.
Support and Self-Service
The old rule of thumb: Great product without great support is like a flower without water. Or some people see it differently: when you don’t need support, everything works like a charm.
The support in the context of private cloud means that those instances are supported for their users, all infrastructure and devices are up and running. Support is naturally much more than DevOps-like infrastructure maintenance. Both of cloud options typically are built around automation, providing variety of different monitoring capabilities and monitoring of the infrastructure, devices and databases is taken care by its provider.
This type of ‘support’ covers also the self-service. It’s absolutely a must to have good documentations, how-tos, tutorials, plenty of tips and tricks and best practices material available for the service users. Especially, when we are talking about mobile test automation that can be tricky and relevantly a new topic for many app developers, it is important to have all these help materials available to make sure the service users are ‘happy campers’: excellent manuals, help pages for self-service and manual options via online live chat systems with operations, social media etc. plus an online channels to provide instant feedback on service. These are typically the highest priority within private cloud setups.
Enhancements and Update Policies
Both cloud models – public and private cloud are SaaS type of services and typically with private cloud setups a ‘service roadmap’ is drafted when service is brought up. This includes all details of how mobile devices are upgraded, what new features are adopted in the infrastructure and service, how users can access the infrastructure and what update policies are adopted.
As this is very different from local on-premise setup where users define their own upgrades locally, with both cloud options devices are either upgraded to consider all users of those or dedicated users. The biggest difference in upgrade policy is for who those devices are reserved for. With public cloud all users will be considered and typically devices should be upgraded to the latest versions of OS, OEM upgrades installed as soon as those become available, but in case of private cloud those are upgraded based on users preferences.
Nevertheless, both public and private cloud setup must have frequent updates for hosted devices, bug fixes or other enhancements for the infrastructure. Cloud models really provide the most efficient option as everything is taken care automatically and users don’t need to do anything. There won’t be any slow-down or down-time for the service itself, no impact on daily activities or need to do any installation or configurations for accessing the latest and greatest.
Integrating the development and testing environments, frameworks and tools is nowadays pretty easy. Agile process backs up this sort of thinking and many of those popular continuous integration and delivery tools are easy to be extended with additional software. For example, the integration APIs with current tools provide an easy start for any integration and can bring totally different purpose tools together.
The integration is the key for efficient use of all tools and software. In this context, it is recommended that open standards and technologies are used as widely as possible to avoid vendor lock-ins, expensive maintenance costs and also time-consuming customization work between different technologies. Ideally, you should look for a cloud-based solutions that provide these capabilities with API, open standards and all required bits and pieces for you to connect with your internal infrastructure. This will make integrations simple, fast and cost-effective.
Good usability, great user or customer experience, are the topic priorities for cloud providers. And as the usability is also one of top criteria for efficient and productive product development, cloud platform must provide compelling features, support for variety of different (open source) frameworks. The open source part is important here as it means both public and private cloud users must have access to all source, all the time. This provides the required flexibility in terms of vendors, frameworks, typically even cross-platform point of view.
The use of the service must be intuitive, easy to start, use and scale up. When it comes to usability, it’s always a good sign when the community provides constant updates through source code commits, webinars, guidance sessions and other online events, blogs, help systems and self-service documentation.
One known factor to build up the confidence and provide a way much better usability experience of the service is when you have a free trial to service to experiment it further. This means transparency and openness – and typically that provides a great foundation for users to get started with a service.
Contractual Flexibility – Subscription vs. License Fee
No company wants to lock themselves in to a deal that doesn’t provide consistently value to them. The greatest benefit of using subscription models – that both public and private cloud provide – is that you can use it when you need it. Ideally, subscriptions should provide diversity in terms of different features and capabilities – and provide a flexible plan for each type of user/need. The very same applies for migrating or switching between the plans. It is very common that while your development and testing is going on your needs for real devices increase as well.
It’s quite common that contractual flexibility is the highest when subscription model is offered. For example, with on-premise setups, not only the costs that are coming in from operational expenses, users of that sort of environment still need infrastructure, devices, server machine, additional software installed and many other things that bring in the hidden costs. With public and private clouds the costs are fixed for a period of time.
With on-premise solutions, you need your own devices and rest of the hardware infrastructure. Typically, this is pretty straightforward to setup but naturally running, maintaining and monitoring the whole system takes a lot of time, needs to be done 24×7 to get efficient environment for developers to use. Also, the scalability isn’t necessarily easy, if you don’t have appropriate products that can handle a large-number of devices in use, communicate with rest of the infrastructure and deliver you results.
Typically, the size of internal development environments/labs with real devices varies from 20 devices to several hundreds of devices. While setting up and even operating some of these we have learned a thing or two on how to build, operate and maintain an enterprise grade test lab. If you seriously consider building an in-house development environment, please check out all possible information about these. Naturally, the easiest option to up-scale the hardware environment is a privately hosted and dedicated devices for your needs.
References and Customer Recommendations
Existing customers and users are a great sign of something. Going through existing user’s success stories, testimonials and case studies can give you the confidence of an existing system in place. Reviewing customer stories and even asking directly from them how they feel about the system is also encouraged. You’ll also learn if they had any other hiccups – not related to the software, service or adoption – but in more general, what were the extra miles in other parts of the development, integration and adoption they had to go through.
To summarize, these are important ‘metrics’ or features to be asked: performance of the system, in terms of responsiveness and uptime, functionality and how to fully exploit all those great features, usability and what does reference customer like about it, and finally, how does the vendor of a service response and support their customers in case of issues, questions or anything that will help you to get value out of these products/services.
By asking the right questions and knowing what sort of answers to look for, cloud-based solutions are easy to evaluate and thorough understanding of what sort of value they can provide for you.