Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« Manage tests, the automation way | Main | Predictive Analytics Changing QA »

Performance Testing in the Cloud

Author: Navin Shankar Patel, Group Project Manager

If a layperson, frozen in time for 10 years, suddenly wakes up and eavesdrops on a conversation between CIOs, he/she might assume that a group of weather forecasters are conversing. That is because the entire discussion is centered on 'cloud' and is interspersed with a lot of mostly unintelligible words.

Cloud is everywhere these days. There isn't a CIO who is not thinking about the 'cloudification' journey of IT. Its multifold benefits are well understood now and the commercial appeal gets more tempting every passing day. By moving IT assets to the cloud from on-premise, the long-term capex can be converted to on-demand, pay-per-use, opex. The cost of computing and storage is witnessing a downward spiral and has resulted in a proliferation of cloud providers, each claiming unsurpassed value proposition.


While IT enterprises chart out their cloud migration strategy, the decision making involves evaluation of many factors. While the economics of the cloud is easy to comprehend, there are other aspects like security, organization structure, compliance needs, etc. that also need to be evaluated before any decision is made.  Cloud can be leveraged in all the three broad life cycle stages of a software project -- development, testing, and production deployment. And the decision to leverage cloud in some or all of these phases again depends on various considerations. However, when it comes to the unique needs of performance testing, the benefits of cloud are very compelling. 

Performance testing (PT) captures the performance bottlenecks of an application and fixes them before the application is deployed to production. More often than not, PT is executed in the same infrastructure ecosystem as the one used by functional testing teams. The storage and computing capacities of the test infrastructure is almost always a down-sized version of the production environment. This means that the testing is performed in this down-sized environment and results are then extrapolated and inferences are made about how the application may behave in production. Unfortunately, it is not unusual to observe that the production performance does not even closely match the extrapolated predictions. As a result, the response times of web screens will often be lower than what was predicted after PT.

 The obvious way to solve this is to perform PT in production-like-environment. However, this will be prohibitively expensive as production environments typically use horizontal and vertical scaling along with high-end hardware. It is difficult to create a business case to provision such environment for PT alone especially when the PT window is often very small in the overall testing phase and functional testing team has no requirement for such hardware.  At the same time, PT is done intermittently and a dedicated production-like-environment is an overkill and cannot be used for other application testing requirements as the deployment environments are usually different.  Apart from cost, the administrative and operational effort it takes to procure and maintain test environments is also very high and complex. 

Therefore, it is much easier and cheaper to setup production-like-environments in the cloud. Most cloud providers make it easier to quickly provision such ecosystems.  The most critical aspect is to get the configuration right. At first, the application deployment may take some time, but every subsequent deployment can be done in a matter of minutes.  The load testing tools which reside on-premise will need to generate synthetic user load on the application in the cloud. Once the testing is complete, the infrastructure may be released back to the cloud service provider until the next PT cycle.

Cloud can be leveraged in other PT scenarios as well. For instance, the provisioning of on-premise test environments many not be an issue in many cases, but CIOs may not want to invest in buying commercial PT tools as requirement may be unique and intermittent. In such cases, businesses can leverage the software-as-a-service (SaaS) option provided by some PT product vendors. Here, PT is executed from the cloud on applications residing on-premise. At the same time, the cost of PT is limited to factors like duration, protocols, and number of virtual users. This setup is especially useful for situations where PT is to be done for a high number of users for special events like a holiday sale. As on-premise license cost is also based on the number of synthetic users that can be simulated, it entails zero investments in a product as full benefits are realized only sporadically.

These are my thoughts about scenarios that leverage cloud for PT.  If we include application performance monitoring (APM) in the cloud or leverage open source PT tools through the cloud, the opportunities and potential benefits multiply further. More on it later!

Comments

Very informative

Awesome Navin, 2 important points worth considering are Security and Setting up an Eco-system similar to Production like with all the interfaces, 3rd parties...A stand-alone system might not provide similar results.
How secure is the data and application would be a cause of concern to Client

very lucid article on cloud and performance testing

Great blog on Cloud APM

Good start for a blog series on building PT framework on available cloud solution.

Cost factor is not only the factor for creating production like environment, but also features provided by cloud providers like Rackspace, AWS etc for Multi-location support, RUM (with own customized scripts or with commercial tools), Mobile performance testing on real devices (AWS Device farm etc.) etc, can provide huge opportunity for PT as pay-as-go service model and a microservice to be integrated with Continuous Agile product development- with Performance check from Day 1

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.