Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« Best Practices in Data Warehouse Validation - Test Planning Phase | Main | Testing for cloud security - What is the data focus of QA teams (Part 2/3) »

Testing Infrastructure - Ignorant or Innocence

While I was thinking about what else can I do to improve testing efficiencies and reducing cost, I asked myself a basic question, what are the fundamental elements which constitute a testing function? After giving a minute to myself, I could quickly pen them down. The three key elements of a testing function are Requirements (Based on which IT system is developed or modified), Testers (Build test plans and execute them to verify concurrence to requirements) and Testing Infrastructure (Test servers, mainframes, middleware components, test data, test labs). In the last 10 years, Testing organizations are focusing profoundly on improving every aspect of testing lifecycle to improve efficiencies by reducing overhead activities. This includes improving requirement stability index (Introducing Requirements Management tools, processes) and testing effectiveness (Productivity improvements, Advance test methods, etc). But when it comes to testing infrastructure (TI), either the status quo has not been challenged much or Infrastructure is not aligned with the Quality goals of the organization. Quality is in fact compromised due to lack of insufficient TI. In many cases, TI is squeezed to an extent, that it can create more than 25% of quality issues in production. You might have noticed one or many of the following in most of the organizations

·         Test environments are limited and do not have connectivity to many integration systems

·         DBA is not available to support test databases

·         Test Data is not refreshed on need basis or test data is not available to test

·         24/7 support is not available for test environments

When I started asking these questions to myself as the possible reasons for this situation, I started listing down and few important ones are

·         Lack of necessary and skill complete understanding of the infrastructure components within the testing organization

·         Need extensive support from organizations external to testing

·         Testing Infrastructure is expensive

With this background, in this paper, I have attempted to make a deeper dive to uncover TI challenges, suggested strategies and recommendations on ways to improve effectiveness and efficiency of TI.

Understanding Test Infrastructure needs and current habits:

Testing organizations are suppose to perform at least the following functions related to Testing Infrastructure

·         Testing Infrastructure Blueprint maintenance

·         Utilization Management

·         Downtime Management

·         Monitoring usage of Testing tools & Licenses, Test data sub setting and provisioning

·         Monitoring testing Infrastructure support

In order to better understand how various testing organizations are managing testing Infrastructure, I collected answers to below questions from my colleagues working for 25+ different testing organizations. You can use this questionnaire as a starting point to better understand the state of your organizations testing infrastructure



Yes/NO %



In your testing organization, do you have a nominated role responsible for test environment management? (SPOC for test environment related management)

YES - 75%

NO - 18%

NA - 7%

Even though 75% said yes, most of the respondents particularly highlighted the inactive role played by testing organizations in this capacity


Testers and Test leads have a GOOD understanding of the Testing Infrastructure (Hardware, configurations, connectivity, DB, etc)

YES - 33%

NO - 67%

Majority of the respondents felt that testers have understanding of their scope and have little clarity of Hardware or configurations


All test environments (Server, DB, connectivity) have 24/7 support

YES - 15%

NO - 85%

All the respondents work in offshore model 


Are you collecting environment downtime data for all test environments?

YES - 90%

NO - 10%

More than 50% of the respondents highlighted that no action has been taken on the data collected


If data is being collected, has action been taken to reduce the downtime?

YES - 43%

NO - 57%

This is a Quick win initiative


Does your test environment have end to end testing capability to simulate 80% of the production scenarios?

YES - 18%

NO - 82%

Majority of the respondents do not believe that this scenario results in Quality issues


Does your testing organizations conduct test environment audits to check for deviations after every 3 releases?

YES - 4%

NO - 60%

NA - 36%

Majority of the respondents were not aware of any audits


Do you have test data refresh strategy for all test environments?

YES - 21%

NO - 79%

Several respondents highlighted initiatives currently in progress for implementing data refresh strategy


Do you use virtualization or any other environment optimization techniques currently?

YES - 12%

NO - 33%

NA - 55%

Majority of the respondents were not sure about environment virtualization initiatives undertaken if any


Are their testing infrastructure improvement programs currently in progress?

YES - 30%

NO - 60%

NA - 10%

Even though 30% respondents said yes, less than 6% had clarity on the initiatives being undertaken

 Note: The data collection was questionnaire based and no interviews conducted. The above table is used to provide clarity on various TI topics. The accuracy of data is evaluated only by the author

Testing Infrastructure - Key Functions and Recommendations:

1.       Testing Infrastructure Blueprint maintenance:

TI Blueprint is the map of all test environments used by testing organization. Blue print provides two dimensional view of various testing hardware, application interfacing points, middleware components, databases, 3rd party application connectivity, etc. The advantage of creating and maintaining this blue print is to get a good understanding of how many test environment you have per application and how robust the TI of the entire organization.  It must also be used for understanding failure points, test data flow patterns for data refresh and data masking, TI support needs, capacity planning and isolating TI bottlenecks for testing activities.

Recommendation is to audit and update the TI blue print every 3 months. This activity calls for collaboration between Testing and IT Infrastructure support teams.

2.      Utilization Management

Testing Infrastructure utilization management is a function to maximize TI usage at reduced cost while ensuring improved application Quality. If you ask this question to your test managers as to how effectively you are managing TI utilization for your application, 99% of them will say, it needs significant improvements. I have listed few reasons for this behavior

·         The most important aspect of utilization management is to ensure that we strike the right balance between TI optimization and Quality of the application performance in production. This correlation is missing

·         The above correlation can be achieved by formation of cross functional team including infrastructure support team and testing organization. The objective of this cross functional team is to study test coverage issues, production issues resulting from TI, data provisioning issues, support issues, 3rd party application issues which results in Quality issues in production environment. This function is rarely performed.

·         ROI from testing tools at an enterprise level is rarely performed. Many tools are bought and remains on shelf with sub optimal utilization


3.      Downtime Management

This is the most simplistic metric that any organization can collect. As majority of testing organizations work 24/7, down time has direct impact on testing schedule and testing productivity. While most of the organizations are trying to reduce unscheduled down time, it is important to reduce planned downtime as well. Down time data provides a good view of Infrastructure robustness and skill of support function

4.      TI Support, Data and Testing Tools

Infrastructure support is the backbone for utilization and downtime management. In majority of the situations, testers are heavily dependent on developers, DBAs for TI support. Based on challenges faced in your testing organization, it is recommended to encourage your testing community to have a better understanding of your application testing infrastructure to perform first level TI issue assessment.

Your ability to increase test coverage is dependent on how good and relevant data you have to test. While there is no silver bullet to solve data provisioning issues, it is important to attempt a solution which can be a combination of standard tools and home grown customized macros to support your application needs. Investments in test data management should be linked with increase in test coverage and improvements in testing productivity.

Testing Tools are almost always underutilized both capacity and feature wise. Lack of ownership, lack of enterprise direction on testing tools, leads to multiple tools, excessive rework, and skill issues minimizing the ROI.


1.       Establish testing infrastructure governance which takes complete ownership of above parameters and devises improvement programs which objectively measures efficiency improvements release after release.

2.       This governance should attempt to collect the following metrics



% Savings in Testing Infrastructure cost

Annual improvement goal

% Reduction in environment downtime

Annual improvement goal

% Reduction in TI related production defects

Trend by release

% improvement in ROI from investments on tools

Annual improvement goal



Now, let me ask the question again about Testing Infrastructure; are testing organizations ignorant or innocent? After reading this paper, you can agree that we are both. Due to lack of skill, understanding, its limited ability to influence change in TI, testing organizations continues to be innocent. But it is ignorant about utilization and downtime management. While the quest to improve testing efficiency and effectiveness continues, Testing Infrastructure improvements will contribute significantly in the next 10 years. As we can understand from this paper, the depth and breadth of Testing Infrastructure is so diverse that it provides significant improvement opportunities to reduce cost of quality. Hence it is necessary for testing organizations to identify a single owner who can start strengthening the foundation for TI improvements. This foundation is necessary to take full advantage of multiple technology solutions which has evolved in the last 2 years. This includes cloud based test infrastructure, TDM tools, mainframe hosting on windows platform, application performance monitoring and many more. Lack of governance and foundation will be a mammoth impediment and will challenge your test organizations future competitiveness.




I completely agree that costs related to testing infrastructure are being ignored by testing organizations while they are attempting to reduce testing costs. Some studies suggest that almost 50% of the existing IT infrastructure in organizations is consumed by QA teams and about 40-50% of defects are detected at an applications production stage because testing environments have been configured incorrectly.

I believe that a comprehensive understanding of the functional and non-functional requirements, the different types of testing requirements and hardware configurations for applications is very important. This translates into understanding the accurate capacity requirement for the QA environment. Further, this helps reduce the total cost of ownership from a QA perspective. Also, many organizations have loosely defined SLAs for QA environment availability and there is a lack of clearly defined roles and responsibilities between the QA environment management team and the QA teams. This largely contributes in increasing the testing cycle, leading to increase in costs.

Similarly, a strategic plan to increase the utilization of test tools can lead to the optimization of costs for it. This can happen through identifying apt testing tools across portfolios that cover test management, defect management, etc. Consolidation of tools and setting goals for improving the utilization of their licenses can help achieve better ROI on Testing Infrastructures.

I concur with you on the point, organizations are not spending time and energy in managing the Testing Infrastructure. I guess TI management is required for the system involving multiple components and not required for the independent systems(products). However most of the systems in today’s world are complex systems involving multiple components.

Which test method would you use to test Infrastructure system and why.

How testing is performed(method) for infra structured testing?

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.