1.    The stakeholders' consensus on the scope for Test Automation

2.    Agreement on the goals for Test Automation

3.    Agreement on the manual effort required in the Test execution phase

4.    The need to constantly update the stakeholders about changing technical risks & their subsequent impact on the final scope

I will explore each challenge listed above in detail, making the assumption that the underlying development methodology of the AUT (Application under Test) is waterfall SDLC, which is non-agile in nature. I'll also explain why and how these challenges are either not applicable or easy to handle in a manual testing scenario.

Stakeholders' consensus on the scope for Test Automation

A business could demand the complete automation of an application undergoing testing, but there could be a test automation expert who is too cautious to commit on any percentage of automation without conducting a thorough technical feasibility study. It's the responsibility of the QA manager to analyze the study and use it to come up with a feasible solution, to finalize the scope for test automation and articulate it to all the involved stakeholders for a consensus.

Whereas in a manual testing project, the QA manager would not face this challenge of defining the test automation scope because the testing scope depends on the scope of release and there aren't any technology specific constraints to deal with.

Agreement on the Goals for Test Automation

All test automation projects are not alike, so test preparation and execution productivity figures might significantly vary from the industry norms/organization baselines. The QA manager needs to ensure that all stakeholders understand the test preparation & execution productivity for the AUT might differ from their expectations, if any.

In a manual testing project, the QA manager does not face this challenge as the test preparation and execution productivity does not have a high dependence on the technology of the AUTs and minor variations are manageable.

Agreement on the Manual effort involved in the Test execution phase

No amount of robustness of the test automation suite can take away the tester's involvement in test execution, completely. The QA organizations need to understand the role played by the machine and a tester during test execution and mutually agree on the manual effort required during the test execution phase.

It is the responsibility of the test manager to clearly distinguish the manual effort (needed for the activities like test data setup, data table updates etc.) and the machine execution time and ensure that the QA organization understands the estimated effort required during the test execution phase and eventually agree to the same.

Once again, this challenge is not applicable in a manual testing project as test execution effort is an integral part of the overall effort estimated.

The Need to Constantly update the Stakeholders about the changing technical risks & their subsequent impact on the final scope

Due to some unforeseen technical incompatibility issues between the application under test and the testing tools, a situation might arise where the estimated percentage of test automation is not met by the project. Instead of waiting till the end of the project, the QA manager can proactively flag such technical risks earlier; keeping the stakeholders informed on the possible decrease in the test automation scope and the need to fall back on manual testing for test cases that could not be automated. The most difficult part of this exercise is to clearly articulate the complications that arise because of technical constraints so that all stakeholders understand and agree with the same.

In contrast, the change in scope for manual testing projects is very straightforward and easy to communicate.

Realizing the need for more predictability and assurance of quality in the UAT testing phase, we (Ravindra Kambhampati & Srinivas Yeluripaty) put together this article to call out how test teams can operate successfully in a center of excellence framework, coupled with a few testing accelerators and methods, to really drive greater quality in the UAT stage. We discuss all this in detail in our latest POV titled 'UAT (User Acceptance Testing) In an Agile Mode of Development' at http://www.infosys.com/IT-services/independent-validation-testing-services/Pages/user-acceptance-testing.aspx.

I can clearly see that most enterprises are keen on cloud adoption, based on my interactions with them. But the first thing that perplexes them is how to go about evaluating and determining the appropriate cloud deployment that fits their business needs.

In an attempt to address these concerns, I talk about the various factors like understanding the QA infrastructure requirements, the existing infrastructure availability, application release calendar and the budget appetite, that need to be gauged for taking this decision in my latest POV.  To know more, please click here http://www.infosys.com/IT-services/independent-validation-testing-services/white-papers/Documents/cloud-based-QA-environment.pdf. As always I look forward to your views and feedback.

Test Case Effectiveness - What does it say?

While I was analyzing operational data of testing organizations for FY2011 (my client organizations), one data point caught my attention. On an average 50,000 test cases executed across application portfolios and less than 5,000 defects logged and fixed. To my surprise, the data was very similar across multiple organizations (Plus or Minus 15%).  Test case effectiveness is around 10% which means, your factor of safety is 10. We are writing and executive 90% of test cases which are non productive in a given test cycle. This is in spite of using all advance methods of testing like orthogonal arrays, model based testing, Risk based testing, etc. Couple of thoughts came to my mind

·         Are testing organizations doing something seriously wrong or do they really need this level of safety net?

·         Can Test Case Effectiveness metric provide more insights and provide a better explanation on why we need this degree of safety and in turn the resultant cost of quality?

·          Is the lower Test Case effectiveness one of the reason why the CXO community perceives testing organizations as an expensive cost center?

·         Do CXO Community and Testing Organizations really understand the degree of safety that is necessary to maintain the Quality of the application?

Well, if the answers to these questions were so straight forward, we would not have asked these questions at the end of the year with the help of a testing metric. But it is very important that every testing organization find its own answers which will be different from each other. Considering the cost of quality of every complex application landscape today, Test case effectiveness is the king of all metrics because it impacts both test planning and test execution efforts.

How can I improve test case effectiveness? I have tried to summarize few basic golden rules which are known to everyone but not always practiced

Conclusion:

Majority of cases testers write test cases based on experience and less on use case conditions (test case writing techniques). Hence in order to control the number of test case written and executed, an aggressive goal to increase test case effectiveness is essential. This will also push testers to understand application functionalities better, use optimization techniques and will increase effectiveness of test planning and execution. This can effectively reduce testing cost significantly. A metric which is hardly captured, can be your solution to optimize your test phase and an answer to reduce cost of Quality.

The conference began with a thought provoking session on "The New Age QA delivery models - A Quantum leap in the way testing is Delivered and Measured" by Manish Tandon, Global Head - Independent Validation and Testing Solutions, Infosys.

·         Test environments are limited and do not have connectivity to many integration systems

·         DBA is not available to support test databases

·         Test Data is not refreshed on need basis or test data is not available to test

·         24/7 support is not available for test environments

When I started asking these questions to myself as the possible reasons for this situation, I started listing down and few important ones are

·         Lack of necessary and skill complete understanding of the infrastructure components within the testing organization

·         Need extensive support from organizations external to testing

·         Testing Infrastructure is expensive

With this background, in this paper, I have attempted to make a deeper dive to uncover TI challenges, suggested strategies and recommendations on ways to improve effectiveness and efficiency of TI.

Understanding Test Infrastructure needs and current habits:

Testing organizations are suppose to perform at least the following functions related to Testing Infrastructure

·         Testing Infrastructure Blueprint maintenance

·         Utilization Management

·         Downtime Management

·         Monitoring usage of Testing tools & Licenses, Test data sub setting and provisioning

·         Monitoring testing Infrastructure support

In order to better understand how various testing organizations are managing testing Infrastructure, I collected answers to below questions from my colleagues working for 25+ different testing organizations. You can use this questionnaire as a starting point to better understand the state of your organizations testing infrastructure

 Note: The data collection was questionnaire based and no interviews conducted. The above table is used to provide clarity on various TI topics. The accuracy of data is evaluated only by the author

Testing Infrastructure - Key Functions and Recommendations:

1.       Testing Infrastructure Blueprint maintenance:

TI Blueprint is the map of all test environments used by testing organization. Blue print provides two dimensional view of various testing hardware, application interfacing points, middleware components, databases, 3^rd party application connectivity, etc. The advantage of creating and maintaining this blue print is to get a good understanding of how many test environment you have per application and how robust the TI of the entire organization.  It must also be used for understanding failure points, test data flow patterns for data refresh and data masking, TI support needs, capacity planning and isolating TI bottlenecks for testing activities.

Recommendation is to audit and update the TI blue print every 3 months. This activity calls for collaboration between Testing and IT Infrastructure support teams.

2.      Utilization Management

Testing Infrastructure utilization management is a function to maximize TI usage at reduced cost while ensuring improved application Quality. If you ask this question to your test managers as to how effectively you are managing TI utilization for your application, 99% of them will say, it needs significant improvements. I have listed few reasons for this behavior

·         The most important aspect of utilization management is to ensure that we strike the right balance between TI optimization and Quality of the application performance in production. This correlation is missing

·         The above correlation can be achieved by formation of cross functional team including infrastructure support team and testing organization. The objective of this cross functional team is to study test coverage issues, production issues resulting from TI, data provisioning issues, support issues, 3^rd party application issues which results in Quality issues in production environment. This function is rarely performed.

·         ROI from testing tools at an enterprise level is rarely performed. Many tools are bought and remains on shelf with sub optimal utilization

3.      Downtime Management

This is the most simplistic metric that any organization can collect. As majority of testing organizations work 24/7, down time has direct impact on testing schedule and testing productivity. While most of the organizations are trying to reduce unscheduled down time, it is important to reduce planned downtime as well. Down time data provides a good view of Infrastructure robustness and skill of support function

4.      TI Support, Data and Testing Tools

Infrastructure support is the backbone for utilization and downtime management. In majority of the situations, testers are heavily dependent on developers, DBAs for TI support. Based on challenges faced in your testing organization, it is recommended to encourage your testing community to have a better understanding of your application testing infrastructure to perform first level TI issue assessment.

Your ability to increase test coverage is dependent on how good and relevant data you have to test. While there is no silver bullet to solve data provisioning issues, it is important to attempt a solution which can be a combination of standard tools and home grown customized macros to support your application needs. Investments in test data management should be linked with increase in test coverage and improvements in testing productivity.

Testing Tools are almost always underutilized both capacity and feature wise. Lack of ownership, lack of enterprise direction on testing tools, leads to multiple tools, excessive rework, and skill issues minimizing the ROI.

Recommendations:

1.       Establish testing infrastructure governance which takes complete ownership of above parameters and devises improvement programs which objectively measures efficiency improvements release after release.

2.       This governance should attempt to collect the following metrics

 Conclusion

Now, let me ask the question again about Testing Infrastructure; are testing organizations ignorant or innocent? After reading this paper, you can agree that we are both. Due to lack of skill, understanding, its limited ability to influence change in TI, testing organizations continues to be innocent. But it is ignorant about utilization and downtime management. While the quest to improve testing efficiency and effectiveness continues, Testing Infrastructure improvements will contribute significantly in the next 10 years. As we can understand from this paper, the depth and breadth of Testing Infrastructure is so diverse that it provides significant improvement opportunities to reduce cost of quality. Hence it is necessary for testing organizations to identify a single owner who can start strengthening the foundation for TI improvements. This foundation is necessary to take full advantage of multiple technology solutions which has evolved in the last 2 years. This includes cloud based test infrastructure, TDM tools, mainframe hosting on windows platform, application performance monitoring and many more. Lack of governance and foundation will be a mammoth impediment and will challenge your test organizations future competitiveness.

1.    Comprehensively Understand the Data Model

The data architecture and model is the blueprint of any data warehouse and understanding it helps comprehend the bigger picture of a data warehouse. It's also important to understand the methods used for the key relationships between the major and critical data sources. The relationship hierarchies and the depth of data throw light on the complexity of transformation rules. The quality of data and size of data warehouse determines the functional and non-functional testing requirements for validating the data warehouse.

2.    Understand the Business Requirements Clearly

It's important to understand the complete business context of the need and implication of data warehouse testing. Mapping the business drivers to the source systems helps increase the testing quality, effectiveness and coverage. Getting the test data early during the test planning stage itself decreases risk and increases the predictability of testing.  The data transformations mapping document is the heart of any data warehouse testing project. Hence, understanding the business rules or the transformation mapping document and running books, early in the testing life cycle, helps the testing team reduce rework and delays in the test preparation and execution phases.

3.    Plan Early for Data Warehouse Testing Environment

Planning for test environments based on the quality, size, criticality and complexity of the data warehouse helps reduce the testing cycle. Also, the option of shared environments for the testing teams can also be explored to help reduce costs. Planning for test environments from the following perspectives help decrease the possibility of potential defects:

·        Reverse planning from the release date and preparing a high level test execution plan

·        Understanding and documenting requirements, mitigating the risks/constraints in preparing and running the tests from specific geographical locations - such as time zone, availability of environment or systems & access constraints

·        Planning for different types of functional and non-functional test requirements and their test data. Test data plays a vital role in Data Warehouse testing. Planning and preparing for test data early in lifecycle helps avoid the subsequent cascading delays during execution.

These global trends have opened up newer challenges for QA practitioners with complex applications, higher end-user expectations, higher ROI, heterogeneous layers, compliance requirements, mergers and acquisitions. The role of QA has matured & now organizations and projects seek multi skilled QA professionals with expertise in SOA automation QA, e-commerce, performance QA, data warehouse performance QA, mobile network performance QA and information security QA etc.

How a multi-dimensional QA model can help address some of these challenges for building tomorrow's financial services enterprise have been spelled out in my latest POV at ...http://www.infosys.com/IT-services/independent-validation-testing-services/Pages/financial-services-testing.aspx. Please share your comments, inputs and feedback. I look forward to them.

Any enterprise subscribing to the cloud cannot completely depend on the cloud service provider's contract for the security of the cloud infrastructure, the QA teams would also be needed to validate the security of the cloud from the infrastructure layer itself. Once the desired computing power is allocated along with the software, QA teams need to scan cloud instances for existing security vulnerabilities, malware and threats. This would help detect security flaws such as unpatched operating systems at the infrastructure layer.  Also, it's important to check if there are adequate security measures in place like user access control, privilege based access and security policies for governing the QA infrastructure itself. Lastly, the encryption of cloud instances need to be validated since there are security threats involved with recovering previously deleted data in case of unencrypted cloud instances.

I just finished putting together a response for a client trying to understand the value of automation during testing of Oracle Applications. Essentially the client was trying to evaluate whether automation is a worthwhile investment and evaluate the returns expected. I figured this topic to be a subject of much discussion, given widespread implementation of Oracle Applications.

Here are a few points one might want to keep in mind while considering functional automation in their strategy to test the implementation of your Oracle Applications

I believe that QA environment is the perfect place for an organization to begin its cloud journey before actually moving live applications on the cloud. By moving the QA environment to the cloud, organizations can see immediate advantages such as increased asset utilization, reduced proliferation, greater agility in servicing requests and faster release cycle times.

To find out more about the detailed benefits associated with Cloud based QA environments...read my latest POV at http://www.infosys.com/IT-services/independent-validation-testing-services/white-papers/Documents/begin-cloud-adoption.pdf. Do share your comments and feedback, I look forward to them.

However, integrating a COTS product in a large IT system landscape opens up a plethora of security breaches and unless validated correctly (from a security perspective), could be a big security threat to an organization's data, resources etc. Today, there is a need for a comprehensive IT security testing strategy that covers aspects like authorization, authentication, accessibility, vulnerability etc. It is very important to capture the right security requirements, validate them with the right scenarios before implementation and have an apt monitoring mechanism for ensuring uncompromised security.

You can also read - A parametric approach for security testing at http://www.infosys.com/IT-services/independent-validation-testing-services/white-papers/Documents/security-testing-internet.pdf