Testing Services

There are many similarities between test automation and manual testing projects from a management perspective. However, there are some fundamental differences which make managing test automation projects very challenging and add to a project manager's existing responsibilities. Some of the most important challenges are:

1.    The stakeholders' consensus on the scope for Test Automation

2.    Agreement on the goals for Test Automation

3.    Agreement on the manual effort required in the Test execution phase

4.    The need to constantly update the stakeholders about changing technical risks & their subsequent impact on the final scope

I will explore each challenge listed above in detail, making the assumption that the underlying development methodology of the AUT (Application under Test) is waterfall SDLC, which is non-agile in nature. I'll also explain why and how these challenges are either not applicable or easy to handle in a manual testing scenario.

Stakeholders' consensus on the scope for Test Automation

A business could demand the complete automation of an application undergoing testing, but there could be a test automation expert who is too cautious to commit on any percentage of automation without conducting a thorough technical feasibility study. It's the responsibility of the QA manager to analyze the study and use it to come up with a feasible solution, to finalize the scope for test automation and articulate it to all the involved stakeholders for a consensus.

Whereas in a manual testing project, the QA manager would not face this challenge of defining the test automation scope because the testing scope depends on the scope of release and there aren't any technology specific constraints to deal with.

Agreement on the Goals for Test Automation

All test automation projects are not alike, so test preparation and execution productivity figures might significantly vary from the industry norms/organization baselines. The QA manager needs to ensure that all stakeholders understand the test preparation & execution productivity for the AUT might differ from their expectations, if any.

In a manual testing project, the QA manager does not face this challenge as the test preparation and execution productivity does not have a high dependence on the technology of the AUTs and minor variations are manageable.

Agreement on the Manual effort involved in the Test execution phase

No amount of robustness of the test automation suite can take away the tester's involvement in test execution, completely. The QA organizations need to understand the role played by the machine and a tester during test execution and mutually agree on the manual effort required during the test execution phase.

It is the responsibility of the test manager to clearly distinguish the manual effort (needed for the activities like test data setup, data table updates etc.) and the machine execution time and ensure that the QA organization understands the estimated effort required during the test execution phase and eventually agree to the same.

Once again, this challenge is not applicable in a manual testing project as test execution effort is an integral part of the overall effort estimated.

The Need to Constantly update the Stakeholders about the changing technical risks & their subsequent impact on the final scope

Due to some unforeseen technical incompatibility issues between the application under test and the testing tools, a situation might arise where the estimated percentage of test automation is not met by the project. Instead of waiting till the end of the project, the QA manager can proactively flag such technical risks earlier; keeping the stakeholders informed on the possible decrease in the test automation scope and the need to fall back on manual testing for test cases that could not be automated. The most difficult part of this exercise is to clearly articulate the complications that arise because of technical constraints so that all stakeholders understand and agree with the same.

In contrast, the change in scope for manual testing projects is very straightforward and easy to communicate.

Posted by Jaya Bhagavathi Bhallamudi at 11:38 AM | Permalink | Comments (0) | TrackBacks (0)

User Acceptance Testing is the most critical phase in a  software development life cycle, as it is the last quality gate that checks product quality, compliance and accuracy before its deployment into production. A successful UAT is crucial for deploying systems devoid of errors and business risks. However, implementing UAT in an agile environment leads to reduced and frequent cycles of testing, which in turn mandates UAT testers to develop skills of optimization testing techniques, automation and work in cohesion with the development and QA teams, which come with their own set of challenges.

Realizing the need for more predictability and assurance of quality in the UAT testing phase, we (Ravindra Kambhampati & Srinivas Yeluripaty) put together this article to call out how test teams can operate successfully in a center of excellence framework, coupled with a few testing accelerators and methods, to really drive greater quality in the UAT stage. We discuss all this in detail in our latest POV titled 'UAT (User Acceptance Testing) In an Agile Mode of Development' at http://www.infosys.com/IT-services/independent-validation-testing-services/Pages/user-acceptance-testing.aspx.

Posted by Ravindra Kambhampati at 6:22 AM | Permalink | Comments (1) | TrackBacks (0)

The unavailability of environments for QA purposes is a very common challenge faced by most organizations. This is because a lot of delay is associated with the acquisition, installation, setup of the QA infrastructure and in gaining access to external & dependent systems. In my recent article(http://www.infosys.com/IT-services/independent-validation-testing-services/Pages/virtualized-systems.aspx), I give detailed review and help businesses understand how they can easily overcome these challenges with Service Virtualization along with cloud adoption. Let me know if this paper helped you and do share your feedback.

Posted by Vijayanathan Naganathan at 5:44 PM | Permalink | Comments (0) | TrackBacks (0)

Today's tough economic environment has put a lot of pressure on organizations to deliver business applications faster and at lower costs.  The rapid growth of the cloud coupled with the current economic environment constraints, has led to the growing adoption of SaaS based applications by organizations.SaaS based applications help organization's focus on their core business than on non-core activities like managing hardware, building applications and maintaining them. However, the adoption of SaaS demands comprehensive testing to reap all benefits associated with it. In my earlier paper, I had identified and described the challenges associated with SaaS testing (http://www.infosys.com/IT-services/independent-validation-testing-services/white-papers/Documents/saas-testing.pdf).

Continue reading "Overcoming challenges associated with SaaS Testing" »

Posted by Vijayanathan Naganathan at 7:45 AM | Permalink | Comments (0) | TrackBacks (0)

CXO Guide Series - Topic 4

Test Case Effectiveness - What does it say?

While I was analyzing operational data of testing organizations for FY2011 (my client organizations), one data point caught my attention. On an average 50,000 test cases executed across application portfolios and less than 5,000 defects logged and fixed. To my surprise, the data was very similar across multiple organizations (Plus or Minus 15%).  Test case effectiveness is around 10% which means, your factor of safety is 10. We are writing and executive 90% of test cases which are non productive in a given test cycle. This is in spite of using all advance methods of testing like orthogonal arrays, model based testing, Risk based testing, etc. Couple of thoughts came to my mind

·         Are testing organizations doing something seriously wrong or do they really need this level of safety net?

·         Can Test Case Effectiveness metric provide more insights and provide a better explanation on why we need this degree of safety and in turn the resultant cost of quality?

·          Is the lower Test Case effectiveness one of the reason why the CXO community perceives testing organizations as an expensive cost center?

·         Do CXO Community and Testing Organizations really understand the degree of safety that is necessary to maintain the Quality of the application?

Well, if the answers to these questions were so straight forward, we would not have asked these questions at the end of the year with the help of a testing metric. But it is very important that every testing organization find its own answers which will be different from each other. Considering the cost of quality of every complex application landscape today, Test case effectiveness is the king of all metrics because it impacts both test planning and test execution efforts.

How can I improve test case effectiveness? I have tried to summarize few basic golden rules which are known to everyone but not always practiced

Conclusion:

Majority of cases testers write test cases based on experience and less on use case conditions (test case writing techniques). Hence in order to control the number of test case written and executed, an aggressive goal to increase test case effectiveness is essential. This will also push testers to understand application functionalities better, use optimization techniques and will increase effectiveness of test planning and execution. This can effectively reduce testing cost significantly. A metric which is hardly captured, can be your solution to optimize your test phase and an answer to reduce cost of Quality.

Posted by Vasudeva Naidu at 5:27 AM | Permalink | Comments (2) | TrackBacks (0)

I recently attended the STC conference by QAI that was held in Bangalore on Dec 1st and 2nd. It is one of the largest international testing conferences in India and this year's theme was 'Testing Enterprise 2.0 - Getting Ready for the Quantum Leap". I was particularly looking forward to the insightful sessions by leading testing leaders and practitioners. There were some sessions that were conspicuous with their new perspectives.

The conference began with a thought provoking session on "The New Age QA delivery models - A Quantum leap in the way testing is Delivered and Measured" by Manish Tandon, Global Head - Independent Validation and Testing Solutions, Infosys.

Continue reading "STC conference by QAI " »

Posted by Kiran Marri at 8:52 AM | Permalink | Comments (0) | TrackBacks (0)

In my early blog on testing for cloud security (http://www.infosysblogs.com/testing-services/2011/12/testing_for_cloud_security-_wh.html), I had discussed the security concerns of cloud adoption from an infrastructure standpoint. Now, let us take a look at what would be the focus of cloud security testing from a data perspective. Enterprises are highly concerned about the security of their data in the cloud. They are well aware that any sort of data security breach could lead to non-compliance, resulting in expensive legal law suits that could cause long term damage to the overall credibility of the organization

Continue reading "Testing for cloud security - What is the data focus of QA teams (Part 2/3)" »

Posted by Vijayanathan Naganathan at 2:19 PM | Permalink | Comments (0) | TrackBacks (0)

While I was thinking about what else can I do to improve testing efficiencies and reducing cost, I asked myself a basic question, what are the fundamental elements which constitute a testing function? After giving a minute to myself, I could quickly pen them down. The three key elements of a testing function are Requirements (Based on which IT system is developed or modified), Testers (Build test plans and execute them to verify concurrence to requirements) and Testing Infrastructure (Test servers, mainframes, middleware components, test data, test labs). In the last 10 years, Testing organizations are focusing profoundly on improving every aspect of testing lifecycle to improve efficiencies by reducing overhead activities. This includes improving requirement stability index (Introducing Requirements Management tools, processes) and testing effectiveness (Productivity improvements, Advance test methods, etc). But when it comes to testing infrastructure (TI), either the status quo has not been challenged much or Infrastructure is not aligned with the Quality goals of the organization. Quality is in fact compromised due to lack of insufficient TI. In many cases, TI is squeezed to an extent, that it can create more than 25% of quality issues in production. You might have noticed one or many of the following in most of the organizations

·         Test environments are limited and do not have connectivity to many integration systems

·         DBA is not available to support test databases

·         Test Data is not refreshed on need basis or test data is not available to test

·         24/7 support is not available for test environments

When I started asking these questions to myself as the possible reasons for this situation, I started listing down and few important ones are

·         Lack of necessary and skill complete understanding of the infrastructure components within the testing organization

·         Need extensive support from organizations external to testing

·         Testing Infrastructure is expensive

With this background, in this paper, I have attempted to make a deeper dive to uncover TI challenges, suggested strategies and recommendations on ways to improve effectiveness and efficiency of TI.

Understanding Test Infrastructure needs and current habits:

Testing organizations are suppose to perform at least the following functions related to Testing Infrastructure

·         Testing Infrastructure Blueprint maintenance

·         Utilization Management

·         Downtime Management

·         Monitoring usage of Testing tools & Licenses, Test data sub setting and provisioning

·         Monitoring testing Infrastructure support

In order to better understand how various testing organizations are managing testing Infrastructure, I collected answers to below questions from my colleagues working for 25+ different testing organizations. You can use this questionnaire as a starting point to better understand the state of your organizations testing infrastructure

 Note: The data collection was questionnaire based and no interviews conducted. The above table is used to provide clarity on various TI topics. The accuracy of data is evaluated only by the author

Testing Infrastructure - Key Functions and Recommendations:

1.       Testing Infrastructure Blueprint maintenance:

TI Blueprint is the map of all test environments used by testing organization. Blue print provides two dimensional view of various testing hardware, application interfacing points, middleware components, databases, 3^rd party application connectivity, etc. The advantage of creating and maintaining this blue print is to get a good understanding of how many test environment you have per application and how robust the TI of the entire organization.  It must also be used for understanding failure points, test data flow patterns for data refresh and data masking, TI support needs, capacity planning and isolating TI bottlenecks for testing activities.

Recommendation is to audit and update the TI blue print every 3 months. This activity calls for collaboration between Testing and IT Infrastructure support teams.

2.      Utilization Management

Testing Infrastructure utilization management is a function to maximize TI usage at reduced cost while ensuring improved application Quality. If you ask this question to your test managers as to how effectively you are managing TI utilization for your application, 99% of them will say, it needs significant improvements. I have listed few reasons for this behavior

·         The most important aspect of utilization management is to ensure that we strike the right balance between TI optimization and Quality of the application performance in production. This correlation is missing

·         The above correlation can be achieved by formation of cross functional team including infrastructure support team and testing organization. The objective of this cross functional team is to study test coverage issues, production issues resulting from TI, data provisioning issues, support issues, 3^rd party application issues which results in Quality issues in production environment. This function is rarely performed.

·         ROI from testing tools at an enterprise level is rarely performed. Many tools are bought and remains on shelf with sub optimal utilization

3.      Downtime Management

This is the most simplistic metric that any organization can collect. As majority of testing organizations work 24/7, down time has direct impact on testing schedule and testing productivity. While most of the organizations are trying to reduce unscheduled down time, it is important to reduce planned downtime as well. Down time data provides a good view of Infrastructure robustness and skill of support function

4.      TI Support, Data and Testing Tools

Infrastructure support is the backbone for utilization and downtime management. In majority of the situations, testers are heavily dependent on developers, DBAs for TI support. Based on challenges faced in your testing organization, it is recommended to encourage your testing community to have a better understanding of your application testing infrastructure to perform first level TI issue assessment.

Your ability to increase test coverage is dependent on how good and relevant data you have to test. While there is no silver bullet to solve data provisioning issues, it is important to attempt a solution which can be a combination of standard tools and home grown customized macros to support your application needs. Investments in test data management should be linked with increase in test coverage and improvements in testing productivity.

Testing Tools are almost always underutilized both capacity and feature wise. Lack of ownership, lack of enterprise direction on testing tools, leads to multiple tools, excessive rework, and skill issues minimizing the ROI.

Recommendations:

1.       Establish testing infrastructure governance which takes complete ownership of above parameters and devises improvement programs which objectively measures efficiency improvements release after release.

2.       This governance should attempt to collect the following metrics

 Conclusion

Now, let me ask the question again about Testing Infrastructure; are testing organizations ignorant or innocent? After reading this paper, you can agree that we are both. Due to lack of skill, understanding, its limited ability to influence change in TI, testing organizations continues to be innocent. But it is ignorant about utilization and downtime management. While the quest to improve testing efficiency and effectiveness continues, Testing Infrastructure improvements will contribute significantly in the next 10 years. As we can understand from this paper, the depth and breadth of Testing Infrastructure is so diverse that it provides significant improvement opportunities to reduce cost of quality. Hence it is necessary for testing organizations to identify a single owner who can start strengthening the foundation for TI improvements. This foundation is necessary to take full advantage of multiple technology solutions which has evolved in the last 2 years. This includes cloud based test infrastructure, TDM tools, mainframe hosting on windows platform, application performance monitoring and many more. Lack of governance and foundation will be a mammoth impediment and will challenge your test organizations future competitiveness.

Posted by Vasudeva Naidu at 6:14 AM | Permalink | Comments (3) | TrackBacks (0)

The complexity and criticality of data warehouse testing projects is growing rapidly each day.  Data warehouses need to be validated for functionality, quality, integrity, availability, scalability and security based on the defined business requirements by an organization. Based on my experiences as a testing practitioner, I believe the following best practices in the test planning phase can significantly contribute to successfully validating the data warehouse.

1.    Comprehensively Understand the Data Model

The data architecture and model is the blueprint of any data warehouse and understanding it helps comprehend the bigger picture of a data warehouse. It's also important to understand the methods used for the key relationships between the major and critical data sources. The relationship hierarchies and the depth of data throw light on the complexity of transformation rules. The quality of data and size of data warehouse determines the functional and non-functional testing requirements for validating the data warehouse.

2.    Understand the Business Requirements Clearly

It's important to understand the complete business context of the need and implication of data warehouse testing. Mapping the business drivers to the source systems helps increase the testing quality, effectiveness and coverage. Getting the test data early during the test planning stage itself decreases risk and increases the predictability of testing.  The data transformations mapping document is the heart of any data warehouse testing project. Hence, understanding the business rules or the transformation mapping document and running books, early in the testing life cycle, helps the testing team reduce rework and delays in the test preparation and execution phases.

3.    Plan Early for Data Warehouse Testing Environment

Planning for test environments based on the quality, size, criticality and complexity of the data warehouse helps reduce the testing cycle. Also, the option of shared environments for the testing teams can also be explored to help reduce costs. Planning for test environments from the following perspectives help decrease the possibility of potential defects:

·        Reverse planning from the release date and preparing a high level test execution plan

·        Understanding and documenting requirements, mitigating the risks/constraints in preparing and running the tests from specific geographical locations - such as time zone, availability of environment or systems & access constraints

·        Planning for different types of functional and non-functional test requirements and their test data. Test data plays a vital role in Data Warehouse testing. Planning and preparing for test data early in lifecycle helps avoid the subsequent cascading delays during execution.

Posted by Raja Sundaram Subramanian at 7:46 AM | Permalink | Comments (0) | TrackBacks (0)