Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« June 2016 | Main | August 2016 »

July 27, 2016

The three ingredients for a perfect test strategy

Authors: Gayathri V- Group Project Manager; Gaurav Gupta- Senior Project Manager

Last week, we won a testing proposal for a mandatory program that cuts across multiple applications. Although we, the 'pursuit team,' were celebrating the win, a voice in our head incessantly kept saying, "Winning is just one part of the game; but delivering such huge programs on time is always a tall order!"

Over the past years, we have delivered multiple, large testing engagements for various clients. Some of these projects were as large as setting up an entire new bank, while some involved introducing new schemes or regulations. The impact of these programs varied from 'medium' to 'complex,' in terms of applications, rigidity of timelines, impacted lines of businesses (LOBs), and so on. Nevertheless, the challenges were always very similar.

During such projects, there are inevitable environmental challenges that our teams struggle to pull through.  Even after they solved these initial hiccups and data set-up issues, they are constantly bombarded with conflicting stakeholders and changes in requirements. However, in spite of such issues, we always pull through, successfully!

There are common underlying levers for the success of any of these programs. As a first step in the testing process, the potential challenges and the strategies to mitigate them need to be diligently thought-out. To ensure a successful implementation every time, our team possesses ample experience in handling three major aspects of any test strategy:

Test environment

Recent surveys have shown that up to 40%-50% of a tester's time can be consumed by test environment issues, thereby affecting quality and productivity.

In major testing engagements, selecting the testing environment is one of the most crucial decisions, as the entire development and testing process relies on it. Here, we can select:
  1. A brand new environment ---- do we need an environment that requires multiple platforms to co-exist or whether we can work in silos
  2. An existing environment ---- - is it  a shared environment or an independent one

Both these options come with their own set of challenges. For instance, a brand new environment will require a huge investment in terms of infrastructure, time, and budgets; while on the other hand, an existing environment will need an agreement with ongoing projects in order to share the resources efficiently without causing any obstructions.

In some testing engagements delivered for major banking clients, we could not go with a new environment but had to work with the existing one.

These engagements have posed various challenges, wherein we had to share the test environment with multiple programs and had to alter the test data for our needs. We even timeboxed the event, made changes to the test data with prior approval, carried out our testing, reverted the environment to the previous state, and then handed it over to the other program team.

A Strategic approach adopted to mitigate the challenge of a Shared test environment depicted below:

Environment strategy depends on the selected option and has to cover the planning, strategizing, and timeboxing of events as part of the logical day plans. To this effect, we have observed that conducting workshops between all participating platforms in a lifecycle is a very effective mechanism to arrive at a workable strategy.

Test data
In the past, we have faced a few scenarios where we had to provision test data from scratch or had to upgrade the test data in the existing test environments. There have also been challenges where we had to share the test data amongst multiple programs and teams. The selected test data strategy has to consider all such aspects of the program and needs to call out the test data mining, timeboxing, dependencies, and ownership details.

Change requests
The ever-fluidic nature of these programs, coupled with changing business requirements in the form of change requests (CRs), only serve to add more complexity. CRs can break the schedule and hence, implementing them requires a thorough impact analysis that cuts across platforms. Workshops have been quite fruitful in achieving this, wherein we discuss whether the CR and its need to implement in the environment is a 'must have,' or 'good to have' . All impacted stakeholders need to agree upon the exact testing cycle in which the change must go. Additionally, it is crucial to document the exact steps to handle CRs, in the strategy.

 The phrase, "Man proposes, but God disposes," is applicable to the pragmatic world of large engagements too. No matter how much we strategize and plan, unforeseen issues can always materialize during the course of the program. It is here that we require a strong management presence; one that can provide course corrections swiftly and align the program with the defined strategy.

To summarize - employing a program-level strategy coupled with a strong management focus, will do wonders and ensure excellent results in large testing engagements.

July 26, 2016

Performance engineering in Agile landscape and DevOps

Author: Aftab Alam, Senior Project Manager

Over the last couple of years, one of the key shifts in the software development process has been to move away from the traditional waterfall approach and instead, embrace newer models like DevOps. One of the main goals of development and operations (DevOps) is to monetize investments as soon as possible. In the traditional waterfall model, UI mockups are all business owners (investors) have access to, before agreeing to invest.

In fact, it can take up to four months to create something tangible that they can see and feel. Sometimes, the end results are not what they might have expected, or in some cases, market scenarios might have changed. Teams thus end up working late-nights and extra hours to cope with the new requirements in order to deliver something that is useful and that adds value to the business. The results are a frustrated team and disappointed investors.

An agile and DevOps software development process is the key to solving this problem.

Key objectives of DevOps are to:
• Reduce the time between feature concept / discovery and actual release in production (going from months to weeks / days)
• Minimize disruption in production (downtime / other issues need to be resolved in minutes)

Key enablers of DevOps are people, organizational culture, extreme automation, right tools and frameworks, flexible application architecture, and knowledge-sharing. While teams change the development process and delivery model from big bang to small, incremental delivery / change, we cannot plan for production as if it is a lab setup and simulate production behavior for every release. Performance engineering must also use DevOps principles, adapt to the change, and be a part of the DevOps model. I would call this 'PerfOps' (Performance engineering in DevOps).

Transition from Waterfall SDLC to DevOps

A crucial change required to fit performance engineering into the agile model involves shifting performance engineering to left(toward development) and right(in production) in the software development life cycle (SDLC). This blog will discuss what PerfOps is, the challenges faced in PerfOps, the means to overcome these challenges, and also key PerfOps activities.

PerfOps starts at the very beginning of the product/ feature discovery phase and continues in production. Here, everyone needs to work towards a common goal to deliver a high performance enabled application.

Where PerfOps lies in devOps

Four key challenges that need to be overcome include:

  1.  People and cultural barriers
    - Enable everyone owns performance in DevOps (Performance awareness)

  2. Short release cycles
    - Explicit performance requirements (It should be a functional requirement throughout the SDLC)
    - Developers need to have the means to validate performance before check- in the source code for build
    - Mine the performance API from unit and integration tests automatically
    - Performance environment setup, design, testing, execution and validation has to be completely automated.

  3. Identifying valuable tools and frameworks
    - It is essential to select application performance management (APM) tools that integrate seamlessly with business reporting tools, IDE, test automation framework, continuous integration/ continuous delivery (CI/ CD) tools, and the operation/ ticket management system
    - Each team member should use one tool throughout the product life cycle
    - Tools and frameworks, along with the monitoring dashboard and threshold, need to be updated as part of the development cycle

  4. Fear of performance defects slipping to production
         - It is possible that some performance defects might slip into production. The PerfOps team should use real production data using the same APM tool, find out optimization opportunities, and decide if changes need to be pulled back
         - Develop a continuous feedback mechanism from production

Performance engineering activities in DevOps 

Requirement validation
   - Make sure performance requirements (NFRs) are a part of the business requirement or user story. Performance requirements should be treated at par with functional requirement. 
             - Example, response time, page size, expected load / user traffic, memory usage per user, number of SQLs / Database calls, and number of service calls
    - Make sure performance monitoring and reporting requirements are captured

Design validation
    - Consider the current production performance and infrastructure usage in the new feature design. For example, if the new feature is going to use an existing service or DB stored procedures, we should take the current performance KPI in design and SLA into consideration
   - Design and implement to avoid duplicate or multiple remote service calls, unnecessary DB calls, inefficient caching strategy, and excessive logging
   - Capture the unit test baseline for metrics like number of downstream calls, number of SQL statements executed, and payload size 
   - Utilize APM tools like Dynatrace that have the means to baseline these performance metrics and capture the deviations

Continuous performance engineering 
   - Automate performance engineering activities and integrate them into CD/ CI build deployment process using tools like Jenkins 
   - Integrate performance testing tools like LoadRunner, JMeter, and CloudTest in Jenkins 
   - Build feature-specific performance test jobs that can be triggered for short durations once the code is deployed in the target environment, in order to validate build's performance 
   - Automate the recording and analysis of performance metrics for each build. This helps in identifying possible performance issues earlier in development, thereby making the overall testing process more agile
   - Automate and maintain APM tool and dashboard setup 
   - Capture the performance metrics from the load testing tools and APM tools like Dynatrace and AppDynamic. Establish a trend for the capture performance metrics between different builds. This would help identify if a particular version of the deployed code has violated thresholds or degraded in performance

Performance engineering in production
   - No matter how hard we try to accurately design our performance testing, it is hard to completely replicate production behavior. The best performance metrics can be captured from the real user data
   - Set up a sophisticated monitoring mechanism to capture performance data that can be used to analyze and provide actionable results to the developers 
   - Instead of going 100 percent live for a new feature, use the concept of A/B testing or multivariate testing to take a staggered approach to go live with a feature. For example, assume there is an existing feature (F) in the production application and a new version of the feature (F') has been developed. Instead of switching all the traffic from F to F' in one shot, use the concept of A/B testing and divert only 1 percent of traffic to the new feature (F') and keep the remaining 99 percent on the existing feature (F). Use the real user data on the new feature (F') to analyze and measure the performance. If the performance of the new feature is satisfactory, then the traffic on F' can be increased to five percent, 10 percent, 25 percent, and so on, until all of the traffic is on F'. If there is a performance issue at any of these points, pull the new feature back, fix the issue, and then start the process again.This whole activity can span across multiple days.
   - Load and scalability testing in production is a very common practice. This gives us the ability to run the load on the actual production environment, thereby eliminating all issues with the environment's size and configurations. Right frameworks and stubs are required to carry out load testing in production

Enabling teams with tools and knowledge

    - Performance engineering best practices should be advocated to the everyone in the teams
    - Performance engineers should participate in code and design reviews 
    - Teams should be given training in  performance engineering tool and demo of performance test/issues analysis

Benefits and key takeaways
    • People: Select team members believe in automation. They will automate everything that they do
    • Tools and framework: Select a PerfOps solution and not just a performance tool
    • Communication: Everyone in the team should know why devOps transition is happening. It's all about bringing efficiency and applying engineering practices in everything we do (not about job insecurity)
    • Key benefits of transition from traditional model to PerfOps are increased software quality, reduced cost, shorter release cycle, performant and scalable application, reduced time to detect issues, and reduced time to recover from any production issues. Of course, a happy team and satisfied investors will lead to more and better work coming our way.

July 18, 2016

Four approaches to big data testing for banks

Author: Surya Prakash G, Delivery Manager

Today's banks are a stark contrast to what they were a few years ago, and tomorrow's banks will operate with newer paradigms as well, due to technological innovations. With each passing day, these financial institutions experience new customer expectations and an increase in interaction through social media and mobility. As a result, banks are changing their IT landscape on priority, which entails implementing big data technologies to process customer data and provide new revenue opportunities. A few examples of such trending technology solutions include fraud and sanctions management, enhanced customer reporting, new payment gateways, customized stocks portfolio-based on searches, and so on.

While the implementation of big data solutions is a complex task, the testing of such implementations offers additional challenges ― mainly due to the volume of test data, infrastructure requirements, newer complex technologies, and different formats of data from multiple sources. Also, banks are moving from traditional warehouse systems to big data technology like Hadoop for cost savings and ease of storing.

As of today, four different kinds of big data implementations are possible, and each require a specific type of testing. However, banks must evaluate the following '3 Vs' of big data before testing applications:

• Volume: A huge amount of data flows through systems, and requires validation for quality
• Velocity: This is the speed at which new data gets generated. Generally, the greater the velocity with which data can be analyzed, the bigger will be the profit for the organization
• Variety: Big data comprises of large data sets which may be structured, semi-structured, or unstructured

The following infographic elaborates on the four different big data implementations and their corresponding testing categories:

The four testing approaches to validate big data:

I. Migration testing: A number of banks are shifting data from traditional data warehouses to big data stores to save on license costs and benefit from big data technology (distributed storage and ease of retrieval). In the case of migration testing, data from the source is compared with the target to ensure that all records are moved. Data quality and visualization testing are also performed on the target database.

II. End-to-end testing: The trends in big data implementations focus on creating new data lakes / data hubs, with big data replacing existing data warehouse systems to store data in different zones for easy retrieval. Such implementations require different testing methods.

In fact, the comprehensive data lake implementations require the following four testing approaches:

  1. Data ingestion testing: Data from various external sources, like social media, web logs (unstructured), and sourcing systems like RDBMS (structured), are validated for transformation, format changes, masking, etc., to ensure that the right data is getting ingested into the data lake. As a result, data will be validated at every stage of data ingestion
  2. Data processing testing: Data quality analysis is the second test step to be followed to ensure data integrity and to validate business rules used for transformation. This needs to be performed in the big data store once the data is moved from the source systems
  3. Data mining testing: Data available in data lakes will be retrieved on the basis of specific business logic to ensure that the right data is filtered and made available for other data stores or relational databases. This takes care of validating the transformation / retrieval query logic
  4. Data visualization testing: Reports and visualization testing is related to end users, where output is validated against actual business requirements and design. Reports are the basis for many decisions, but are also the critical components of the control framework of the organization. As a result, reports, dashboards, and mobile outputs are validated

III. Reports testing: In this kind of testing, reports which get data from data warehouses are modified to get data from big data stores. Two validations are performed: Comparison of data getting displayed in reports versus data available in big data stores. Reports are visually compared or validated against a predefined format or template designed by users or data architects

IV. Data archival testing: This kind of testing is seen in rare and predominantly big data stores used for storing data for audit and compliance purposes. The data is not processed and is stored 'as is' to ensure that it can be retrieved easily. The validation approach involves source-to-target comparison, where data is validated from source databases with target big data stores.


The testing of big data is gaining momentum in banks, which are adopting big data technologies because they are cheaper, handle more volumes, and process much faster. This demands different ways of testing, where testers must have expertise in varied technologies, formats of data (structured and unstructured), data conversions, etc. The four, testing stages discussed above can be used as a starting point to understand the different validation stages for different kinds of implementations.

Automation is the key to success in big data testing, as manual testing is impractical due to data volume and variety. The automation strategy needs to be planned along with test planning to ensure that automated tools are made available for testers during execution.

As more and more financial institutions implement big data solutions, testers must sharpen their skills to test these complex implementations.

July 15, 2016

Big data and performance testing approach

Author: Shreya Banerjee, Project Manager

A big performance: Performance testing of big data
Making it big: Performance testing approach for big data
Seeing the big picture with big data
Thinking big: The right approach to big data performance testing

Big data came into the picture with a big bang. In today's information world, we are constantly bombarded with data. We like our data to be processed, that is, we want information. However, with the overload of data from several sources, we are losing sight of structured data.

According to a new report by the analyst firm International Data Group (IDG), 70 percent of enterprises have either deployed, or are planning to deploy, big data projects and programs in the coming years due to the increase in the amount of data they need to manage. Traditional databases are inadequate and will no more be around to hold such an avalanche of data.

More and more organizations are looking for ways to streamline complexities and get more out of their data-related investments. At the same time, these companies are realizing the usefulness of the power of big data and how it can be utilized for expansion and growth.

Data growth challenges and opportunities are now being defined as three-dimensional -- increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources). Gartner, and now much of the industry, use this 3V model for describing big data.
The emphasis is not only on managing data complexities and integrity, but also on the performance of the system which will make the data useful. Hence, a large chunk of the investment is being put into failover, performance testing of the framework, and data rendition.

In the ideal scenario, even before the performance testing starts, architectural testing is considered crucial to ensure success. Inadequate or poorly designed systems may lead to performance degradation.

The following points need to be considered for a performance-testing strategy:
1. Data ingestion: It is the process of ingesting or absorbing data into the system, either for storage or for immediate use. The focus is not only on the validation of files from various sources, but also on routing them to the correct destination within a given time frame

2. Data processing: Data, when gathered from sources, has to be processed or mapped within a certain framework. This is usually done in batches due to sheer volume. The focus needs to be on the system's scalability and reliability

3. Data persistence: Irrespective of the data storage option (relational database management system, data marts, data warehouse, etc.), the focus is on data structure, which needs to remain constant or easily adaptable for various storage options

4. Reporting and analytics: It is the process of examining large data sets containing a variety of data types to uncover hidden patterns, unknown correlations, market trends, customer preferences, and other useful business information. The focus is on applying correct algorithms and reporting useful information within a certain SLA

Due to data complexity, the approach to performance testing is also different. Here we need to deal with huge volumes of both structured and unstructured data. The following depicts a five-step approach to performance testing:


Performance testing includes testing of job-completion time, CPU and memory utilization, data throughput, and similar system metrics. The two things which need to be kept in mind while defining the approach are:

1. The speed with which a system is able to consume data -- the data insertion rate

2. The speed with which queries are processed while data is being read -- the data retrieval rate

The system is usually made of multiple components; hence, it is wise to first test them in isolation ― at the component level ― before proceeding to test them altogether. Testers have to be well-versed with big data technology and framework, such as Hadoop, NoSQL, messaging queues, etc. Some of the tools which are gaining popularity in the market include:

1. Yahoo! Cloud Serving Benchmark (YCSB): Cloud service testing client that reads, writes, and updates according to specific workloads

2. Sandstorm: Automated performance testing tool that supports big data performance testing

3. Apache Jmeter: Provides plug-ins for testing Cassandra database

Big data performance testing is challenging, but a mix of right tools, skill sets, and a robust strategy, will go a long way in driving a successful project.