Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« December 2015 | Main | April 2016 »

January 14, 2016

Crowdsourced Testing: Leveraging the power of a crowd to do the testing

Author: Harsh Bajaj, Project Manager

What is crowdsourcing?

The term 'crowdsourcing' was coined in 2006 when Jeff Howe wrote an article for Wired Magazine - "The rise of crowdsourcing." Crowdsourcing is a combination of two words - crowd and outsourcing. The objective is to get the work done from an open-ended crowd in the form of an open call. 

It is the combined effort of a crowd working at different places through an online medium - leading to significant results. The most common example of crowdsourcing is Wikipedia - the most comprehensive information created by crowd. According to Jeff Howe - "The amount of knowledge and talent dispersed among the human race has always outstripped our capacity to harness it. Crowdsourcing corrects that - but in doing so it also unleashes the forces of creative destruction."

What is Crowdsourced Testing?

Testing has always been a painful area for development companies. This is primarily because of a huge gap between planning and reality for the development of a project - as shown in the figure below. 


Planning and post production.png

Actuals (reality):


One possible way to solve the above situation is to increase the resources and address the scheduling issues - but this may result in huge costs. Another way is to outsource the work to other companies. This requires a fixed schedule, lengthy sales process, and sometimes you end up with the same challenges faced by the internal team.              

However, both these solutions have their own pitfalls. In today's competitive world, organizations are aiming to deliver a good quality project with lower costs and delivery on time.

Therefore, the solution to the above problem is - crowdsourced testing. It is an emerged trend in software testing, which allows individuals to participate at reduced costs and with a better testing quality. Testers are paid for their testing skills and capabilities. The key advantage of crowdsourced testing is that it enables organizations to get their applications tested by skilled people who are closely similar to their end users.

Crowdsourced testing on an average delivers 300 percent more work in the same amount of time*. As a result, it is faster and addresses the time parameter. You have skilled testers testing your application across the globe and as rightly said by James Surowiecki - "Large group of people are smarter than an elite few, no matter how brilliant." Therefore, we can count on a high quality product. At the same time, it keeps your cost down, as you do not have to go through a lengthy sales process.

Stages of crowdsourced testing:

Stages of crowdsourced testing is not different from a software testing lifecycle. The only difference is that resources are taken from a crowd.

Preparation stage:

  • Gather the requirements from a client and prepare a software requirement specification (SRS) document
  • Prepare test strategy / test plan document, decide on testing activities, and the approach to be followed

Initiation stage:

  • Requirements for the staff with the right skill sets are posted on the company portal
  • Based on the responses received from a crowd, the hiring process begins
  • Once hiring is done, details such as requirement specification document, application URL, environment details, and other testing details are provided
  • Test cases are designed

Execution stage:

  • Crowd testers are assigned with different modules for test execution
  • Tester executes system test cases and does a bit of exploratory testing
  • A project lead - monitors the testing activity of crowd testers
  • All the testers share the daily status report with the project lead
  • The project lead also acts as a mediator between crowd resources, client, and clarifies all queries from crowd testers

Evaluation stage:

  • The crowd testers raise defects in a defect management tool
  • A project lead validates the defects raised by the crowd testers
  • Crowd testers are paid based on the number of valid defects raised
  • Defect summary report is prepared.

Completion stage:

  • After a defect is fixed the crowd resource does the retesting
  • Testing activities are closed once all the milestones are met
  • Once the test cycle is closed - then all payments are settled

Some crowdsourcing examples:

Awardesigns.com is a Spanish crowdsourcing platform launched in 2011. More than 2000 designers and writers from over 15 countries (Spain and Latin America) help entrepreneurs, startups, and small business with logo design, Web design, and other types of graphic design.

uTest is a community of more than 20,000 testers who charge on - per bug basis. A client specifies to uTest the type of tests needed and the devices to be tested.

Pros and cons of crowd testing:


  1. The Costs are maintained:

  • It is cost-effective - as you are paying only for the valid bugs reported
  • It is cheaper than hiring engineers, designers, and specialists

2.  Shorter time-to-market:

  • It is faster as you have a large number of testers across the globe testing the software simultaneously

3.  Multiple language testing - very easy to get it done:

  • It helps you to test the application based on localization as there are testers across the globe with variations and skills in different languages and locales
  • As there are more hands - the job will be completed faster within the stipulated time


1.  Management issues:

  • Managing testers across the globe and working in different time zones can be challenging

2.  Compromise on security and confidentiality:

  • Your application and the other environment details are exposed to strangers
  • As the  crowd has access to your software and application, any breach of customer data can lead to security damage

3.  Communication issues:

  • Communication issues may pop-up when an urgent response is needed from the people working online.


               In today's competitive world, there is a lot of pressure on an organization to deliver good quality products in a timely manner. Crowdsourced testing approach is more cost-effective and involves skilled testers working from different places to ensure that all the testing areas are covered. It is faster as you have more hands to get the task done, while maintaining lower costs at the same time. This proven method is an also an effective way to get a multilingual testing done - as there are testers working across the globe.

*This information provided by Crowdtesters.com 

January 4, 2016

Improving Customer Experience Using Focused QA Strategy

Author: Harleen Bedi, Principal Consultant

While I sit here pouring my thoughts, we as QA professionals have come a long way in the last decade and a half. From pure play functional testing, which was mostly done manually, today we endeavor to predict defects rather than detect them. 

But have you ever wondered, what is causing this change? Apart from a host of technical advancements in recent years, the most important factor has been "end customers/ users".

Flashback: 20 years ago, when most transactions could be predicted in a controlled environment, it made sense to test as per those business requirements only. But today, is it possible to define all the requirements in the first place? Is it even possible to constraint your customers with a set of "features"?

Consider what it may cost a company, if a customer complains about not being able to perform specific actions on its website or app and all they receive  is a response like "our app does not provide this feature". In the blink of an eye, you lose your customer to  competition. Needless to mention are all the potential negative sentiments that this customer might share on all avenues of social media, which can go viral and impact your brand reputation.

Ensuring Customer Experience should thus be an indispensable aspect of your QA strategy.

How do we define Customer Experience (CX)?  CX is a Customer's perception (conscious or subconscious) while interacting with your applications/products. Obviously, seeing the definition, we can "define" requirements and validate those related to the "conscious" part of interactions. How do we define and validate those related to the subconscious ones?

Let's look at, some obvious aspects that may impact your customer experience. To name a few - how easy it is to navigate the app, how  it looks and feels, its speed, does it work well on various devices, and  do the intended functions work seamlessly?

From QA strategy, these translate into various types of testing that you may need to perform to assure -

  • Usability
  • Accessibility
  • Performance
  • Multi-channel support
  • Compatibility

Now, the subconscious aspects - do I as a customer trust this brand? What kind of sentiments do I see from other similar customers when I look around? Does it give me enough confidence to get engaged with this brand? Does this brand provide me value for money and something to look forward to?

To address these, as QA experts we shall warrant that we've tested enough for ensuring a secure site or app so that customers are comfortable performing any financial transactions on the site/app. Other than providing a defect-free application, we shall learn from feedback obtained from existing customers and consider updating the test strategy or test scenarios. This will help get good reviews on social media.

QA should also anticipate customer needs and provide recommendations as part of the process. E.g.: design thinking techniques when applied in QA context can help articulate these.

From QA strategy, these translate into various testing techniques you may need to follow, like -

  • Security testing
  • Analytics on existing Customer sentiments to define test scenarios
  • Apply Design Thinking practices for QA

Measure Customer Experience

Just validating various aspects of usability, accessibility, performance etc. in silos will not help define/ measure overall customer experience. These parameters needs to be tested and corroborated in conjunction. Thus, we need to first define the mechanism to measure the experience the application currently delivers and view it in a holistic manner. We should quantify customer experience across various impacting parameters and find ways to benchmark ourselves against the competition.


However, how these parameters define/impact overall customer experience will vary from brand to brand. For each application, QA should understand the parameters impacting customer experience and define a holistic Customer Experience Index score that needs to be assured. QA should help predict and monitor customer experience across releases.  Thus, expanding strategies to ensure customer experience along with other customary testing strategies will ensure better engaged customers and thus, more successful implementation of digital programs.