Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« Improving Customer Experience Using Focused QA Strategy | Main | Is Performance Testing really Non-Functional Testing? »

Crowdsourced Testing: Leveraging the power of a crowd to do the testing

Author: Harsh Bajaj, Project Manager

What is crowdsourcing?

The term 'crowdsourcing' was coined in 2006 when Jeff Howe wrote an article for Wired Magazine - "The rise of crowdsourcing." Crowdsourcing is a combination of two words - crowd and outsourcing. The objective is to get the work done from an open-ended crowd in the form of an open call. 

It is the combined effort of a crowd working at different places through an online medium - leading to significant results. The most common example of crowdsourcing is Wikipedia - the most comprehensive information created by crowd. According to Jeff Howe - "The amount of knowledge and talent dispersed among the human race has always outstripped our capacity to harness it. Crowdsourcing corrects that - but in doing so it also unleashes the forces of creative destruction."

What is Crowdsourced Testing?

Testing has always been a painful area for development companies. This is primarily because of a huge gap between planning and reality for the development of a project - as shown in the figure below. 


Planning and post production.png

Actuals (reality):


One possible way to solve the above situation is to increase the resources and address the scheduling issues - but this may result in huge costs. Another way is to outsource the work to other companies. This requires a fixed schedule, lengthy sales process, and sometimes you end up with the same challenges faced by the internal team.              

However, both these solutions have their own pitfalls. In today's competitive world, organizations are aiming to deliver a good quality project with lower costs and delivery on time.

Therefore, the solution to the above problem is - crowdsourced testing. It is an emerged trend in software testing, which allows individuals to participate at reduced costs and with a better testing quality. Testers are paid for their testing skills and capabilities. The key advantage of crowdsourced testing is that it enables organizations to get their applications tested by skilled people who are closely similar to their end users.

Crowdsourced testing on an average delivers 300 percent more work in the same amount of time*. As a result, it is faster and addresses the time parameter. You have skilled testers testing your application across the globe and as rightly said by James Surowiecki - "Large group of people are smarter than an elite few, no matter how brilliant." Therefore, we can count on a high quality product. At the same time, it keeps your cost down, as you do not have to go through a lengthy sales process.

Stages of crowdsourced testing:

Stages of crowdsourced testing is not different from a software testing lifecycle. The only difference is that resources are taken from a crowd.

Preparation stage:

  • Gather the requirements from a client and prepare a software requirement specification (SRS) document
  • Prepare test strategy / test plan document, decide on testing activities, and the approach to be followed

Initiation stage:

  • Requirements for the staff with the right skill sets are posted on the company portal
  • Based on the responses received from a crowd, the hiring process begins
  • Once hiring is done, details such as requirement specification document, application URL, environment details, and other testing details are provided
  • Test cases are designed

Execution stage:

  • Crowd testers are assigned with different modules for test execution
  • Tester executes system test cases and does a bit of exploratory testing
  • A project lead - monitors the testing activity of crowd testers
  • All the testers share the daily status report with the project lead
  • The project lead also acts as a mediator between crowd resources, client, and clarifies all queries from crowd testers

Evaluation stage:

  • The crowd testers raise defects in a defect management tool
  • A project lead validates the defects raised by the crowd testers
  • Crowd testers are paid based on the number of valid defects raised
  • Defect summary report is prepared.

Completion stage:

  • After a defect is fixed the crowd resource does the retesting
  • Testing activities are closed once all the milestones are met
  • Once the test cycle is closed - then all payments are settled

Some crowdsourcing examples:

Awardesigns.com is a Spanish crowdsourcing platform launched in 2011. More than 2000 designers and writers from over 15 countries (Spain and Latin America) help entrepreneurs, startups, and small business with logo design, Web design, and other types of graphic design.

uTest is a community of more than 20,000 testers who charge on - per bug basis. A client specifies to uTest the type of tests needed and the devices to be tested.

Pros and cons of crowd testing:


  1. The Costs are maintained:

  • It is cost-effective - as you are paying only for the valid bugs reported
  • It is cheaper than hiring engineers, designers, and specialists

2.  Shorter time-to-market:

  • It is faster as you have a large number of testers across the globe testing the software simultaneously

3.  Multiple language testing - very easy to get it done:

  • It helps you to test the application based on localization as there are testers across the globe with variations and skills in different languages and locales
  • As there are more hands - the job will be completed faster within the stipulated time


1.  Management issues:

  • Managing testers across the globe and working in different time zones can be challenging

2.  Compromise on security and confidentiality:

  • Your application and the other environment details are exposed to strangers
  • As the  crowd has access to your software and application, any breach of customer data can lead to security damage

3.  Communication issues:

  • Communication issues may pop-up when an urgent response is needed from the people working online.


               In today's competitive world, there is a lot of pressure on an organization to deliver good quality products in a timely manner. Crowdsourced testing approach is more cost-effective and involves skilled testers working from different places to ensure that all the testing areas are covered. It is faster as you have more hands to get the task done, while maintaining lower costs at the same time. This proven method is an also an effective way to get a multilingual testing done - as there are testers working across the globe.

*This information provided by Crowdtesters.com 


can you help me understand what would be the size of crowdtesting market?

Hi Navita
Thanks for the wonderful post. I am a gold tester of uTest, now the strength has increased to more than 2 lakhs. When I joined in 2012 it was less than 85k

Hi Harsh
Thanks for the wonderful post. I am a gold tester of uTest, now the strength has increased to more than 2 lakhs. When I joined in 2012 it was less than 85k.

Hi Navita
Crowdtesting market is huge, you can expect them around to 2-3 millions.

Crowdsourced testing is an emerging trend in software testing which exploits the benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform. I have come across qualitrix which allows its users to select their crowd among the community of testers and tests the applications in a real world environment..

Thanks for the article.
Cheers Jack!

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.

Subscribe to this blog's feed

Follow us on

Infosys on Twitter