Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« ETL, ELT, ETLT...How to devise the best test strategy for your Data warehouse applications? | Main | Next-Gen QA - Five Paradigm Shifts that can change the game »

8 Key Test Data Management Challenges

Today's uncertain business environment is witnessing a lot of budget cuts, high competition, job cuts and much more.  In such a scenario, addressing Test Data Management (TDM) with the right approach will not only save huge effort and costs but also bring in business agility, reduced time to market with reliability and coverage of the test cases along with reduction in test environment costs.  As per analysts, the effort spent in doing TDM ranges anywhere between 12% - 14% while in some cases where the applications are data intensive, the effort can go beyond 21%.  That is phenomenal amount of effort and time spent on TDM activities that can be addressed.

Following are the 8 key challenges w.r.t. Test Data Management:

  • Lack of awareness on Test Data Management - Often, the testing team themselves provision the test data required resulting in improper coverage in turn leading to production defects.  It is noticed that almost over 10% of the defects raised in production are due to data that could have easily been captured during the various testing phases
  • Lack of Standardization - As different teams request data in different formats for different types of testing - System testing, Data warehouse testing, Performance testing, UAT, etc., there is no standard data request form and provisioning process followed resulting in longer test cycle times
  • Poor data quality and data integrity - Lack of streamlined process and inconsistent approach to test data refresh process results in poor data quality and integrity issues.  With complex and heterogeneous systems coupled with different file formats having multiple touch points,  inappropriate process followed leads to serious data quality and integrity issues
  • Regulatory and Compliances -   With increased adherence to regulatory compliances such as PCIDS, Gramm-Leach-Bliley Financial Act (GLBA),  BASEL II, Dodd Frank Act, Solvency II etc., test data would need to ensure that sensitivity of the data is maintained along with adherences to compliances which could also be geo-specific
  • High storage cost - High storage, license and maintenance cost as different teams take full production copies.  Due to lack of reuse, redundant data sets and production clones are spread across various test environments increasing the overall CAPEX and OPEX
  • Absence of traceability - No traceability between test data to test cases to business requirements leading to issues on the test data coverage for a particular test case
  • Reduced efficiency  -  With no standard  processes followed, teams working in silos doing manual operations for data engineering, data provisioning, data mocking etc. results in inefficiency as there are no plans for reuse of the test data artifacts and optimization of data, /environment.
  • Huge effort spent on TDM - Significant amount of effort & time is spent in taking huge volume of production data to various environments for different types of testing. Test data identification, extraction and conditioning consume large effort in testing life cycle as much as 12-14% and at times much higher.

With proper TDM processes and frameworks deployed, most of the above challenges can be addressed resulting in good coverage as well as faster time to market.


Good article...
1. For each test case, it is imp to visualise & document input and expected output
2. Complete coverage of business scenario
3. normalisation of test data, tabular form helps

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.

Subscribe to this blog's feed

Follow us on

Infosys on Twitter