Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« Business Value of Testing - The 4R Framework | Main | The Changing face of Testing »

The 5W and 1H methodology of Metrics Reporting

Without knowing what to improve or what to deliver, it is extremely difficult to manage projects and project management in software testing is no exception. Metric driven approach eases the life of the project manager in handling requirements with regards to people, client expectations, quality, productivity, releases, etc.  However, there is a school of thought that software metrics have evolved but the interpretation of the metrics is yet to evolve. There is a shift that is taking place between having and reporting process driven metrics to reporting value-added metrics. Hence the clarity of metrics, and the way one sees the value, has a strong influence on the making of critical decisions.

Take an example of releasing a business critical application. What does it take for a project manager to sign-off on a release? What are the data points that he/she has to validate to certify that the release is successful? While test coverage, defects, etc., are just few of the dozen things to check, the data points list could be exhaustive depending on the impact of the failure of the release. The idea here is that there should never be a scope for interpreting a metric data. If the release has 300 defects, the QA manager should be in a position to defend that it is good-to-go or place the release on hold. How will this number 300 hold the decision for the release? Will the manager be looking for just the # of defects or more information to assure the quality of release?

Metrics are derived from data points and data points are what we see and get in the projects. Every attribute of the project and people working on the task is a data point (requirement, requirement change, defects, effort, time, productivity, cost, activities associated with test case, test data are few examples).

However, the issues in the current metrics scheme are -

  1. Inability to gather data due to lack of tools or process
  2. Measurement of data is too complex
  3. Too many metrics versus too little metrics
  4. Right metrics versus metrics of little or no value
  5. Intelligence to analyze data and metrics leading to wrong interpretation

While the first 3 points are normally managed in most of the top organizations through use of tools and setting up process and methodology, the challenge for a working project manager would be identifying the right metrics and analyzing data.

So how does one arrive at right metric that makes sense for the project and the stakeholders? The issue is that metrics are provided by the process or governance body. The typical project manager takes all that is recommended for the project. Hence in most of the situations, the project manager either has many metrics to manage or ends up leaving the necessary data points out. One way to check the use and need of metrics is to apply the 5W and 1H methodology (What-Why-When-Who-Which-How). In order to understand this approach, an example is illustrated below.

1. Take a metric, say # of defects, that is recommended to be tracked in a project

2. Apply the first W to it - "What"

  • What is the significance of "# of defects"?
  • What does the "# of defects" mean for the project?
  • What does the "# of defects" mean for the client and the release?
  • What data is required to gather that # of defects?
  • What tool is required to gather this metric?
  • What will different stakeholders do with "# of defect"?
  • What is the definition of defects as per the client and project standards?
  • What are the variations of defects classification?
  • What is the bench mark for "# of defects"?

3. Apply the next W - "Why"

  • Why is "# of defects" important for the different stakeholders?
  • Why is "# of defects" related to the business outcome of the project?
  • Why is "# of defects" mapped to the success of the project/release?
  • Why is "# of defects" high or low?

4. Apply the third W - "When"

  • When is "# of defects" measured?
  • When is "# of defects" critical?
  • When can "# of defects" vary beyond limits?
  • When can "# of defects" value put project into risk?

5. Apply the fourth W - "Who"

  • Who all are interested in "# of defects"?

6. Apply the fifth W - "Which"

  • Which are the sources of "# of defects"?
  • Which is the factor that can influence "# of defects"?

7. Now apply the last question - the H "How"

  • How is the "# of defects" to be tracked?
  • How is the "# of defects" measured in the system?
  • How is the "# of defects" going to impact the downstream process?
  • How is the "# of defects" going to influence the stakeholders?
  • How is the "# of defects" computed and defined?

Using the 5W-1H method, one can derive the various questions and arrive at the metric and its interpretation/relevance for the project. The above example has an illustrative list of questions for a metric and same can be expedited for any metric. It is also recommended to map the metric to $ value and thus bring it closer to the business value. Metrics driven project management is relatively superior as long as the right metrics are chosen.

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.