Six key considerations while conducting assessments
On April 18th, 2013, at QUEST 2013 QAI Conference & Expo @ Chicago, Allyson Rippon, Executive Global IT Manager, UBS and I presented our views on establishing a standard for testing activities across the organization. Based on our presentation at the conference, this post will contain some important insights on how to build the brand image of testing across the organization.
Success of the process improvement initiatives depends on the assessment of current state as it provides inputs to decision making process. However, most of the organizations face complications due to the heterogeneous nature of the Quality Assurance (QA) teams considering that
- These teams are distributed in several locations across the globe but operate in a federal structure
- The staffing models they follow could be one or combination of in-house, co-sourced, single vendor or Multi vendor outsourced
- Different QA processes such as waterfall, agile or iterative etc. are adopted to the organisations' delivery methodology
- Projects execution is in multiple technologies (Legacy, package implementations, emerging technologies, integration projects etc)
- We may see independent QA teams conducting testing or developers executing testing or in some cases business analysts/business users executing testing - Varied QA team autonomy
- The different taxonomy practised by teams resulting in poor communication.
- Difficulty in identifying common set of processes or deliverables that satisfy the objectives of the process improvement initiative because teams are at different levels of maturity.
Six considerations to reduce the impact of the diverse nature of the teams
- Assessment by an independent team - Usually bringing an independent vendor to conduct the assessment will help in fair assessment process of the test process improvement initiative. In case this is not feasible, members from Project Management Office (PMO) or business analyst areas having testing background can be considered for this activity.
- The model used for assessment should be capable of providing numerical output based on an acceptable statistical model and should be agreed by the stakeholders before the start of the assessment.
- Coverage is a critical element for ensuring validity - The greater the number of data collected; the higher is the confidence of the output from the model. For small organizations, all the QA team members can be easily assessed. However for medium to large organisations a balance has to be struck. Data from more people is desirable and can result in near accurate results, but we always face time and budget pressures. Moreover, when teams of different sizes are assessed, full coverage of small teams and scant coverage of large teams can skew the assessment results. From my experience, I recommend 10-15% of the group size is optimum coverage for an assessment.
- Collection of evidence is another critical element to be considered during assessments. Any testing process to be assessed will have a deliverable that is typically maintained in a tool, which may not be same across all the teams. It is recommended that teams declare the tools they are going to analyse for collecting evidence and seek agreement with the stakeholders prior to the start of the assessment.
- Typical assessments are a black box to the participants or management until the teams produce the recommendations. However, in process improvement scenarios, it is best to communicate constantly during the assessment process. For example, the participants should be informed of the progress at each step of the assessment process. Also, they should be involved during the recommendation elaboration stage as they are closer to the process and are the best people to provide improvement suggestions.
- One of my best practice recommendations would be to maintain complete traceability right from the process improvement objectives (requirements) to the evidence collected, gaps identified and the recommendations. This traceability forces the assessor to ensure complete coverage and demonstrates rigour in the entire process.