5.1 Chapter 5 describes the various stages involved in planning, commissioning and undertaking an evaluation. Considering each of these steps before the evaluation is undertaken will help to:
• identify the information requirements for the evaluation;
• ensure an appropriate evaluation approach is adopted;
• identify key dates and milestones; and
• ensure the quality, transparency and policy relevance of the evaluation findings.
5.2 Evaluation planning is an important part of policy design. However, as policy making and evaluation are often iterative; it may be necessary to review some of the evaluation objectives and questions as the project progresses.
5.3 A summary of the steps to be considered in planning and undertaking an evaluation was presented in Chapter 4 and is represented in Table 5.A. The remainder of this chapter discusses each of the steps in more detail.
Table 5.A: Steps involved in planning an evaluation
Defining the policy objectives and intended outcomes | • What is the programme logic or theory about how inputs lead to outputs, outcomes and impacts, in the particular policy context? |
Defining the audience for the evaluation | • Who will be the main users of the findings and how will they be engaged? |
Identifying the evaluation objectives and research questions | • What do policy makers need to know about what difference the programme made, and/or how it was delivered? • How broad is the scope of the evaluation? |
Selecting the evaluation approach | • Is an impact, process or combined evaluation required? • Is an economic evaluation required? • How extensive is the evaluation likely to be? • What level of robustness is required? • Can proportionate steps be taken to increase the potential for good evaluation? • What adjustments to policy implementation might improve evaluation feasibility and still be consistent with overall policy objectives? |
Identifying the data requirements | • What data are required? • What is already being collected / available? • What additional data need to be collected? • If the evaluation is assessing impact, at what point in time should the impact be measured? • Who will be responsible for data collection and what processes need to be set up? • What data transfer and data security considerations are there? |
Identifying the necessary resources and governance arrangements | • How large scale / high profile is the policy, and what is a proportionate level of resource for the evaluation? • What is the best governance structure to have in place? • What budget is to be used for the evaluation and is this compatible with the evaluation requirements? Has sufficient allowance been built in? • Who will be the project owner, provide analytical support, be on the steering group? • What will the quality assurance processes be? |
Conducting the evaluation | • Will the evaluation be externally commissioned or conducted in-house? • Who will be responsible for specification development, tendering, project management and quality assurance? • When does any primary data collection need to take place? • Is piloting or cognitive testing of research instruments required? • When will the evaluation start and end? |
Using and disseminating the evaluation findings | • What will the findings be used for, and what decisions will they feed into? • How will the findings be shared and disseminated? • How will findings feed back into the ROAMEF cycle? |