5.22 As set out in Part A of the Magenta Book, an evaluation should be proportionate to the scale, risk and profile of the policy, and the extent of the existing evidence base related to the effects of the policy and/or delivery process. Judgements need to be made about the scale and form of evaluation that is required for a particular policy, including whether it should be commissioned externally or conducted (either partly or wholly) in-house. Having a clear idea about the available resources for the evaluation will also influence selection of the most appropriate evaluation approach.
5.23 In some circumstances, it may be useful to undertake a scoping or feasibility study to support this decision making process and assess whether particular evaluation methods are possible. This can foster greater understanding of what can and cannot be evaluated, and therefore what level of investment is required, and can support the development of an appropriate evaluation design. It is also important to consider whether an evaluation requires external evaluators in order to ensure objectivity and transparency. Chapter 4 provides more detail on the factors that should be taken into account when deciding how much resources should be dedicated to an evaluation.
5.24 Evaluations, whether conducted internally or commissioned to an external contractor, will often require significant input to ensure they are designed and delivered successfully. For larger evaluations involving dedicated data collection, this will generally require an appropriate internal project manager with the relevant skills to oversee the evaluation, a senior responsible owner (SRO) or project director, and a steering group to govern the evaluation (Table 5.C).
5.25 The level of input required of different members of the project team will be greatest at key points (in particular, the design, commissioning and reporting stage), but there will be an ongoing resource requirement even if the project is externally commissioned and this should not be underestimated.
Table 5.C: Examples of typical evaluation governance responsibilities
Internal project manager | Senior Responsible Owner/ Project Director | Steering group |
Drafting a project specification | Ensuring appropriate resources are committed to the evaluation | Ensuring delivery of a high quality and policy-relevant evaluation |
Obtaining any necessary data security clearance | Ensuring the information necessary for the evaluation is collected and made available to the evaluators | Providing advice on how to proceed in the event that circumstances change |
Commissioning (if appropriate) | Ensuring the relevant policy makers and analysts are prepared to engage in setting the evaluation questions, contribute to the design of the evaluation methods and interpretation of its results, and take custody of its findings and conclusions | Facilitating the work of external evaluators |
Day-to-day management, including management of risks |
| Providing access to information and contacts |
Ensuring the evaluation stays on track, meets its objectives, is on time and is delivered within budget |
| Quality assuring the research design and suggesting evaluation questions, methods and research tools |
Advising any contractors and reacting to issues that develop |
| Assisting in the analysis and interpretation of the emerging evidence |
Quality assuring or arranging for quality assurance of intermediate and final products (e.g. project design, research instruments, final reports and presentations) |
|
|
Ensuring the findings are fed back to the relevant audience |
|
|