4.2 Planning and undertaking an evaluation will involve a number of steps and considerations. It can be helpful to develop a structured plan at an early stage, which ensures all aspects have been considered and helps guide the evaluation activity. This will normally be linked to the steps outlined in Table 4.A. Part B of the Magenta Book provides greater detail related to these steps.
Table 4.A: Steps involved in planning an evaluation
Steps involved in evaluation | Questions to consider | |
Defining the policy objectives and intended outcomes | • | What is the programme logic or theory about how inputs lead to outputs, outcomes and impacts, in the particular policy context? |
Considering implications of policy design for evaluation feasibility | • | Can proportionate steps be taken to increase the potential for good evaluation? |
• | What adjustments to policy implementation might improve evaluation feasibility and still be consistent with overall policy objectives? | |
Defining the audience for the evaluation | • | Who will be the main users of the findings and how will they be engaged? |
Identifying the evaluation objectives and research questions | • | What do policy makers need to know about what difference the programme made, and/or how it was delivered? |
• | How broad is the scope of the evaluation? | |
Selecting the evaluation approach | • | Is an impact, process or combined evaluation required? |
• | Is an economic evaluation required? | |
• | How extensive is the evaluation likely to be? | |
• | What level of robustness is required? | |
Identifying the data requirements | • | At what point in time should the impact be measured? |
• | What data are required? | |
• | What is already being collected / available? | |
• | What additional data needs to be collected? | |
• | Who will be responsible for data collection and what processes need to be set up? | |
Identifying the necessary resources and governance arrangements | • | How large scale / high profile is the policy, and what is a proportionate level of resource for the evaluation? |
• | What budget is to be used for the evaluation and is this compatible with the evaluation requirements? Has sufficient allowance been built in? | |
• | Who will be the project owner, provide analytical support, and be on the steering group? | |
• | What will the quality assurance processes be? | |
Conducting the evaluation | • | Will the evaluation be externally commissioned or conducted in-house? |
• | Who will be responsible for specification development, tendering, project management and quality assurance? | |
• | When does any primary data collection need to take place? | |
• | Is a piloting or cognitive testing of research instruments required? | |
• | When will the evaluation start and end? | |
Using and disseminating the evaluation findings | • | What will the findings be used for, and what decisions will they feed into? |
• | How will the findings be shared and disseminated? | |
• | How will findings feed back into the ROAMEF cycle? | |