4.10 Any evaluation needs to be proportionate to the risks, scale and profile of the policy. The feasibility and significance of obtaining robust evaluation findings will also be relevant and there may be certain circumstances where an evaluation is not feasible or appropriate, for example: when the specific policy can be regarded as part of a broader programme and evaluated at a higher level; when a policy is generally unpredictable or is changing; where costs for a full evaluation are prohibitively high; where there is a lack of consensus or clear direction about program goals; or where the evaluation findings won't be used.
4.11 It may also be argued, even for a relatively important intervention, that it is not possible to afford a full evaluation, in line with the recommendations in the Magenta Book. Certainly the guidance on proportionality should be taken seriously - evaluation research should only be carried out to answer questions in which there is genuine interest, and the answers to which are not already known.
4.12 But even after the overall affordability is queried, it is important to consider the opposite question - can one afford not to do a proper evaluation? Skimping on the research can have serious consequences. It is almost certain to be more cost-effective to conduct a robust evaluation, rather than have to repeat an evaluation because it was not adequately resourced. Furthermore, without a solid basis of evidence, there is a real risk of continuing with a programme which has negligible or even negative impact, or of not continuing with a cost-effective programme.
4.13 Judgement therefore needs to be made about the scale and type of evaluation that is required or possible and the trade-offs that this would require, including whether it should be commissioned externally or conducted (either partly or wholly) in-house. Table 4.C presents some of the factors to be considered when determining the level of resourcing required.
4.14 In some circumstances, a scoping or feasibility study may be conducted to support this decision making process. This can provide greater understanding of what can and cannot be evaluated, and therefore what level of investment is required, and can support the development of an appropriate evaluation design.
4.15 If it is still necessary to reduce evaluation budgets, the following additional questions may provide pointers to how this could be done without rendering the evaluation worthless:
• Is it possible to accept increased risk of drawing a false conclusion about the impact/cost-effectiveness of the intervention? Are all stakeholders content to accept the risk?
• Is it necessary to produce results for sub-groups of the targeted population? Or would the overall impact be sufficient? (The risk here is that a programme which works for some people but not all may be judged as ineffective)
• If face to face surveys are planned, could they be replaced with telephone interviews, postal or online surveys, possibly by reducing the amount of data collected?
• How long do outcomes need to be tracked for? Are there proxy or intermediate outcome measures that could be used? What are the risks of shortening the tracking period? (Very often, tracking over a longer period increases the costs.)
Table 4.C: Factors affecting appropriate resourcing of an evaluation
Factor | Explanation |
Innovation and risk | High risk policies are likely to require robust evidence to understand both how they are working in practice and whether they are having the predicted impacts. In those cases where the innovative initiatives might offer "low cost solutions" evaluation resources might be "disproportionately" high but are still needed to demonstrate the scale of the returns on the policy investment. |
Scale, value and profile | Large scale, high-profile, or innovative policies or policies that are expected to have high impact are likely to require thorough, robust evaluation to help build the evidence base on what works, meet accountability requirements, assess returns on investment and demonstrate that public money is well spent |
Pilots | Pilot or demonstration projects, or policies where there is a prospect of repetition or wider roll out, require evaluation to inform future activities. |
Generalisability | If it is likely that the findings will have a much wider relevance than the policy being evaluated, more resource may need to be allocated to ensure that the results can be generalised with confidence. |
Influence | If the evaluation is capable of providing information which can have a large influence on future policy (for example, it can report at a strategic time-point and/or meet a key evidence gap) more resource is likely to be justified |
Variability of impact | The effects of policies with highly uncertain outcomes or with significant behavioural effects are likely to be more difficult to isolate, and there is likely to be a greater case for conducting a more extensive evaluation. |
Evidence base | Where the existing evidence base is poor or under-researched an evaluation is likely to require more resources in order to fill the gaps |