2.5 Answering the question of what difference a policy has made involves a focus on the outcomes of the policy. Outcomes are those measurable achievements which either are themselves the objectives of the policy - or at least contribute to them - and the benefits they generate.
2.6 Questions under this heading might ask:
• What were the policy outcomes, were there any observed changes, and if so by how much of a change big was there from what was already in place, and how much could be said to have been caused by the policy as opposed to other factors?
• Did the policy achieve its stated objectives?
• How did any changes vary across different individuals, stakeholders, sections of society and so on, and how did they compare with what was anticipated?
• Did any outcomes occur which were not originally intended, and if so, what and how significant were they?
2.7 For example, a policy to recruit unemployed individuals onto a new training scheme which provides seminars to improve work skills might have the ultimate objective of reducing the costs of unemployment. It might attempt to do this by increasing the number of participants who receive and take up job offers, and increasing the duration of their employment. It might try and achieve this by improving participants' skills and qualifications, through seminar attendance and learning. Each of these measures - seminar attendance, number of job offers, duration of employment spells, the costs of unemployment, and so on - could be regarded as intended outcomes of the policy, and hence the subjects of the types of questions just described.
2.8 Questions relating to what difference the policy made concern the change in outcomes caused by the policy, or the policy "impact" - hence the term "impact evaluation", described briefly in Box 2.B. Issues around the reliability of impact evaluation results and how they are affected by the design of the policy are covered in Chapter 3, with further technical discussion provided in Chapter 9.
Box 2.B: What difference did the policy make? Impact evaluation Impact evaluation attempts to provide a definite answer to the question of whether an intervention was effective in meeting its objectives. Impact can in principle be defined in terms of any of the outcomes affected by a policy (e.g. the number of job interviews or patients in treatment), but is most often focused on the outcomes which most closely match with the policy's ultimate objectives (e.g. employment rates or health status). The key characteristic of a good impact evaluation is that it recognises that most outcomes are affected by a range of factors, not just the policy. To test the extent to which the policy was responsible for the change, it is necessary to estimate - usually on the basis of (often quite technical) statistical analysis of quantitative data - what would have happened in the absence of the policy. This is known as the counterfactual. Establishing the counterfactual is not easy, since by definition it cannot be observed - it is what would have happened if the policy had not gone ahead. A strong evaluation is one which is successful in isolating the effect of the policy from all other potential influences, thereby producing a good estimate of the counterfactual. Sometimes the original business case for a policy might have made some estimates of this and forecast the difference the policy might make; this could be used in designing an evaluation. An evaluation might also be able to explain how different aspects of the policy contributed to the impact. Whether a good impact evaluation is possible depends on features of the policy itself, the outcomes it is targeting, and how well the evaluation is designed. If a good evaluation is not possible, or the evaluation is poorly designed, the estimated counterfactual will be unreliable, and there will be uncertainty over whether the outcomes would have happened anyway, regardless of the policy. Then it will not be possible to say whether the policy was effective or not, and even if policy outcomes appear to move in desirable ways, any claims of policy effectiveness will be unfounded. |
2.9 Clearly, there is overlap between the types of questions answered by process evaluation and those addressed through impact evaluation. Policy delivery can be described in terms of output quantities such as the numbers and characteristics of individuals that were recruited, how many training seminars were provided and how many individuals were in gainful employment after the training programme completed. But these are also measurable outcomes of the policy (although not necessarily outcomes which directly deliver benefits). This means that process evaluations often need to be designed with the objectives and data needs of impact evaluation in mind and vice versa. Using and planning the two types of evaluation together will, therefore, help to ensure that any such interdependencies are accounted for. The ability to obtain a convincing explanation will depend on the underlying "theory" of the intervention - that is, how the intervention was supposed to work (see section below on "What type of evaluation for the policy?")