5.15 There are a variety of approaches to evaluation, which differ in a number of respects. These include the analytical techniques they adopt, the types of data they use, and the nature of the results they generate. Box 5.B provides a brief description of some of these broad approaches. These categories are not necessarily distinct; however each can comprise a number of different approaches.
| Box 5.B: Types of evaluation Process evaluation Process evaluations can use a variety of qualitative and quantitative techniques to explore how a policy was implemented describing the actual processes employed, often with assessments of the effectiveness from individuals involved or affected by the policy implementation. Further discussion appears in Chapter 8. Empirical impact evaluation Empirical impact evaluations use quantitative data to test whether a policy was associated with any significant changes in outcomes of interest. Various approaches are available which differ in their ability to control for other factors which might also affect those outcomes (the counterfactual, either directly measured or imputed) and hence in the confidence it is possible to place in the results. Empirical impact evaluation is discussed further in Chapter 9. Economic evaluation Economic evaluation involves calculating the economic costs associated with a policy, and translating its estimated impacts into economic terms to provide a cost-benefit analysis. (When only a costing exercise is undertaken, the result is a cost-effectiveness analysis.) Economic evaluations will often make use of existing evidence and assumptions to facilitate the translation of inputs and actual measured outcomes into economic measures, making them akin to theory-based evaluations (see below). The HM Treasury Green Book provides detailed guidance on economic evaluation and cost-benefit analysis. Theory-based evaluation Theory-based evaluation approaches involve understanding, systematically testing and refining the assumed connection (i.e. the theory) between an intervention and the anticipated impacts. These connections can be explored using a wide range of research methods (both qualitative and quantitative), including those used in empirical impact evaluation. More information is provided in Chapter 6. Meta-evaluation and meta-analysis Meta-evaluations (covered in more detail in Chapter 6) can use quantitative or qualitative techniques to bring together a number of related evaluations to derive an overview or summary conclusion from their results. Simulation modelling Simulation modelling is one way in which the results of different evaluations of separate parts of the impact pathway or logic of an intervention can be combined and requires that the evidence relating to the different links in the logic model are expressed in quantitative terms (e.g. effect sizes). Chapter 6 provides more information. |
5.16 The choice of evaluation approach will depend on a number of factors, some of which are considered in Table 5.C. The exact evaluation approach will generally be developed by analytical colleagues, and/or recommended by an evaluation contractor (for externally commissioned evaluations) or other evaluation expert. However, having a clear idea about the required type of evaluation at the planning stage will help inform its design and ensure this meets the evaluation requirements. This will greatly aid decisions about the scope and scale of the evaluation, development of the specification, and the external expertise required.
5.17 There are therefore a wide range of evaluation approaches which will be more or less suitable to the specific evaluation questions and context. Process evaluation is discussed in more detail in Chapter 8 and experimental and quasi-experimental impact evaluation approaches are discussed in Chapter 9. Systematic review, meta-evaluation, theory-based approaches and simulation modelling are discussed in Chapter 6.
| Box 5.C: Issues affecting the choice of evaluation approach Evaluation objectives and research questions The overall objectives of the evaluation and the specific research questions it needs to answer are important factors in deciding which evaluation approach(es) to use and should be developed from the logic model. General research questions which are not overly specific to the intervention in question might be answerable via a qualitative review (or more formal analysis) of the existing literature. Questions which are more specific to the intervention will involve one of the other evaluation types listed in Box 5.B. Questions relating to the wider or ultimate objectives of an intervention will generally require some form of impact evaluation - possibly as part of a theory-based evaluation approach if the associated impact pathways are very extended or complex. Questions relating to detailed aspects of the workings of the policy will generally imply some form of process evaluation (although a combined impact evaluation might be warranted if more definitive answers about effectiveness are required). Complexity of the logic model and importance of confounding factors Where the logic model is particularly complex, restricting the scope of the evaluation to consider shorter, simpler "links" in the logic chain can increase the ability of process evaluations to provide good evaluation evidence. However, if significant confounding factors remain, a robust impact evaluation with suitable controls might be necessary to generate reliable findings. The feasibility of this might depend on data availability (for quasi-experimental approaches) and time and resources (for approaches needing dedicated data collection). Detailed evaluation of changes in very complex systems (especially those with a significant geographical component) might only be possible through theory-based evaluation or simulation modelling. Availability and reliability of existing evidence Large amounts of strong existing evidence increase the relevance of review based methodologies, facilitate greater use of simulation models, and enable evaluations to be simplified to focus more closely on those specific questions which the current evidence base leaves unanswered. Existing data sources and measurability of outcomes If there is already a wide range of good quality data sources covering outcomes of interest, the feasibility of undertaking robust impact evaluations (sometimes to relatively short timescales) is greatly increased. Outcomes which are difficult to measure require either dedicated data collection (e.g. through surveys) or a way of estimating them from changes in intermediate indicators. The former implies a more resource- and time-intensive study, as does a lack of existing data (which might be the case particularly when the focus of the evaluation is the specifics of a very localised intervention). The latter might be addressed through a simulation model, subject to existing data availability. Time and resource availability In most cases, process evaluations (including action research and case studies) will require a formal commission and a dedicated research team, often externally contracted. This can imply a considerable time and resource commitment. Impact evaluations requiring specific data collection and outcome measurement can similarly involve heavy resource commitment and long project durations. Impact evaluations which are able to use existing datasets can provide rigorous results in relatively short timescales but this same reliance on existing data can restrict the questions they can attempt to answer and, in some cases, the ability to confidently attribute the impacts to the intervention. Simulation models can also sometimes be undertaken relatively quickly but this depends on a range of assumptions being made to limit their scope. Empirical impact evaluation issues The two principal strengths of empirical impact evaluation approaches are that they can isolate the effect of an intervention from the possible multitude of factors which might have an influence on the outcome of interest; and in this way, they can provide a rigorous test of whether the intervention has an effect or not. However, these strengths can come at a cost. That is that the approaches are often less able than other approaches to explain exactly why any difference occurred (or not), or how it varied across circumstances.2 Much of this can (and should) be overcome by using a mixed design, whereby process and impact evaluations complement each other, and the process evaluation can help to explain the impact evaluation findings. In other cases based on statistical regression analysis the relationship between the intervention and the outcome of interest might be so complex that the evaluation will only be able to say whether the intervention had an effect, not what aspects of it, how or why. Some "procedural" explanation might be possible, but only if the scope of the evaluation is restricted to simpler relationships, for instance, between the intervention and some intermediate outcome rather than the ultimate objective of the intervention (e.g. the impact of the intervention on the take up of training, rather than the impact on employment and wages). |
_____________________________________________________________________________________________________________
2 Regression-based analysis of data obtained from randomised control trials might be able to provide some explanation of how an observed impact varies across subjects, but is still limited in its explanatory power, and subject to the other weaknesses of the counterfactual impact evaluation approach.