10.20 When considering the evaluation findings it is vital not to neglect the broader context. In addition to analysing the findings from different parts of the evaluation for reinforcement or contradiction, it is important to review the broader research evidence, including related evaluation studies and any other relevant literature. Evaluation findings will be strengthened when they are in line with earlier research. In contrast, differing findings can be explored further in order to seek explanations, thereby making valuable extensions to existing knowledge.
10.21 When seeking to understand why there are differences, it is important to look at the context in which evidence is gathered. For example, the findings may be from research undertaken abroad, such as the USA, and differences in context between the two countries need to be taken into account. For example when looking at issues around health and disability the differences in the healthcare infrastructure might be relevant. While research into criminal justice would need to take into account the differences in sentencing policy.
10.22 Another major difference in context might be temporal differences between the previous research and the current evaluation. For example, the economy may be at a different stage of the business cycle or there may have been legislative or societal changes, such as the increase in access to the Internet, which could explain the differences in the findings observed. If two evaluations are separated in time, the context in which they are carried out, for example economic, social, political, legal or technical, will inevitably have changed.
10.23 It may sometimes seem that the results of an evaluation are almost identical to previous work, questioning its value. But the conditions under which the evaluation takes place will always be different. A labour market programme, for example, might have been found to be effective at a time when the economy was expanding; finding that it is still effective when the economy is in recession would be important learning. Box 10.C provides a list of questions to consider when reviewing the broader research evidence
• What was the economic, social, political, legal or technical context within which the research was undertaken? • Was the research undertaken in the UK? If not, are there any relevant differences in context between the UK and the country in which the research was undertaken? • If undertaken in the UK are the geographical areas comparable in nature? For example urbanisation, levels of deprivation, etc. |
• How long ago was the research undertaken? Have there been any relevant changes in context since the study was undertaken? • Were the studies conducted at the same time of year? Could there have been any seasonal or temporal differences? |
10.24 Quantitative estimates of a policy's impact may lend themselves to meta-analysis.5 This can be used either to get more precise estimates of a policy's impact using findings from a number of different evaluations than are available from a single study; or to understand what factors are associated with varying scales of impact. Suppose there is a policy which is expected to have different impacts at different stages of the economic cycle. In principle a statistical model can be built incorporating the impact estimates at different stages of the economic cycle, and a suitable measure of the state of the economy, to test and quantify the relationship.6
10.25 Even where such formal meta-analysis is not possible, either because there are not enough comparable studies, or because the evidence is qualitative rather than quantitative, it is important to look at the degree of consistency between the evaluation and previous evidence (which should not be limited to previous evaluations). There are clear parallels to the previous section on synthesising evidence within an evaluation. Where the new evidence is at odds with previous studies, it may be possible to develop hypotheses about which factors influence the results. And when the new evidence is weak, it is more likely to be given credence if it is broadly consistent with earlier findings. Where it is not, it may well be an anomalous result.
____________________________________________________________________
5 Meta-analysis is the process of combining statistical information from separate studies, using a range of statistical techniques.
6 An example of meta-analysis in the criminal justice field, which explores which programme features are associated with greater effectiveness, is: A rapid evidence assessment of the impact of mentoring on re-offending: a summary, Home Office Online Report 11/07, Jolliffe and Farringdon, 2007, http://homeoffice.gov.uk/. An example from the labour market field is: When welfare-to-work programs seem to work well: explaining why Riverside and Portland shine so brightly; Greenberg et al ;Industrial and Labor Relations Review, vol 59, no.1, pp34-50