Theory-based evaluation
6.5 The examples presented in Box 6.B show that, with different levels of detail, logic models seek to explain how the linkages work between the stages of the logic model as well as simply stating what they are. This is an important feature of an effective logic model, namely that it is a representation of the causal theory underlying the impact and any associated intervention - i.e. the understanding about why something occurs and how an intervention might work. Logic models such as this are therefore an important component of the general class of evaluation approaches called "theory-based evaluation".
Box 6.B: Examples of logic models Logic Model for the Price Marking Order 2004
Logic model for the Sector Skills Councils
Impact pathway for the health costs of air pollution
Logic model for Local Area Agreements and Local Strategic Partnerships
Source: Department for Trade and Industry (2008), UK Commission for Employment and Skills (2009), Department for Communities and Local Government (2008), Externe2 |
6.6 Theory-based evaluation approaches provide an overarching framework for understanding, systematically testing and refining the assumed connections (i.e. the theory) between an intervention and the anticipated impacts.
6.7 The focus of theory-based evaluations is not only on understanding whether a policy has worked, but why, and under what conditions a change has been observed. Theory-based evaluation will therefore generally seek to identify each of the various elements in the underlying logic model, and examine the links between each element. This process is intended:
• to identify clearly the key inputs, and the expected activities, outputs, outcomes and impacts;
• to articulate how inputs are expected to lead to outputs, outcomes and impacts, and the links and processes in place. These are sometimes called "impact pathways";
• to identify the assumptions about how the policy will be delivered, and any additional factors which need to be in place for the policy to succeed;
• to provide a transparent assessment framework for the evaluation to inform the scope, purpose and data requirements of the evaluation; and
• to inform the evaluation objectives and development of the research questions.
6.8 Evaluations of policy issues and interventions within social settings will generally be based on theories of how individuals, groups, organisations and institutions will respond to the intervention given the context in which it is implemented. Three types of theory-based evaluation approach are commonly used in the evaluation of government social policy. Two of these - Theory of Change and Realist (also known as Realistic) Evaluation - are described in Box 6.C below. Box 6.D provides an example of a theory-based evaluation.
Box 6.C: Theory of Change and Realist Evaluation Theory of Change Evaluation Theory of Change evaluation is a systematic and cumulative study of the links between activities, outcomes, and context of a policy intervention. It involves the specification of an explicit theory of "how" and "why" a policy might cause an effect which is used to guide the evaluation. It does this by investigating the causal relationships between context-input-output-outcomes-impact in order to understand the combination of factors that has led to the intended or unintended outcomes and impacts. Theory of Change therefore normally develops and tests, the implementation theory of the policy and allows this to be modified or refined through the evaluation process. A range of research methods, often both quantitative and qualitative, can be used in order to gather data that contribute to this task. The evaluation often leads to a map showing which factors at which levels have combined to produce the observed outcomes, building on the logic model for the policy. |
Realist Evaluation Whilst Theory of Change tests implementation theory, Realist Evaluation seeks to identify those - often psychological - triggers that change human behaviour as a result of an intervention, taking into account the context within which the intervention sits. Realist Evaluation typically asks: "what works, for whom, under what circumstances?" It begins by developing a set of hypotheses (or theories) on those factors or processes that explain why an intervention has had a particular result (called a mechanism), and what effect the context of an intervention has on these mechanisms. A mechanism can be defined as capturing "people's reasoning and their choices. They describe how people react when faced with a policy measure". Source: DfT (2010); Befani B et al (2007)3 |
Box 6.D: Using Theory of Change to evaluate investment in cycling The Department for Transport has commissioned an evaluation of investment in initiatives aimed to increase cycling in 12 areas across England. This employs a Theory of Change evaluation approach, to assess the impacts of investment and also to learn about what works, how, and why in enabling behaviour change. Each local area has developed investment strategies in response to local need and contextual circumstances. The holistic nature of the approach enables it to test the complex causal relationships involved in changing travel behaviour. The evaluation is seeking to triangulate evidence to strengthen conclusions about the impacts which can be attributed to the investment programme. The evaluation draws on a quantitative assessment of behaviour and attitudinal change, objective monitoring of cycling trends, analysis of cycling behaviours in comparator areas, qualitative insight into the motivators and barriers to behaviour change, an understanding of the effectiveness of different types of initiatives in overcoming these barriers, an assessment of the role of wider national and local contextual factors and an analysis of the design and delivery processes to identify the barriers and enablers to successful implementation. Source: Department for Transport (2011)4 |
6.9 The third commonly used theoretical framework for modelling the effects of social policy is the economic model. This model emphasises the role of choices and incentives in driving behaviour of individuals and organisations. Colleagues from the Government Economic Service can provide assistance in developing logic models and evaluation frameworks which incorporate economic principles. The Treasury Green Book can also provide guidance on developing economic evaluations.
6.10 Theory-based evaluations seek to systematically test and refine the underlying logic model. As Box 6.B demonstrated, these logic models are often highly complex, and evaluations based on them will often need to consider large numbers of relationships, and significant quantities of diverse existing evidence and data, including evidence gathered through (existing and new) process evaluations and counterfactual impact evaluations.5
6.11 Theory-based evaluation approaches provide the overarching conceptual framework within which specific evaluation studies can be designed and evidence structured to answer the policy questions which are being posed. They are therefore complementary, rather than an alternative, to primary process and impact evaluation studies, which provide new data and evidence which can then be incorporated into the evaluation framework as appropriate. One practical way in which this can be done with quantitative evidence and data is through simulation models (see below).
_______________________________________________________________________
2 The Impact of Regulation: A pilot study of the incremental costs and benefits of consumer and competition regulations, Department of Trade and Industry, 2006,; SSC Performance Management Handbook, UK Commission for Employment and Skills, 2009; Long Term Evaluation of Local Area Agreements and Local Strategic Partnerships: Developing a 'Theory of Change', Department for Communities and Local Government, 2008; ExternE - Externalities of Energy: A Research Project of the European Commission (http://www.externe.info/ )
3 Guidance for transport impact evaluations: choosing an evaluation approach to achieve better attribution, The Tavistock Institute for the Department for Transport, 2010, http://www.dft.gov.uk/ ; Realistic Evaluation and QCA: Conceptual Parallels and an Empirical Application, Befani, B et al, 2007, Evaluation, Vol. 13 No 2, p. 178
4 Evaluation of the Cycling City and Towns Programme, AECOM, the Centre for Transport and Society, and the Tavistock Institute for the Department for Transport, January 2011, http://www.dft.gov.uk/