2.3 The question of how the policy was delivered is concerned with the processes associated with the policy, the activities involved in its implementation and the pathways by which the policy was delivered. These might vary quite considerably according to the nature of the policy in question, so there is no simple, generic characterisation of questions such as those that tend to be applicable in for impact evaluation.
2.4 However, using a practical example, such as the example of a policy of recruiting people onto a new training scheme to raise employment levels that is discussed at paragraph 2.7, questions might, for instance, seek to describe how individuals were recruited onto the scheme, what criteria were used to recruit them, and what the qualifications of training providers were. It might explore to what extent these factors varied across different parts of the country, and whether recruitment processes operated in favour of or to the detriment of particular groups, such as disabled people or those from particular ethnic groups. It could examine whether there were any difficulties or barriers to delivering the intervention as planned, and what steps were taken to increase course attendance. Box 2.A describes some of the approaches and methods which could be used to evaluate policy processes. Chapter 8 in Part B provides a more detailed description of process evaluation.
Box 2.A: How was the policy delivered? Process evaluation Questions relating to how a policy was delivered cover the processes by which the policy was implemented, giving rise to the term "process evaluation". In general, process-related questions are intentionally descriptive, and as a result, process evaluations can employ a wide range of data collection and analysis techniques, covering multiple topics and participants, tailored to the processes specific to the policy in question. Process evaluations will often include the collection of qualitative and quantitative data from different stakeholders, using, for example, group interview, one to one interviews and surveys. These might cover subjective issues (such as perceptions of how well a policy has operated) or objective aspects (perhaps the factual details of how a policy has operated). They might also be used to collect organisational information (for instance, how much time was spent on particular activities), although "administrative" sources (timesheets and personnel data, for instance) might be more reliable, if available. Although essentially descriptive, these types of information can be vital to measuring the inputs of an intervention (which might not be limited to simple financial budgets, but might also include staff and other resources "levered in" from elsewhere) as well as the outcomes (surveys might be used to measure aspects of a scheme's participants' quality of life, for instance). This illustrates the practical link between process and impact evaluations, which often implies a need to consider the two together. |