8.3 Process evaluation primarily aims to understand the process of how a policy has been implemented and delivered, and identify factors that have helped or hindered its effectiveness. It can take place at any time that the policy is being delivered (the timing of the evaluation will depend on the policy and research questions that need to be answered). Process evaluation can generate a detailed description of what interventions are involved in a service or policy, who provides them, what form they take, how they are delivered and how they are experienced by the participants and those who deliver them. It can also provide an in-depth understanding of the decisions, choices and judgments involved, how and why they are made and what shapes this. It can therefore provide timely information to answer the types of questions in Box 8.A.
How was the policy delivered? In what context was the policy delivered? What did participants and staff feel worked in delivering the policy, why and how? What did they feel worked less well in delivering the policy, and why? What, therefore, might act as facilitators and barriers to desired impacts? How can barriers be overcome and facilitators harnessed? Which particular aspects of the policy seem to have led to an observed outcome (in conjunction with an impact evaluation)? Was the policy implemented "on the ground" in the way it had been planned? (This could include observation of the "take up" of a service or policy, or "compliance" where the policy includes regulation or legislation. It could also include identification of unintended outcomes.) How consistently was the policy implemented across multiple sites or did local variations mean that effectiveness was diluted? Did the policy meet its targets for inputs and outputs? (To establish the need to investigate causes of any difference between expectation and delivery.) Was the logic model (see Chapter 5) linking policy and outcomes supported in the experience of the people delivering or receiving the policy? Did recipients and staff understand the intervention? What was the experience of recipients and staff who received and delivered the intervention? Which aspects were most valued or caused difficulties? Was this different for different groups of people? What was the nature of interactions between staff and recipients during the roll out? Who did not engage, or dropped out, and why? How effective were risk management strategies in anticipating and mitigating risks? Did the policy meet budgetary expectations when rolled out, or were there unforeseen issues and hidden costs? How might the policy be refined or improved? |
8.4 Process evaluation can therefore provide information to assess how a policy is performing, improve the quality of the policy, if needed, and inform future policy development. As noted, this information can be important in explaining the results of an impact evaluation. In particular, without a process evaluation it may not be possible to assess whether a policy that appears not to have had an impact is actually flawed itself or has been affected by poor implementation and delivery. Additionally, where a policy is shown to "work", a process evaluation might indicate which elements of the policy appear to be most influential and therefore how resources might be most efficiently used. It might also supply the data which can be used in an impact evaluation to test the influence of different aspects of the intervention. Where local contexts appear to have influenced the success of a policy this can also help policy makers consider how likely results are to be duplicated in other situations and circumstances.
8.5 There is no single way to conduct a process evaluation. The evaluation can be designed to meet the exact information needs of a particular policy. The methods, timeframes and costs will all therefore depend on what information is required. It is very important to be clear about the aims and objectives of a process evaluation by:
• identifying all the policy questions the research will need to answer and when these will need to be answered;
• drafting clear research questions that reflect these policy priorities;
• identifying what data will be needed to answer these questions: including who will have this information, which groups will be studied (and what sampling techniques will be needed if not all participants will be included), what format it will be collected in (for example. paper or electronic), and when the data will be available;
• deciding at what stage in the policy a process evaluation will provide most value (if not throughout the study); and
• understanding how the resulting evaluation will support an assessment of the policy's performance, refinement of the policy, or an impact evaluation.
8.6 Process evaluations might collect and analyse quantitative or qualitative data to answer the research questions, or a combination of both. It is important that whichever data is used, that it is collected accurately, analysed robustly and presented appropriately. This could mean identifying an appropriate and credible sampling technique to choose research participants, ensuring appropriate statistical techniques are used when quantitative data is analysed, or choosing a range of methods or groups of participants to corroborate findings or deepen understanding (also known as triangulation of data). Triangulation of data, or the use of multiple methods, which explore similar research questions adds credibility to and confidence in the findings of an evaluation and strengthens the conclusions and recommendations that can be made as a result (triangulation is discussed further in Table 8.A below).
8.7 Input and outcome measures can feed into any economic evaluation, so it is important to consider the information requirements for cost-benefit analysis when planning a process evaluation. Often, the process evaluation might be the principal or even the only source of additional data for an economic evaluation. Therefore, if the special data requirements for economic analysis are not considered when designing the process evaluation, a meaningful economic evaluation might be effectively precluded, as it will not be possible to collect the information retrospectively.
8.8 Further information about methods that can be used to support process evaluations (and action research and case studies) and how they can be chosen to best answer the research questions set for an evaluation is discussed below.
Methodological triangulation |
| This refers to combining different research methods. This can include "within research" triangulation (where, for example, a range of different lines of questioning might be used to approach the same issue) and "between method" triangulation (where different data collection methods are combined). This can also include the combining of qualitative and quantitative data. |
Data triangulation |
| This means combining data from more than one source, for example, a number of settings, points in time or groups of people. |
Investigator or analyst triangulation |
| This involves more than one researcher looking at the data so that they can either check or challenge each other's interpretation or deliberatively approach the data from different angles. |
Theory triangulation |
| This means looking at the data from different theoretical positions in order to explore the fit of different theories to the data and to understand how looking at the data from different assumptions affects how it is interpreted. |
_________________________________________________________________________
1 The Research Act: A Theoretical Introduction to Sociological Methods, Denzin, 1989, Englewood Cliffs, NJ, Prentice Hall.