Interviews

Interview data can provide rich information about the attitudes, opinions and experiences of people involved in a policy to provide in-depth information about how it is working in practice. They allow participants to explicitly explain their views, decisions or actions, describing what has shaped them. Interviews with key participants can be structured (a set list of questions is used with all interviewees), semi-structured (a list of questions with flexibility to probe further) or unstructured (no set list of questions). Interviews most commonly take place face to face between an interviewer and one interviewee, but might also take place over the telephone. The key people to interview will vary from policy to policy but may include those implementing a policy (including a range of levels of seniority and job roles), those receiving a policy, and also stakeholders with an interest in the policy. Usually the analysis of interviews is based on examination of the content, but less frequently techniques of conversational analysis can be used to analyse the way that things are said, by looking at speech patterns and/or body language.

Group interviews

Group interviews provide an opportunity to collect information for a group of people on their attitudes, opinions, perceptions and experiences, building and reflecting on each other's ideas and suggesting a variety of viewpoints and proposals. In group interviews data can be shaped through group interaction. Group interviews can be used with the range of people delivering or receiving a policy. They can work very well in tackling abstract or conceptual topics, where on a one-to-one basis a participant might "dry up". In group interviews, the researcher usually acts as a facilitator and works to a core script which sets out key questions or issues to be discussed by the group. Group interviews can work well in combination with one-to- one interviews or other research techniques. For example, at the beginning of a study they can be used to understand people's current practise, behaviour and beliefs, and test understandings of issues that can then be investigated later in one-to-one interviews. At the end of a study, they offer a deliberative forum for examining the implicationsof the study's findings for service delivery or policy development, and/or generating or prioritizing solutions, with a focus on providing practical suggestions to improve the policy or service. Group interviews can be particularly useful with research participants who may find one to one interviews "scary".6

Observation/ participation

Observing or participating in a policy as it is being delivered provides researchers with direct experience of how a policy is working in practice, for example, a researcher may observe court hearings or benefit interviews. Data will tend to be recorded by the researcher either in narrative form or in a pro-forma, at the same time as the intervention they are observing/ experiencing or shortly afterwards. (In practice, most observational research is non-participatory.)

Surveys

Survey or questionnaire data provides structured, often quantitative data on people's attitudes, opinions and experiences. It may be possible to repeat surveys to map changes in these factors during the life of the policy. This means that surveys can provide statistical data to understand the people, organisations and areas affected by a policy at one or a number of points  f time. Depending on how the survey is set up, this can provide data that can be generalised to the whole population of interest. Surveys may be administered in a number of ways including face to face, telephone, internet and postal, each of which has positive and negative implications with regard to issues such as response rates and cost. Questionnaires are most often used to collect quantitative data but can also contain free text questions to collect qualitative data. When designing a survey it is important to consider what sample of participants and what type of analysis will be needed to answer the research questions and this should be built into the evaluation design. (Further information on survey design is provided in supplementary guidance.)

Consultative and deliberative methods

This describes methods that are used for consultative purposes (for example by local government). Boundaries between consultative research and other types of qualitative research are not absolutely clear cut, and some consultative methods involve the application of established research methods to situations where issues are being debated or deliberated. These types of methods will tend to be used when analysts and policy makers want to go beyond exploring people's views and behaviours, to getting them to come up with, or appraise, solutions and strategies. A wide range of public participation methods might be used including meetings, interactive websites, citizens' panels and juries, deliberative polls and participatory appraisal. Consultative research generally involves intensive exercises with relatively small groups, and thus raises questions about value for money and representativeness. However, well-conducted consultative  research will help to highlight and explain areas of difference, as well as agreement, among participants. A careful balance therefore needs to be struck between the need for consultative research to identify an agreed way forward and the danger that it produces an artificial consensus.

Statistical analysis of quantitative data

A number of sources of quantitative data (including administrative and monitoring data, survey data, and numerical case file data) can provide statistical data on a policy's delivery that is very useful to a process evaluation. For example, quantitative data may be used to calculate numbers of participants receiving an intervention, their characteristics and initial information about costs.

Document analysis

Access to and analysis of documents relevant to the policy being evaluated can provide rich data on all aspects of the policy, including direct commentary on it by those involved in its implementation. These might include computer records, case files, referral letters, diaries, pictures etc. These data can be collected and analysed using appropriate content analysis techniques.

Ethnography

Ethnography is a method used by anthropologists which has been adopted by social researchers more generally. It is the detailed description of a culture, group or society, and of social rules, mores and patterns around which that culture, group or society are based. Ethnography is able to elicit the cultural knowledge of a group or society and also involves detailed investigation of patterns of interaction within it, in order to understand the values, processes and structures of that group. Ethnography tries to study social groups and activity in as 'natural' a way as possible. Observation, listening, remembering and detailed note taking are key techniques for researchers using ethnographic or participant-observation methods of inquiry. Amongst other benefits, this type of data can provide robust evidence on how front-line agencies work, identify variations in the social and cultural environment within which policies, projects and programmes are expected to work, and key personnel who might operate as "product champions" for policies, programmes and projects

Qualitative research sampling has a quite different logic from that of quantitative research. The objective is to select the individual cases that will provide the most illuminating and useful data to address the research questions. The intention is not to provide a precise statistical representation of the research population but to reflect aspects of its diversity which are expected to generate insight. The two main approaches are:

•  Purposive sampling: sample cases are chosen deliberately to represent characteristics known or suspected to be of key relevance to the research questions. These selection criteria are set at the first stage of evaluation design, based on existing research, expertise, or hypotheses. The composition and size of the sample is then determined and individual cases selected to fit the required composition.

•  Theoretical sampling: in this case the researcher makes decisions about the type of data to collect and participants to involve next as the study proceeds, on the basis of emergent theory from their analysis of initial data.

Qualitative samples need to be large enough to include key subgroups and to reflect diversity. The emphasis is on mapping and understanding issues, rather than counting or numerical representativeness. In fact, large samples can be a hindrance as data gathered in qualitative research are rich and intensive. Depth lies in the quality of data collection and analysis, not quantity. The appropriate size of a sample will vary and is always a matter for judgment, but it also needs to be reviewed during fieldwork and as fieldwork draws to a close so that gaps in sample coverage can be filled. The same principles apply for group data collection methods, such as group interviews. Finally, the sample frames used in qualitative research are varied, as in quantitative research, and might include existing data sources such as survey samples, administrative records, registers or databases, or sources which are generated specifically for the research.