Principles

Explanation

There must be a clear set of research questions that can be addressed through the delivery and implementation evaluation.

Research questions that are broad or vague can easily lead to unsatisfactory studies that simply do not produce new insights or do not have sufficient relevance or reliability to aid future decision making, which means that the evaluation will not offer value for money.

There should be coherence between the research questions and the populations and data studied.

Populations and data that are going to give the most direct and insightful information on the subject matter should be selected, taking into consideration which subsets of these populations are critical for inclusion or exclusion. (More information on sampling is provided in supplementary guidance). For example, researchers and policy makers may want an overall assessment of how a policy has been experienced/ implemented for everyone receiving it, but also to understand if there were different issues/ experiences for different genders, ages or ethnic groups. If there is already robust evidence on the experience of a particular group of service recipients, then an evaluation may want to focus on gathering data on other recipient groups rather than duplicating previous research.

Building comparisons into the design can be helpful and lead to more in-depth understanding.

For example, a study looking at a particular phenomenon among lone parents (such as attitudes to work) might be enhanced by including couple parents. Comparing the responses of the two groups will help with understanding of what is a function of being a lone parent, as opposed to that of simply being a parent.

There should be coherence between the research questions and the settings studied.

For example, sites should be chosen to provide coverage of the populations of interest to the policy makers. This could range from specific locations, organisations, contexts or groups of people all the way to collecting national data.

There should be a logic between the research questions and the data collection methods used.

For instance, are naturally occurring data needed because what is being researched is best illuminated by observing behaviour or interaction? (This might be the case where there is reason to believe that people's self-reported behaviour might not reflect what actually happens in practice.) Or do the research questions require evidence of people's own experiences, opinions and views? In which case data might be best collected through individual interviews or group discussions. Alternatively, if quantitative data (for example statistics on service take- up) is required then this might be most appropriately met by using existing monitoring data or commissioned surveys.

There should be a logic to the timing of the episodes of data collection.

This would include deciding at what stage of delivery and implementation information should be collected, and if data is required at a number of intervals. For example data may be collected to assess levels of attendance on an employment course at the start, middle and end of the course, or assessments of educational achievement may be made of a group of students before and after they receive a new educational intervention.

It is important to consider the feasibility and appropriateness of a proposed methodology within the actual research setting.

For example, it would be important to check that researchers would be allowed to observe particular aspects of service delivery, such as counselling sessions, before adding this technique into the evaluation design.

Key question

Considerations

What type of data will be required to answer each research question?

•  Is numerical data required?

•  Is factual (documentary) data required?

•  Is observational data required?

•  Is data to describe people's experiences, opinions, and views required?

•  Will a combination of these types of data be required?

Who or what can provide this data?

•  Which participants, service providers, stakeholders, databases etc., would have this data and/ or need to be consulted?

•  Do/ will researchers be able to get access to this data?

•  Are there any potential sensitivities/ ethical issues in collecting data from these groups, areas, databases etc.?

What section of the population of interest should data be collected from?

•  Who is the population of interest?

•  Will the research be a census of all available data/ population of interest or will a sample of the population be studied?

•  For qualitative sampling - what range of people, experiences, organisations, contexts etc. need to be covered?

•  For quantitative data, what types of estimate will the data need to provide and how precise? Which sub-populations need 

to be included? What impact does this have on the sample size required?

•  For qualitative and quantitative data - what sampling frames are available or will need to be created?

How will the data be collected?

•  Which research method is best placed to provide the required type of data from the required sources (see below for summaries of key research methods)?

•  Is the data already being collected or will new data collection be required for the research?

•  When should/ can the data be collected?

•  How will data be validated/ triangulated?

•  Who will collect the data?

How will the data be analysed?

•  Does the method of analysis that will be used require a particular sample size or type of data to have been collected?