Who should collect the data?

7.19  In considering whether it is feasible for existing frontline staff to carry out the data collection task, analysts will want to consider issues such as:

•  whether there is a culture that is open to research in the participating organisations;

•  whether the participating organisations have a particular interest in a certain outcome;

•  how heavily the new requirements would impact on the business as usual of frontline staff;

•  whether frontline staff are well placed to know the information;

•  whether using frontline staff will result in biased data;

•  whether there is any means of verifying the completeness and accuracy of the data; and

•  whether any necessary changes or additions to IT systems are feasible.

7.20  It is also important not to burden staff with a broad ranging request "for completeness" where there is not a clear match between the level of detail in the data being requested and the analyses actually planned. Indeed the researcher should be able to demonstrate how the data requested will enable the policy to be improved. Where in-house data collection is not feasible, or appropriate, potential alternatives include bespoke surveys, perhaps undertaken and quality-assured by internal or external analysts, and embedded research staff. It is worth noting that monitoring data can be distorted by changes in recording practices, for example, as awareness increases during the course of policy implementation, therefore it is important to ensure that data recording practices remain constant.

7.21  Box 7.C illustrates the key questions and considerations that need to be taken into account to design an effective monitoring system and subsequently to facilitate a good quality evaluation.

Box 7.C: Designing an effective monitoring system

7.22  What if existing monitoring data is insufficient to answer the evaluation research questions?

7.23  Before launching new data collection processes it is important to review existing financial, administrative and monitoring data generation processes to identify whether the required evaluation data can be sourced from existing data sets, or an extension of an existing data set collection processes.

7.24   Frequently, however, new data, whether new monitoring data, or other forms of primary data, will need to be collected. This requires advance planning and ideally should be specified when designing a policy to ensure that the systems are in place to provide evaluators with the required data.

7.25  In the absence of regular data collection on the inputs, outputs and outcomes of a policy (which may be particularly important for an impact evaluation), subsequent evaluation may need to:

•  highlight this as a shortcoming and identify the reasons for the data not being available; and

•  take steps, as far as possible, to retrospectively collect and analyse data on the performance of the project.

7.26  However, attempting to retrospectively collect data in this way is not recommended. It is very likely to be more expensive than collecting data at the same time the policy was taking place. In addition, data may no longer be available or may be inaccurate or piecemeal and the opportunity to validate this data may have been lost. Information may not have been collected on drop-outs which may bias the findings. This is particularly relevant where this information is required to contact participants or where it is needed in order to identify them in other datasets.In summary, it can mean that an evaluation is not possible or that its findings are much less reliable than if data had been collected at the same time the policy was being delivered. Planning an evaluation, and its data requirements, early will therefore minimise the need to collect data retrospectively.