7.12 All data collection, just like the policy allocation itself, needs to be planned before policy activity commences on the ground. This is to ensure that data are obtained about the baseline before the policy (or evaluation) started (this might be used in an impact evaluation as the counterfactual), as well as the situation once the policy is in operation. Consideration of what data are required, when they will be required and how they will be collected should be undertaken at the appraisal and implementation stage of a policy, Box 7.B covers the key areas to consider.
What data need to be gathered to give reliable and consistent measurement against a policy's objectives? What additional data should be collected to meet the policy maker's requirements for feedback on the policy and to support any planned evaluations? Who will have responsibility for gathering data? When will the data be gathered? What are the key timeframes for collection? How will the data be gathered, transferred and stored? What format are the data required in? How will the data be verified to ensure it is accurate and consistent with the relevant requirements? |
7.13 Considering data requirements at the design stage of a policy has a number of benefits:
• policy makers and analysts can identify what regular information they need about the policy, the frequency with which they need it and ensure that this will be available to them throughout the life of the policy;
• data requirements can be designed into the policy so that it delivers this data as a routine process. This means that it can be costed and planned for by the people delivering the policy;
• baselines and counterfactual data can be collected; and
• where external organisations need to provide some of the data, the requirement to do this can be built into their contract (or Service Level Agreement or Memorandum of Understanding) from the outset - it may not be possible to add it later.
7.14 The development of data collection plans should involve both policy makers and analysts, to ensure comprehensive coverage of all requirements and accuracy of research instruments and supporting policy descriptions. Where appropriate, it may also be useful for an external evaluation team, or the people who will deliver a policy, to contribute to the design process prior to the implementation of a policy. Where data for evaluation will be collected via monitoring data, the appropriate monitoring procedures and systems should be set up and embedded from the outset of an intervention, to ensure they systematically generate the appropriate data throughout the duration of the policy.
7.15 Final policy outcomes can take a long time to exhibit and so the collection of monitoring data must take into account the proposed time frames for each intervention. Where it takes too long to capture the final outcomes, or it is simply not possible to directly measure long-term outcomes it may be necessary to collect data on "intermediate" or "proxy" outcomes. Where this is the case these intermediate or proxy outcomes should be identified during the development of the logic model.
7.16 Careful planning for all data collection types is also required to ensure that ethical issues are fully considered, to account for the costs of data collection and to plan how data will be quality assured and transferred and stored.
4Evaluation Guidance Note, Scottish Enterprise, 2008