7.29 Surveys can be used to seek different types of information as covered in Table 7.B. However, it is worth noting that although surveys can be used to ask questions about behaviour it may not be the most reliable measure. Respondents may give socially acceptable answers (though good design and experienced interviewers can reduce this), or be genuinely uncertain about the true answer. For this reason, it might be necessary to observe behaviour rather than simply ask about it. Observation methods are discussed in more detail in Chapter 8.
Types of questions | The type of information collected |
Factual questions | Surveys often offer the only practical and affordable way of collecting such information, and in some cases there is no other source or way of measuring the attribute of interest. This can include respondents' assessments of their health status, life satisfaction and so on. |
Knowledge questions | Assess what respondents know about a particular topic and their awareness of the intervention being evaluated. |
Attitudinal questions | Seek to measure respondents' opinions, beliefs, values and feelings which cannot be verified by reference to observation or external data sources. |
Behavioural questions | Measure what people do or intend to do and how that has changed as a consequence of the intervention. |
Preference questions | Respondents' preferences for different possible options and outcomes, including trade-offs between competing policy objectives. These can be used to elicit monetary values for different outcomes, including those not readily possessing market prices (e.g. changes in air quality, health status) for use in cost-benefit analyses. |
7.30 When designing surveys there are four golden rules that are useful to consider:
• Can the respondents understand the question - and do they understand it in the same way that you do?
• Are respondents able to answer the question?
• Are they willing to answer the question?
• Will the question produce a reliable response?
7.31 Most data collection tools, whether qualitative or quantitative and their associated
materials (e.g. show cards) will require developmental effort, possibly involving cognitive testing or pilots, to ensure they collect information effectively. This will be particularly true with complex questionnaires, such as those attempting to elicit preferences for social impacts, and further advice on writing and testing survey questions is provided in the supplementary guidance. It is important to ensure that new research is consistent with relevant existing or previous data monitoring and collection tools to enable comparison. Where possible, it is helpful to use standard formats for survey questions, or interview schedules, to ensure this consistency. This can have benefits not only for the particular evaluation study, but for building a wider evidence base, particularly where evaluations are being undertaken at a local level. There are some particular points to bear in mind when developing questions for quantitative surveys:
• the Office for National Statistics (ONS), in recent years, has been working towards harmonised questions for common variables such as age, gender, ethnic origin;6
• some questions have been extensively validated in previous studies - examples are the GHQ-12 set of questions for measuring mental well-being and the EG-5D questions for measuring health status.7 Using these will enable comparisons with many other studies, and will ensure the results of the evaluation can be correctly interpreted;
• it may be appropriate in many cases to repeat surveys at the same time of year as a previous one, in the same geographical areas, or using the same sampling frame. A good example of this would be evaluating crime reduction programmes. Some crime types are seasonal, for example, bicycle theft increases in the summer, in contrast burglary increases in winter. Therefore to be certain that a new programme was as effective as one previously evaluated, any data collection would need to be timed appropriately;
• in general there are likely to be tensions: between collecting precisely the ideal data for the current evaluation, and consistency with other studies; or between different, non-comparable, previous studies. To arrive at a balanced view, it is important to be clear as to which (if any) are the key studies to which the results are to be compared; and
• where possible, engage with specialist analysts within your own department or ONS when designing data collection tools.
7.32 It is also important to consider possible subsequent uses of the data during the planning stage and a particular consideration is whether the data can be archived, under what conditions and how much preparation it will need to ensure suitable anonymisation. A common practice is to make suitably anonymised data available through the Economic and Social Data Service.8
7.33 In some cases data may need to be kept for future use in such a way that individuals could be identified for future follow-up.9 The informed consent process for any data collection will have to be designed with storage decisions in mind. Contracts with external research contractors will need to stipulate what outputs, including data sets, are to be provided and can include work to anonymise data. Contracts will also have to consider copyright of intellectual property including research tools as well as datasets.
______________________________________________________________
6 See http://www.ons.gov.uk/about-statistics/harmonisation/index.html
7 See http://www.ons.gov.uk/about-statistics/harmonisation/index.html
____________________________________________________________________
8
http://www.esds.ac.uk/
9 There are also Government Statistical Service protocols on data management, documentation and preservation - see http://www.ons.gov.uk/about-statistics/ns-standard/cop/protocols/index.html