9.47 One alternative is the method of difference in difference (DiD; or "two group pre- and post-test design"). Once again, the aim is to adjust for those factors that affect both likelihood of exposure to the policy and the outcome from the policy, and hence that might cause selection bias. But this method does so without having to know what all these individual factors are, and as such is far less data hungry. Instead, it works by comparing how trends in associated outcomes change between treated and untreated groups over a time period relevant to the intervention. While the unobserved factors might affect the outcome, if they do not affect trends in the outcome, then the trends for both groups in the absence of the policy will be the same. This is the so-called parallelism or "common trends" assumption. Any significant difference in trends is therefore interpreted as a policy effect.
9.48 The parallelism assumption should always be verified where possible, either by examining the pre-policy trends in historical time series data or from previous studies. Where the assumption does hold, DiD is a useful method that is able to address selection bias in the absence of rich information about the individuals under study. But the parallelism assumption should not be automatically assumed true, and a DiD approach would not be recommended if, for example, data are only available at two time points (before and after the implementation of the policy). Box 9.J provides an example of an evaluation using a difference in differences method.
Multifaceted evaluation of Workplace Health Connect (Health and Safety Executive) The Workplace Health Connect (WHC) pilot ran from February 2006 until February 2008. It was a free, no-obligation, service which aimed to provide small and medium-sized enterprises (SMEs) with advice on workplace health issues to increase the level of healthy workplaces across England and Wales. The primary research questions were: • whether the visit service made a net impact on the incidence and duration of occupationally related ill-health and injury; and • what the costs, benefits, and perceived barriers to full use of the service were. A multi-stranded methodological approach was developed to meet the objectives, which included surveys to collect data on service inputs; consider regional experiences; provide a comparator group; develop user case studies and; determine costs involved in being a WHC pilot user. In order to define the counterfactual for the quantitative impact study data was analysed on employers operating in regions where the WHC workplace visit service was not provided. These employers were the "comparator" group for WHC pilot users. Organisations in areas where WHC pathfinders were not in operation were selected for participation in the impact survey on the basis that they were similar (in terms of their size and sector) to those participating in the WHC pilot. Their outcomes, therefore, constitute the best available estimate of the counterfactual. The impact survey dataset included 520 organisations within the "treatment group" and 1609 organisations from the "comparator group". Each organisation was interviewed twice, with a year between interviews, regarding a variety of health and safety outcomes. One way of evaluating the impact of the WHC pilot would have been to look directly at the relationship between involvement in the pilot and final outcomes. This approach, however, was considered unlikely to produce robust results because in addition to improving safety using the pilot can change the way that the final outcomes are recorded. Instead the approach taken was to analyse the relationship in two stages, looking first at the effect of the WHC pilot on intermediate outcomes and then looking at the effect of the intermediate outcomes on the final outcomes. These relationships were examined using difference-in-difference analysis. This looks at the changes in outcomes between the two survey waves, and tests whether these changes are different for the WHC pilot user and comparator groups. In addition to the range of health and safety information gathered at the two interviews, information regarding general organisational characteristics was used to allow the analysis to control for these factors. There was no evidence that taking part in WHC had a direct measurable effect on rates of sickness absence. There was, however, evidence that involvement with WHC lead to improvements in a range of health and safety practices. These in turn were linked to a reduction in accident rates. The costs of the service, when the costs incurred by employers were included in the calculation, outweighed the pilot's measurable benefits20 |
_____________________________________________________________________
20 Workplace Health Connect Pilot: Evaluation Findings, Institute for Employment Studies, 2009, http://www.hse.gov.uk/