9.49 A situation where there is no physical comparison group might arise if the policy was introduced everywhere simultaneously, or if there are no data available on non-participants. In this situation, the evaluator can attempt to estimate a counterfactual from a forecast or projection of the outcome measure derived from the pre-policy history, and compare it with the actual outcome. This is the basis of the interrupted time series (ITS) design. In practice, this design can only be used when:
• the nature of external influences is sufficiently well understood to eliminate any alternative causes; and
• the impact is sufficiently large compared with the error inherent in the forecasting procedure. In practice, only very major policy changes, that overturn a persistent historical trend, or overwhelmingly dominate sources of random fluctuation, can be detected with this method; the statistical power of an ITS is generally much lower than for designs involving a comparison group.
9.50 As a result of these restrictions, the analyst should be aware that the ITS can only be used rather rarely in public policy evaluation.
9.51 An alternative approach which is sometimes possible when there is no physical comparison group is to examine alternative outcomes which, other things being equal, have been seen to move in parallel with the one targeted by the policy. For example, a policy targeted against a particular crime type could compare outcomes for a different crime type which historically has had a similar trend; or an intervention based on cancer screening could look at outcomes for a different type of cancer. As with the ITS, the evaluator should remain alert to the possibility of reasons other than the policy why the two outcomes might have diverged.
9.52 The above discussion has not provided a comprehensive "listing" of all of the possible approaches to estimating a counterfactual. Rather, it has sought to explain the thinking behind identification strategy, and how different problems in counterfactual estimation might be addressed. The identification strategy inevitably involves making some assumptions, which in many cases can be relatively strong. Any evaluation should include an explicit acknowledgement of these assumptions, and comment on their plausibility - where it is possible to test the assumptions directly it should be done.
9.53 It is clear that each alternative approach that has been discussed has its advantages and disadvantages and it is often difficult to provide prescriptive guidance and instructions on how to go about deciding which is the best approach for a given problem situation. Judgment and common sense should drive the decision making process.
So far this chapter has been couched in terms of analysing the outcomes of individual people; there are of course other types of evaluation. In many cases, there is interest not only in the outcome of individuals, but of the units to which they belong - a good example is schools and pupils. Ideally the evaluator will have access to data at the individual pupil level, and also know the schools to which they belong. If these data are available,21 then in many cases an appropriate approach is multi-level modelling (MLM) (more detail is provided in the supplementary guidance). This allows the analyst, in this example, to model explicitly the effects on outcomes of both school level factors and individual pupil level factors, and see which of these are more important. In some cases however, either the data for individuals are not available, only the unit level aggregates (such as school league tables), or the outcomes are only meaningful at the unit level, such as profits data for businesses. In such cases exactly the same considerations apply in principle as for evaluation of individual outcomes. There are however likely to be differences in practice. There are likely to be fewer units in the population, making it impractical to have very large samples. The units are likely to be more diverse than individual people. And it is more likely that the intervention affects units to a differing and measurable degree (e.g. some additional source of funding for schools), which can be utilised in the evaluation. A further degree of abstraction is when data are only available at a population level. Again, this can be either because the data are aggregated up from individual outcomes, but only the aggregates are available, or because the data are genuinely available only at population level. An example of the latter might be interest rates. The constraints on the availability of data will guide the available analytical approaches. Where only population data are available, or where all units are affected by the intervention at the same time, time series modelling might be a viable approach. Where the degree to which units are affected is monitored and known, the marginal effect of increasing the intervention intensity can be modelled. |
_________________________________________________________________
21 In some cases, even when data have been recorded, they may not be readily available to the evaluator for a variety of reasons, such as data protection.