What factors affect how a policy should be evaluated?

1.8  Evaluations are a crucial (and in some instances mandatory - see Box 1 .A) part of the policy cycle set out below and offer both strategic and practical benefits. Therefore, while it might be tempting to do without an evaluation, or to 'muddle through' with a less formal, more subjective assessment of a policy's performance perhaps for time or resource related reasons, or the risk of a 'difficult' conclusion - such an approach is not without cost. A decision not to evaluate a policy, or only to evaluate it in a less formal or reliable way, is associated with a number of real risks:

•  a policy which is ineffective might continue;

•  overall adverse or costly impacts will be generated, now or in the future; or

•  opportunities to improve the policy, or to save money or reinvest in other, more worthwhile projects might be missed.

1.9  Conversely, even if the policy is actually highly effective or generates good value for money, a substandard (or absent) evaluation will mean:

•  Policy makers cannot justifiably claim that any positive outcomes they might observe were actually caused by the policy rather than by chance or were attributable to an alternative policy; and

•  as a result, policy makers could not claim that their intervention delivered value for money, or had been demonstrated through sound analysis to be effective.

1.10  The key here is clearly the meaning of the phrase "good evaluation", what defines a good evaluation and what is necessary to achieve one. This is the subject of subsequent chapters of the Magenta Book. A wide range of factors needs to be taken into account when deciding what sort of evaluation is necessary and appropriate in any given case. These include:

•  the nature of the policy, its objective scale, complexity, innovation, form of implementation and future direction;

•  the objectives of the evaluation and the types of questions it would ideally answer;

•  the timing of key policy decisions and the information on which they need to be based;

•  the types of impacts which are expected, the timescales over which they might occur, and the availability of information and data relating to them and other aspects of the policy; and

•  the time and resources available for the evaluation.

1.11  The choice of evaluation will often involve some trade-offs between these factors, which are considered further in Chapter 2. In some cases, it might be proposed that an intensive, rigorous evaluation is not justified, and a more limited, "lighter touch" evaluation is more appropriate. In others, it could be better to choose a more rigorous evaluation with a more restricted scope, since at least then the evidence obtained should be useful and reliable. However, such choices must be made in full recognition of the limits they are likely to place on what can subsequently be said on the basis of the results obtained.

1.12  The earlier that an evaluation can be planned in the policy development process, the more likely it is that it will be possible to consider these trade-offs and choose the most appropriate evaluation. The later in the policy process the evaluation is considered the fewer options there are for undertaking it. Judgement needs to be made during the development of the policy on the scale and form of evaluation that is required, which might even extend to considering whether policy implementation might be adjusted to make a stronger evaluation more feasible. This judgement will involve some technical issues and should therefore be made in consultation with analytical specialists who can advise about the trade-offs involved and the implications of different choices.