A3.7 Identifying the criteria may sound straightforward, but criteria must be clearly defined so that both appraisers and those reviewing appraisal reports have a clear understanding of them. To help in the scoring of options, criteria should be defined as far as possible in service or output oriented terms, and they should generally relate closely to the service objectives and performance measures established at the outset of the overall appraisal Considerable care is also needed to ensure that:
(i) there is no double counting caused by an overlap in the criteria (e.g. aesthetic qualities and attractiveness);
(ii) there is no double counting caused by criteria being covered by costs (e.g. including a 'reliability' criterion when reliability is already provided for by inclusion of maintenance costs); and
(iii) all relevant criteria are included, even if they are common to all the options.
A3.8 Regarding point (iii), it is important to include cover relevant criteria even when all the options appear to impact equally upon them. Omission of common criteria can distort scores and lead to an imbalanced comparison of the differences between the options. For example, Options X and Y may score 200 and 100 respectively, when common criteria are overlooked, giving the impression that X is twice as beneficial as Y. However, if common criteria are worth 300, the correct scores for X and Y should be 500 and 400 respectively, indicating that X has a significantly smaller advantage over Y when all the non-monetary factors are taken into account. Apart from distortion of scores, there is a general risk that the appraisal may focus on criteria that are relatively insignificant while overlooking the most important criteria.
A3.9 Criteria are best defined so that the status quo or do minimum baseline option can be given a score other than zero. For example, if one of the project objectives is to improve access for the disabled, the criterion is better defined as 'accessibility for the disabled' than as 'improvement in accessibility for the disabled'. The first definition allows all of the options, including the baseline option, to be scored, and thus enables the options to be compared in proportion to the baseline. The second definition necessitates a zero score for the baseline option, which means that the scores for the alternatives can not indicate how much better they perform than the baseline option4.
Example: In a certain health service appraisal, the relevant criteria are identified as:
■ number of cases treated;
■ waiting time;
■ patient access; and
■ disruption to services.
__________________________________________________________________________________
4 This is not to say that the baseline option should never be given a zero score. In the accessibility example, the baseline option will deserve a score of 0 if the current provision is completely inaccessible to the disabled. However, the more likely position is that the disabled can access it with a degree of difficulty, in which case a suitably small positive score would be appropriate