Quantitative assessment in the appraisal phase of an investment is dependent on a sound evidence base, wherever possible developed from past procurement experience in both PPP and conventional procurement routes30. This evidence base must be continually updated to reflect the incorporation of new information from projects at all stages of procurement and operation, particularly where there are differences due to the procurement method31. Evidence from the operational phase must be fed back into the appraisal phase in order to improve future procurements.
Hence collecting information on actual outcomes is key to investment appraisal32. This information should be used in appraising all future programmes and projects and should be shared across the public sector. Responsibility for assembling and sharing evidence bases should probably rest with individual agencies, but it could be useful to have an authority responsible for overseeing the process, to ensure that information is shared both by sector and from one sector to the next, where this is relevant. Ex-post evaluation is conducted in a similar manner to an economic appraisal, focusing on cost benefit analysis, based on what actually occurred as opposed to what was anticipated. This is outlined in Figure 2 below33.
Figure 2 - Appraisal and ex-post evaluation
| Appraisal | Ex-post evaluation |
Aim | Ex-ante assessment of whether action is worthwhile and impacts | Ex-post assessment of whether action was worthwhile and impacts |
Use of output | Project procurement, policy and programme design | Feedback for: - future procurement, project management - wider policy debate - future programme management |
Application | Projects, policies and programmes | Projects, policies and programmes |
Timing | Always prior to implementation | - During implementation ("formative") - After implementation ("summative") |
Data | Forecasted | Historic and current, estimated and actual. Estimates of counterfactuals |
Method | - Comparison of options against "do nothing" option - Estimated assessment of risk | - Comparison of results against "do - Comparison of actual outturns against target outturns/alternative outturns - Assessment of risks that did or did not materialise |
Analytical Techniques | - Cost benefit/effectiveness analysis - Discounted cash flow analysis - Multi-criteria analysis - Other statistical analysis | - Cost benefit / effectiveness analysis - Discounted cash flow analysis - Multi-criteria analysis - Other statistical analysis |
Decision Criteria | - Comparison of net present value / cost for different options - Non-quantifiable factors may be included if quantification impossible | Consideration of whether correct criteria were used |
_________________________________________________________________________________
30 See HM Treasury, Value for Money Assessment Guidance 2006, p.12.
31 Ibid.
32 HMS Treasury, Value for Money Assessment Guidance 2004, p.10.
33 Treasury in the UK recognises the following five phases to an evaluation:
1. Establish exactly what is to be evaluated and how past outturns can be measured;
2. Choose alternative states of the world and/or alternative management decisions as counterfactuals;
3. Compare the outturn with the target outturn, and with the effects of the chosen alternative states of the world and/or management decisions;
4. Present the results and recommendations;
5. Disseminate and use the results and recommendations.