The need for, but inherent limitations of, financial modelling

4.8 Quantification of the expected costs and benefits of a project allows for better decision-making. But it also requires public authorities to make fine judgements about the future and risk, and to consider issues which are inherently uncertain.

4.9 The main financial model prepared for PPPs is the Public Sector Comparator (PSC). The PSC compares the costs and benefits of using private finance as opposed to conventional finance. Such comparators are very useful in helping to identify the costs, benefits and risks of a project and forcing the procuring authority to consider alternative procurement approaches. But like any financial model, they cannot be relied upon as a sole source of assurance. They are susceptible to manipulation and we often find problems with their implementation:

(a) PSCs are often used wrongly as a pass or fail test, with departments quoting figures saying that PFI is cheaper or more expensive than the PSC. PSCs produce estimates of the expected (ie average) net present value of a large number of potential scenarios, and thus represent the most likely figure of a range. The Ministry of Defence main office building PFI had a PSC that showed PFI was as just over a 10,000th cheaper than the favoured PFI routes, although there was a 50:50 chance that it was more expensive (Figure 6).60 We found that MoD considered it important for presentational purposes in its final analysis of the proposed deal that the final PFI deal price was below its estimated average cost of conventional procurement, but that in reality the costs of PFI and conventional procurement were similar.

(b) PSCs adjustments for risk are subject to inevitable uncertainties. The Highways Agency relied on the subjective judgement of its advisors to risk adjust the PSC for the National Roads and Telecommunications Services contract, using their experience and judgement. This produced a result that the PSC was marginally more expensive than PFI.61

(c) PSCs do not always represent the best counterfactual. The National Savings and Investment's PSC for its PPP with Siemens Business Services assumed that capital funding would be restricted by annual spending limits, whilst PFI funding was additional. Lifting this restriction would have significantly narrowed the gap between the PFI and the PSC.62

(d) Comparisons to PSCs do not assess the wider net-benefits of a procurement approach. For example, the West Middlesex University Hospital PFI estimates were originally more expensive than the PSC, but the Trust decided that the other benefits made PFI VFM, notably the incentive on the contractor, Bywest, to complete the redevelopment quickly and with price certainty.63

(e) PSCs, like all models, are prone to human error. Furthermore, sense checking of models is more likely to detect errors whose correction is in favour of the desired outcome. The West Middlesex Hospital Trust's advisers strove to make slight adjustments to the PSC within the range of human error to make the PFI cost appear marginally cheaper than the PSC.64 The Dartford and Gravesham Hospital PSC contained £12 million of errors. The Trust did not test the sensitivity of its PFI savings during procurement, and had it known they were marginal might have achieved better value for money.65

(f) Important scenarios are often missed. The 11 September 2001 terrorist attacks led to a significant drop in air traffic, causing the National Air Traffic Services PPP to lose significant income and require refinancing. Of the nineteen scenarios modelled, only one catered for the risk of a brief downturn in income and the Department ignored historical evidence of ownturns in making its overall assessment.66

4.10 The Treasury responded to our criticisms of PSCs in its 2004 revision of its VFM guidance, stressing that the PSC should be only part of the assessment of VFM. The NAO agrees. PSCs are a very useful and vital part of PPP business cases. They provide some assurance that the public authority has carefully considered the costs, benefits and risks of a project and considered alternative procurement routes. They do so in a way that is better than most quantification techniques of costs and benefits that we see across Government. But they are only financial models and have inherent limitations, including being subject to manipulation and challenge. They should not be relied upon alone.

_____________________________________________________________________________________________________________
60 Redevelopment of MoD main building, National Audit Office (HC 748, 2001-02).

61 The procurement of the National Roads Telecommunications Services, National Audit Office (HC 340, 2007-08).

62 National Savings: Public-Private Partnership with Siemens Business Services, National Audit Office (HC 493, 1999-2000).

63 The PFI contract for the redevelopment of West Middlesex University Hospital, National Audit Office (HC 49, 2002-03).

64 Delivering better value for money from the Private Finance Initiative, PAC (HC 764, 2002-03).

65 The PFI contract for the new Dartford & Gravesham hospital, National Audit Office (HC 423, 1998-99).

66 Public Private Partnerships: Airwave, National Audit Office (HC 730, 2001-02).