Evaluation and real cost comparisons
4.14 Evaluation of projects is an important part of learning lessons and improving future procurement. As part of a benefits realisation process, it can also help further the aims of the project. And it is particularly important given the inherent limitations of business case assumptions and the need to rigorously test them in practice.
4.15 A systemic weakness that we have identified across Government but especially for PPPs is the poor management of benefits realisation.70 Benefits realisation is the ongoing process of identifying a project's aims, the means of achieving them, responsibility for each, and monitoring and assessment arrangements to assess their achievement. Without such an approach there is no obvious driver to push and promote the purpose of the project, and yet many public authorities appear not to take benefits realisation seriously:
a Benefits realisation is rarely integrated into the project management. The Highways Agency predicted the benefits of its National Roads and Telecommunications Services contract would be £2.8 billion, but relied on external Gateway reviews to identify the benefits achieved.71
b Projects often focus on delivery of a plan and forget the intended benefits. The Building Schools for the Future programme aims to improve education attainment as well as renew school buildings, but national targets unduly emphasise the timeliness of delivery of new buildings, and local targets lack ownership and detail.72
c Authorities often include benefits in their assessment that they have no concrete plans to deliver or assess. The Channel Tunnel Rail Link business case relied on creating £450 million of regeneration, but the Department for Transport had no mechanisms to ensure others delivered this benefit or to measure whether it was achieved.73
4.16 We are also yet to see robust cost analysis by any Department or public authority. Cost comparisons using real data from life projects would allow the whole-life costs of PPP projects to be compared to similar projects. This can be done across a programme to analyse variations in costs, between procurement approaches to evaluate delivery methods, or for a specific project to assess the reasonableness of its costs. Without such cost comparisons, it will not be possible to ever prove whether using private finance is more efficient or more expensive than conventional funding.
4.17 No two projects are exactly the same, so cost comparisons will not say whether an individual project is VFM. But it can provide useful management information about how costs compare to a norm, or whether costs are generally reasonable. Large homogenous programmes such as hospitals, schools, housing and office buildings can provide enough data to show clear trends, and managers can assess whether variances from these trends are justifiable given the specific circumstances of each project. Cost comparisons are much harder for one-off projects, such as the London Underground PPPs or Defence projects.
4.18 The main reason that we have not seen such costs comparison is because departments do not collect data on whole-life costs of projects in a systematic way:
a Central Government rarely collects data from Local Government funded projects or devolved funding.
b PPP costs are rarely collated centrally, and where they are, they are hardly ever updated for contract variations.
c The costs of ongoing services for conventionally procured buildings are rarely monitored, making whole-life costs very difficult to compare.
d Different procurement routes collect data on different bases.
4.19 On the other hand, cost comparisons are often used during the operational phase of a PFI project to benchmark the cost of specific services and ensure that they closely resemble the market norm. They are also used to benchmark the project costs of new projects developed by Partnerships outside of competition - e.g. by Local Innovation Finance Trusts and Local Education Partnerships (Case study 6).74 Such benchmarking systems are not cost comparisons between systems, but could start as the basis for collecting the data needed for such a cost comparison.
Case study 6 |
Partnerships for Schools' benchmarking system promises to provide robust data across schools funded through the Building Schools for the Future programme. We were able to use this to benchmark capital costs between centrally funded programmes, but we could not assess whole-life costs or compare them to devolved funding streams (the majority of expenditure), using available data. |
The price of BSF schools compared to the price of Academies and PFI schools built before the BSF programme |
This figure shows that there is a great deal of variation in the individual price of each school depending on the type of site and facilities included, but that the prices of BSF schools are not significantly different from PFI schools and generally less than Academies. The price is shown as capital cost to the public sector, adjusted for annual price inflation and location to allow a better comparison. The figure excludes refurbishments, because they vary too much in the amount of work undertaken to allow a comparison. |
Capital cost (£m) in 2007 prices and adjusted for location |
|
__________________________________________________________________________________
70 The procurement of the National Roads Telecommunications Services, National Audit Office (HC 340, 2007-08); The Building Schools for the Future Programme: renewing the secondary school estate, National Audit Office (HC 135, 2008-09); Innovation in the NHS: LIFT, National Audit Office (HC 28, 2005-06); Home Office: the implementation of the National Probation Service Information Systems Strategy, National Audit Office (HC 401, 2000-01).
71 The procurement of the National Roads Telecommunications Services, op.cit.
72 Building Schools for the Future: renewing the secondary school estate, op.cit.
73 Progress on the Channel Tunnel Rail Link, National Audit Office (HC 77, 2005-06); The Thames Gateway: laying the foundations, National Audit Office (HC 526, 2006-07).
74 The Building Schools for the Future Programme: renewing the secondary school estate, National Audit Office (HC 135, 2008-09); Innovation in the NHS: LIFT, National Audit Office (HC 28, 2005-06).