UK research on 'optimism bias'

In the UK, which has been at the forefront of development of the PPP procurement framework, there has been a large body of research on PPPs. The research that is most relevant to the methodology applied in the present study is that carried out by Mott MacDonald (Mott MacDonald, 2002).

The study focussed on measuring the relative degree of 'optimism bias' associated with Traditional procurement. 'Optimism bias' was defined as the percentage differential between the estimated works duration or capex cost at the 'Strategic Outline Case' (SOC) or 'Outline Business Case' (OBC) and Works Completion (WC). The results of the study are summarised in Table 3.1. Given that the UK's PPP projects had relatively neutral 'optimism bias', the table shows that for 'non-standard buildings, for example, the capex estimate for Traditionally procured projects suffered between 4 percent and 51 percent 'optimism bias'.

Another study that appeared in the UK soon after the Mott MacDonald study was by the UK National Audit Office (2003). The NAO compared the results of its 2002 Census on PFI Projects with the 1996 Procurement Survey for Traditionally procured projects. The NAO found that 76% of PFI projects were completed on time, and 78% were completed on budget. By contrast, for Traditional procurement only 30% of projects were completed on time and only 27% were completed on budget.

Evidence of value for money (VFM) calculations undertaken by the NAO was summarised by Allen (2001, p.30-33). There have also been a number of UK reports and academic studies that have looked at satisfaction levels and performance of PPPs. Cambridge Economic Policy Associates (2005) reported to the Scottish Executive that Scottish PPPs were generally performing well. An academic study by Kakabadse et al (2007, p.61), concluded that the 'emerging evidence is favourably inclined towards PFI' in the UK schools sector. However, a study sponsored by the UK Association of Chartered Certified Accountants (ACCA, 2004) provided a negative view of PFI projects in the roads and hospitals sectors.

 Table 3.1 UK: Mott Macdonald study of large public procurement (2002)

_____________________________________________
Source: Mott MacDonald (July 2002), Review of Large Public Procurement in the UK, Report to HM Treasury, United Kingdom.

*  Note: 'U' and 'L denote upper and lower bounds respectively.

In a wider review of evidence, case studies, positive and negative experiences, Pollitt (2005, p. 227) concluded:

It seems difficult to avoid a positive overall assessment. The UK PFI seems to have been generally successful relative to what might have happened under conventional public procurement. Projects are delivered on time and to budget a significantly higher percentage of the time.

Pollitt (2005, p. 226) raised the prospect that the full benefits of PPPs were not confined to PPP projects, but extended to Traditional procurement as a 'vehicle for learning'. Pollitt (2005, p. 227) also felt it was important that the benefits of the PFI were not disproportionately captured by the private sector, but was confident that financial windfalls to private investors could be addressed via 'appropriately specified contracts'

The relatively higher bid costs of PPPs have often been commented on. Allen (2001 p.34) reported on the Adam Smith Institute's (1996) study, which concluded that tender costs expressed as a percentage of total costs were in the region of 3 percent for PPPs, and just under 1 percent for Traditional procurement.

For the purposes of making value for money assessments, the results of the Mott MacDonald study have been interpreted as requiring that an addition to cost estimates under the Traditional procurement alternative (the Public Sector Comparator, or PSC) be made in the range given by the table.

In a critique of the Mott MacDonald (MM) and UK NAO results, Unison (2005) outlined a number of methodological problems that it considered were biasing the empirical findings in favour of PPPs. These methodological issues were as follows:

• Transparency – The Mott MacDonald and NAO sampling methodology was not described.

• Population analysis – The populations from which the PPP and Traditional samples were not described.

• Representativeness of samples – There was no detailed description of how representative the samples were of the PPP and traditional populations.

• Consistency and relevance of time period - Some Mott MacDonald traditional projects were drawn from an earlier period, which pre-dated the procurement reforms of 1999.

• Relative complexity selection bias – A greater proportion of Traditional projects were 'non-standard', and therefore involved a higher degree of complexity than PFI (PPP) projects.

•  Measurement bias - According to Unison, Mott MacDonald measured PFI (PPP) projects from the later time of the full business case (FBC), and traditional projects from the strategic outline case (SOC) or outline business case (OBC), which came earlier, and were therefore likely to involve greater time and cost uncertainty.

We have outlined these methodological issues here because the present study's methodology has undertaken to address each of them. Whilst not intending to provide a defence of the Mott MacDonald and UK NAO methodologies, we would note that the Mott MacDonald study did point out that its results showed an improvement (i.e. considerable lessening of optimism bias) over time, and that is why it provided a significant range for its estimate of the relative optimism bias of Traditional projects.