I am sorry that it has taken so long for me to reply to the questions raised by Mr Bacon. We have been looking back to see what material we can find. In the event, I am sorry that we are not able to provide the information requested by Mr Bacon. We agreed with Mott MacDonald that they should collect the information from the project owners, analyse it and provide advice on the basis of their analysis. Project owners were given assurances about confidentiality on the part of Mott MacDonald in order to encourage them to participate in the survey in an open and frank fashion and we did not seek to obtain the underlying project data. We have some data on a small number of major transportation projects but nothing else.
Mr Bacon asked about the selection of projects. Unfortunately, the exact process for selecting the final 50 projects that formed the basis of the Mott MacDonald sample has not been documented but I am advised that the main determining factor was the availability of suitable evidence. Nevertheless, the adjustment factors that Mott MacDonald recommended (derived from the sample) were well within their range of expectations and experience.
We do not consider that the underlying data for each project would provide added benefit to managers seeking to allow for optimism bias. As the Treasury's guidance on investment appraisal (the Green Book) points out:
"5.61 There is a demonstrated, systematic, tendency for project appraisers to be overly optimistic. This is a worldwide phenomenon that affects both the private and public sectors (Flyvbjerg, Underestimating Costs in Public Works Projects-Error or Lie, APA Journal, 2002). Many project parameters are affected by optimism-appraisers tend to overstate benefits, and understate timings and costs, both capital and operational.
5.62 To redress this tendency, appraisers should make explicit adjustments for this bias. These will take the form of increasing estimates of the costs and decreasing, and delaying the receipt of, estimated benefits. Sensitivity analysis should be used to test assumptions about operating costs and expected benefits."
Broadly, there are two main aspects of optimism bias-poor scope definition during the appraisal, and poor project management during implementation. However, while it is generally possible to predict in advance that cost and time overruns are likely to take place, it is not possible to predict precisely why. The detailed guidance based on the Mott MacDonald survey is an attempt to ensure that at least the common reasons for overruns are addressed; but it can never be a perfect science.
Like any other project parameter, there is of course uncertainty over optimism bias adjustments, especially early on in a project's lifecycle, and this should be reflected in sensitivity testing but the evidence from an even wider variety of projects (Megaprojects: Anatomy of ambition, Flyvbjerg et al) indicates that such adjustments could vary from around "20% to !200%, which is hardly useful in practice. Instead we have preferred to recommend that an average adjustment must be made early on, to encourage better costing and treatment of risk later on.
The Green Book points out that there will always be a need for judgment in carrying out this kind of analysis, rather than the application of a purely mechanistic approach:
"5.63 Adjustments should be empirically based (eg using data from past projects or similar projects elsewhere) and adjusted for the unique characteristics of the project in hand. Cross-departmental guidance for generic project categories is available, and should be used in the absence of more specific evidence (Review of Large Public Procurement in the UK, published July 2002). But if departments or agencies have a more robust evidence base for cost overruns and other instances of bias, this evidence should be used in preference. When such information is not available, departments are encouraged to collect data to inform their estimates of optimism, and in the meantime use the available data that best fits the case in hand.
5.64 Adjusting for optimism should provide a better estimate, earlier on, of key project parameters. Enforcing these adjustments for optimism bias is designed to complement and encourage, rather than replace, existing good practice, in terms of calculating project specific risk adjustments. They are also designed to encourage more accurate costing. Accordingly, adjustments for optimism may be reduced as more reliable estimates of relevant costs are built up, and project specific risk work is undertaken. Both cost estimates and adjustments for optimism should be independently reviewed before decisions are taken."
The Green Book contains the following example (page 30):
The capital costs of a non-standard civil engineering project are estimated to be £50 million in a strategic outline business case (SOBC). No detailed risk analysis work has taken place at this stage, although significant costing work has been undertaken. The project team reports to the project board and applies an optimism bias adjustment of 70%, showing that, for the scope of work required, the total cost may increase by £35 million to £85 million in total. This is based on consultants' evidence, and experience from comparable civil engineering projects at a similar stage in the appraisal process.
As this potential cost is unaffordable, the chief executive requests reductions in the overall scope of the project, and more detailed work for the outline business case stage (OBC). As the project progresses, more costs and specific risks are identified explicitly, despite the reduced scope. For the final business case, the optimism bias adjustment is reduced until there remains only a general contingency of 5% for unspecified risks.
Without applying optimism bias adjustments, a false expectation would have been created that a larger project could be delivered, and at a lower cost.
As the example shows, the expectation is that project appraisers will use evidence and experience to derive appropriate adjustments at different stages in the appraisal process. In assessing PFI, for instance, suitable adjustments should be made to both PFI and PSC options, as the Treasury's new draft guidance states (the draft can be viewed on the Treasury's website at http://www.hm-treasury.gov.uk/documents/ public private partnerships/key documents/ppp keydocs vfm.cfm). The size of the adjustments will depend on, among other factors, the level of risk transferred, and the stage the procurement has reached. By contract close, there will be little or no adjustment for optimism bias for the PFI option; the evidence from Mott MacDonald suggests this is reasonable. More generally, departments and agencies should be gathering evidence of cost overruns and other instances of bias to inform future appraisals.
The figures taken from the Mott MacDonald work exist to ensure that some adjustment is always made, which should help to ensure that costings are more accurate and risk work is undertaken.
Mr Bacon referred to GCHQ apparently having used the study results directly for their assessment of optimism bias, contrary to Mott MacDonald's advice. I have been in touch with GCHQ who have informed me that their contract predated the Mott MacDonald report and that the figure they used for optimism bias was based on earlier Treasury guidance from the then Treasury Taskforce on PFI. I am sorry that, at the hearing, my reference to the Mott MacDonald Report may have misled members as to the origin of the adjustment carried out by GCHQ.
Brian Glicksman
Treasury Officer of Accounts
15 March 2004
Printed in the United Kingdom by The Stationery Office Limited
6/2004 928625 19585