The value-for-money test that compares the cost of P3s with conventional procurements lies at the heart of the P3 procurement process. This is because it helps the public sector procurement authorities determine not only which projects should be pursued as P3s but also how a project should be structured (e.g., which risks should be retained, transferred to the private partner, or shared between the two parties) in order to deliver the most value to the public sector. However, there is some skepticism, including in the academic literature, as to whether the VfM test is a genuine test or whether it can be arbitrarily managed to generate desired results. This section reports the results of a high-level review of selected VfM studies in the four jurisdictions considered here, including any guidance documents regarding the methodology used for these studies.
First, we should note that VfM studies have been conducted for every Canadian P3 project undertaken as part of the second-wave of P3s. This is considered standard practice for P3s in most countries in Europe as well as in other pioneering jurisdictions in this area, such as Australia. However, it represents a significant achievement when viewed in the context of conventional infrastructure procurement, which is not usually subject to a VfM assessment that compares the chosen method of procurement with alternatives.
Second, we should note that the VfM test is a process that begins well before the request for proposal (RFP) is issued and culminates in a final report issued after the financial close. The first VfM test for a project is finalized before the RFP is issued in order to confirm the procurement decision before engaging the market in a competitive bid process. The VfM test is then finalized after financial close of the project, based on the financial information contained in the proposal of the winning bidder.
Our review of the available VfM studies and guidance documents suggests that each of the four jurisdictions under consideration has developed a rigorous methodology for comparing the costs of P3s and traditional procurements. (VfM studies are not published for P3 transactions in Alberta, but the VfM methodology is available through Alberta Infrastructure and Transportation.)8 This means that it is generally clear which data inputs have been used, what analysis was undertaken, especially regarding the assessment of risks, and what key assumptions were made (e.g., regarding the choice of discount rate for the two options). In addition, the choice of methodology and underlying assumptions are generally conservative.9 Although there are some differences in methodology between jurisdictions (e.g., the method of determining the appropriate discount rates for the analysis), we have not undertaken a detailed assessment to determine whether the methodological differences have a material impact on the VfM results.
|
A recent review of P3s suggested that vfm studies should be based on a full cost-benefit analysis of the difference between the two procurement options. |
The results of the VfM studies have in some cases been reviewed by the provincial auditor general, as in the case of British Columbia's P3s.10 In Ontario, the internal audit division of the provincial Ministry of Finance reviewed the VfM methodology, which was "found to be sound."11
A recent review of P3s suggested that VfM studies should be based on a full cost-benefit analysis of the difference between the two procurement options, which we discuss in the box "The Role of Cost-Benefit Analysis in the Evaluation of P3 Projects."12
Our review of the VfM studies and the methodology leads us to the following observations:
◆ It would be worth comparing the P3 procurement option with the next best available procurement option.13 In many cases, this is the conventional procurement option, which is the usual reference point for these studies (e.g., CCDC 2 contracts and conventional maintenance contracts when the P3 includes a maintenance phase). However, in some cases it may be a different type of conventional contract, such as a construction management contract. This modification of the methodology, when relevant, would be consistent with provincial capital management frameworks that indicate that all procurement options should be evaluated.
◆ The risk assessment process is at the heart of the VfM methodology and is necessarily based on historical outcomes regarding the cost and timing out-comes of both conventional and P3 infrastructure projects. We think it would be worthwhile developing and maintaining an evidence base of pan-Canadian infrastructure projects covering key outcomes such as public sector project costs and key milestones relative to their respective budgets and delivery timelines. This can already easily be done for the second wave of P3 projects. However, it is likely to be more challenging, but more valuable, to under-take for conventional infrastructure projects.
◆ As we noted earlier in this report, a VfM test is necessarily ex ante. It could therefore be valuable to update the VfM study after completion of the project or after a major milestone such as completion of the construction phase. The resulting data could provide some valuable lessons regarding best practices for infrastructure procurement.
The Role of Cost-Benefit analysis in the Evaluation of P3 Projects Several authors have argued that cost-benefit analysis should play a more important role in the evaluation of major infra-structure projects in Canada. For example: Steven Globerman and Aidan Vining suggest that ultimately the effectiveness and desirability of P3s and related instruments depend on their ability to meet the needs of society as a whole, that is, whether the net social benefits of P3s are likely to be higher (or are actually higher) than government provision. This criterion has a strong normative rationale and has been used to evaluate the privatization of state-owned enterprises.1 We fully agree that major infrastructure projects must be subject to a rigorous and comprehensive cost-benefit analysis and that the results of the analysis should be an important factor in deciding whether to proceed with the project (or what version of the project to proceed with). The analysis must include not only the financial costs and benefits but also other quantifiable social costs and benefits that fall in the public domain and are not captured in financial business cases (e.g., health, environmental, and safety impacts). However, this type of study must be undertaken at the project evaluation stage (i.e., well before the procurement stage) in order to contribute meaningfully to the decision about whether the project is in the public interest and worth undertaking. In the passage below, we explain what the role of cost-benefit analysis should be in transportation infrastructure projects. However, the same rationale applies to all publicly owned infrastructure projects, from hospitals to concert halls: . . . the planning process should favour those trans-portation projects with the highest benefits per unit of cost. The public sector is the only participant in a position to ensure that projects that generate significant net benefits in the public domain (e.g., environmental or even journey-time benefits, which are reflected in the benefit-cost analysis but not in the financial business case) are prioritized and realized. Private firms' investment decisions are based on the financial case, which necessarily ignores the costs and benefits that fall in the public domain. It is therefore the unique and sole responsibility of governments (and planning agencies) to identify and promote transportation projects with relatively high benefit-cost ratios.2 Once a project proceeds to the procurement stage, it is worth revisiting the cost-benefit analysis undertaken at the project planning stage to assess whether the social costs and benefits (i.e., the external impacts that fall in the public domain) would vary depending on the procurement option. But it is not clear to us whether the type of procurement tool materially alters the social costs and benefits resulting from a project.3 Vining and Boardman argue that the procurement decision (i.e., P3 or conventional project) should be based on the following criterion: . . . recognizing the importance of externalities and quality differences, Anthony Boardman and Erica Hewitt argue that governments should minimize the sum of total social costs defined as production costs incurred by government or paid to third parties, plus transaction costs, plus (net) negative externalities, holding quality constant.4 We note that the VfM methodologies discussed above take into account all the factors in the quote above, with the exception of the externalities (i.e., the social costs and benefits in the public domain). But we have no evidence to suggest that externalities differ significantly between procurement options, as noted above. Nor do we believe it is the role of VfM studies to serve as a full cost-benefit analysis in any instance where a cost-benefit study may not have been undertaken at the planning stage.5 ______________________________________________________________________________ 1 Vining and Boardman, "Public-Private Partnerships," p. 14. 2 Iacobacci, Steering a Tricky Course, p. 9. 3 This is particularly true when comparing conventional procurement approaches with P3 approaches based on availability payments. Projects that transfer substantial use risk (or demand risk) to the private sector could affect the balance of social costs and benefits, but second-wave P3 projects do not typically do so. 4 Vining and Boardman, "Public-Private Partnerships," pp. 14-15. 5 Such a case would be considered a planning failure and should be corrected prior to completing a procurement process for the project in question. |
___________________________________________________________________________________________________________
8 See Alberta Infrastructure and Transportation, "Management Framework."
9 For example, Infrastructure Ontario's VfM methodology assumes the same base capital costs under the PSC and the shadow bid (with the exception of the risk premium under the shadow bid). In other words, it assumes that the private partner does not bring any innovations to the project, although in practice this would likely occur in a well-designed project. See Infrastructure Ontario, Assessing Value for Money; Partnerships BC, "Draft Discussion Paper"; and Alberta Infrastructure and Transportation, "Management Framework." A shadow bid refers to the financial model of the costs of undertaking the project in question as a P3 procurement. This model is developed by the P3 agency (or procurement authority) for the purpose of comparing the cost of a P3 against its PSC prior to receiving final bids from the private partners.
10 The B.C. Auditor General reviewed the VfM documents for the Abbotsford Regional Hospital and Cancer Centre Project, the Sea-to-Sky Highway Improvement Project, and the Canada Line Project. See Partnerships BC, Project Report: Achieving Value for Money-Abbotsford Regional Hospital; Partnerships BC, Project Report: Achieving Value for Money-Sea-to-Sky Highway; and Canada Line Rapid Transit, Canada Line Final Project Report.
11 Auditor General of Ontario, "Brampton Civic Hospital," p. 121.
12 Vining and Boardman, "Public-Private Partnerships," pp. 14-15.
13 This is already the standard in British Columbia, as set out in the CAMF.