Economic appraisal assessment

4.12  The purpose of the economic appraisal assessment phase was to critique the BCRs submitted with candidate initiatives so as to identify whether the proponent's economic analysis could be relied upon in developing the Interim Priority List. In April 2010, the Office of the Infrastructure Coordinator summarised its approach in advice to ANAO as follows:

The proponents' economic appraisals were reviewed by a team of consultants with expertise in economic appraisal of projects. Appraisals were reviewed against 22 criteria, and the reviewer's comments and rating were then moderated by Infrastructure Australia's lead economic appraisal consultant. The criteria enabled Infrastructure Australia to look behind the headline benefit‐cost ratio, and thus establish whether the appraisal was based on solid inputs or not. At the very least, it enabled Infrastructure Australia to flag areas of doubt in relation to the economic assessment of proposals.

4.13  Of the 94 initiatives that were shortlisted for detailed evaluation, 27 (29 per cent) did not at that time have a BCR. The 1 December 2008 Evaluation Report recorded these and other shortcomings in respect to the BCRs submitted by proponents as follows:

It became apparent to the six independent experts (on day one of their work) that, in addition to the variation in methodology, the state of readiness and applicability of the economic Cost‐Benefit Analysis (CBA) also differed greatly among submissions. For example:

•  Incomplete or high level 'Strategic' or 'Rapid' economic appraisals were submitted, and it was clear that the proposer intended to submit more comprehensive and specific CBA evidence in the future, or

•  CBAs were not specific to the initiative: either elements had simply been transferred from similar initiatives, or the BCR quoted was simply the BCR from different but similar initiatives, or

•  Important pieces of information or analysis were missing from the CBA material submitted, or

•  Economic CBAs had been prepared which were in effect financial analyses, treating financial revenues as benefits, which are not incorporated in economic CBAs, or

•  Benefits relied on associated private sector investment to promote demand, where that investment was not part of the initiative and so speculative.

4.14  Rather than exclude submissions exhibiting these shortcomings from further analysis, the approach taken was to record shortcomings in the information supplied, with the advisers to consider whether an economic cost‐ benefit analysis was the most applicable method of assessment and, if so, whether an assessment could be undertaken on the basis of the information that had been submitted. The Chair of the Infrastructure Australia Council has informed ANAO that this approach was taken so as to ensure that candidate nation building projects were provided with the maximum possible opportunity to be considered, rather than being excluded from consideration because the initial business case did not adequately demonstrate the benefits of the project. In respect to the analysis, the Evaluation Report recorded that:

After two days, the rate of progress was discussed between the experts and Infrastructure Australia staff, and the experts were asked to focus on initiatives with a reported BCR of above 1.5 in order to prioritise those likely to be candidates for the Infrastructure Priority List in the time available (therefore some initiatives with a BCR of below 1.5 were assessed before this administrative decision was taken - this information was retained).

…The experts assessed all initiatives with a BCR of above 1.5 in the two week period. However, four experts were brought back to Infrastructure Australia to assess a number of initiatives which were either late or for which updated information was provided.

4.15  The approach taken to the economic appraisal was robust and comprehensive. Each initiative subject to the appraisal assessment was examined against 22 factors (see Table 4.2) so as to decide whether the BCR submitted by the proponent for the initiative could be relied upon, or required moderation.

Table 4.2  Factors assessed in deciding whether an initiative's Benefit-Cost Ratio should be moderated

Source:  ANAO analysis of Office of the Infrastructure Coordinator data.

4.16  The Office of the Infrastructure Coordinator's records evidence that initiatives were critically examined against the 22 criteria so as to identify whether the approach taken by the proponent supported the BCR that had been submitted or whether the BCR was either understated or overstated. Initially, an ordinal scale was used to record the result of the assessment for each initiative against the 22 criteria. This ordinal scale was then converted into a numerical scale, as outlined in Table 4.3.

Table 4.3  Appraisal assessment scoring system

Source:  ANAO analysis of the Office of the Infrastructure Coordinator records.

4.17  Using a predetermined methodology outlined in the Evaluation Plan, the individual scores against each of the criteria for each initiative were totalled and an overall moderation assessment allocated.111 Initiatives were then ranked, using a rule‐based combination of the BCR submitted by the proponent and the moderation assessment to rank projects into four categories of:

•  Exceptional BCR-where the proponent submitted a BCR greater than 3.5 which the Office of the Infrastructure Coordinator assessment concluded was not significantly overstated;

•  Strong BCR-where the proponent submitted a BCR above 1.5 but less than 3.5 with the BCRs at the lower end of this range having been assessed as being understated and the BCRs at the upper end of the range having been assessed as significantly overstated or flawed (which is why they were not categorised as Exceptional);

•  Satisfactory BCR-comprising BCRs submitted by the proponent between 1.5 and 2.5 (with the BCRs at the lower end of this range having been assessed as being significantly understated, and the BCRs at the upper end of the range having been assessed as significantly overstated or flawed (which is why they were not categorised as Strong)); and

•  Poor BCR-the BCR submitted by the proponent was less than 1.5 and the Office of the Infrastructure Coordinator had concluded that the BCR was overstated.

4.18  In addition, due to information limitations, the appraisal assessment did not include a review of certain non‐monetised impacts or inter‐ generational impacts. In this respect, in May 2010, the Office of the Infrastructure Coordinator advised ANAO that:

Some assessment of claimed Wider Economic Benefits (WEBs) was undertaken. Often, the information supporting these benefits was quite limited and/or the methodology was not considered robust. As a result, in the main, the claimed WEBs were not included in Infrastructure Australia's appraisal assessments at the Interim Infrastructure Priority List stage.

4.19  In addition, in a significant number of instances, the Office of the Infrastructure Coordinator concluded that the information provided by the proponent was insufficient112 to allow an assessment to be made against one or more of the 22 criteria. The Evaluation Plan had proposed that a distinction be drawn between those initiatives for which adequate information was available and those for which more information was needed before an informed assessment could be completed. So as to provide a detailed indication of the nature of any outstanding issues with each proposal, initiatives were categorised into one of five categories:

•  Satisfactory information provided, such that (despite some areas of poor information) the general assessment was robust and an assessment could therefore be made;

•  Major doubts raised, such that significant further information was required to support an assessment;

•  More information required, which was similar to the Major doubts raised category except that, rather than questions being raised about the BCR methodology, the concern was that too little information had been provided;

•  Information incomplete or early stage, such that no assessment was possible; and

•  Information not comparable, which was the category applied in circumstances where the assessment had concluded that the economic analysis provided by the proponent was not an appropriate or 'fair' tool of assessment (for example, where financial revenues or indirect benefits, which are not usually counted in economic cost‐benefit analysis, formed the principal source of the claimed project benefits).

•  Identify and test credible 'highly beneficial' or 'highly detrimental' non-monetised impacts; or

•  Identify significant equity or intergenerational impacts.

However, the experts did note whether Wider Economic Benefits (WEBs) were identified, and made an assessment of their validity

4.20  With one exception, the records held by the Office of the Infrastructure Coordinator supported the ranking of projects in terms of their economic appraisal. Specifically, the Evaluation Report included the New South Wales Regional Water Feasibility Studies initiative as a project with a Strong BCR but with Major doubts raised about the BCR. In particular, the Evaluation Report recorded the conclusion that the BCR was critically flawed but whereas another initiative in this same category (the Northbridge Link Rail Cutting) also recorded as being critically flawed was included in the merit matrix, the New South Wales Regional Water Feasibility Studies initiative did not progress any further in the evaluation.

___________________________________________________________________________________________

111  The options were: significantly understated, significantly overstated, slightly understated, slightly overstated, broadly neutral, or critically flawed.

112  The Evaluation Report also outlined a further departure from the Evaluation Plan: 'Due to the quality of information provided and the time available, the experts were not asked, as originally envisaged in the Evaluation Plan, to: