1.38 The DMO's assessment of the likelihood of delivering the key capabilities (or Measures of Effectiveness (MOEs), now referred to as Materiel Release Milestones (MRMs)), for major capital acquisitions, was specified publicly at the project level for the first time in the 2009-10 MPR. This data was set out as a 'traffic light' percentage pie chart in Section 3.5 of each project's PDSS. It is important to emphasise that MOEs/MRMs predominantly focus on the expected future achievement of certain technical, regulatory and operational requirements. Budgetary and schedule requirements are generally not included.
1.39 As at 30 June 2010, while some MOEs were historical (for example, that a certain capability is satisfied by a date which has passed), the majority of other MOEs required the DMO project managers to assess the likelihood of delivering in the future the aspects of the key capability for which the DMO is responsible.
1.40 Necessarily, this data usually involves making certain assumptions in forecasting achievements and is therefore subjective in approach (noted by the DMO in the 2008-09 MPR).54 In light of this subjectivity and inherent uncertainty, this information continued to be excluded from the scope of the ANAO's review. Nevertheless, noting this caveat, the ANAO included analysis of the DMO's data relating to its capability forecasts in addition to our analysis of projects' performance in regard to budgeted cost and schedule in Chapter 2.
1.41 In 2009-10, the ANAO's examination of MOEs, which were drawn from the MAAs between the Defence Organisation's Capability Development Group (as the purchaser) and the DMO (as the supplier), noted that the MOE framework was not sufficiently developed to ensure consistency in the level and scope of MOEs across projects.
1.42 The DMO subsequently advised that it was unlikely that MOEs would be reported in their previous form in the 2010-11 MPR as the DMO had established a new MAA template during 2009-10 for implementation in 2010-11.55 The new template does not include MOEs, but instead requires the specification of completion criteria for the achievement of materiel release to the ADF. In this context, the issue of key capability measures was to be a matter for consideration by the JCPAA and further examination by the ANAO in the 2010-11 MPR.
1.43 The MRMs and Completion Criteria are designed to define at a strategic level the key elements required to achieve IMR and FMR, which supports the achievement of IOC and FOC respectively. The DMO advised that the introduction of IMR and FMR milestones would provide greater clarity of responsibilities between the DMO, Capability Development Group and Capability Managers; set the two key milestones for delivering the materiel elements of capability to the Capability Manager; and provide a direct and more effective measurement of DMO performance. Further ANAO analysis of capability performance is set out from paragraph 2.49.
__________________________________________________________________________________
54 Australian National Audit Office, 2008-09 Major Projects Report, Part 2, p. 121.