Project snapshot-Key capabilities expected to be delivered and project maturity progress

2.49  The ADF defines capability as the capacity or ability to achieve a particular operational effect.78 An operational effect is achieved by combining the eight Fundamental Inputs to Capability (FIC): organisation; personnel; collective training; supplies; facilities; major systems; support and command; and management.79

2.50  In acquiring Defence platforms and systems, a range of documentation (including capability definition documents, operational concept documents, functional and performance specification and test concept documents) is developed and sets out the detailed requirements/performance attributes to be achieved. In the case of an aircraft, for example, this would include elements such as its range and speed, handling characteristics, self protection abilities, requirements for runway length and other ground support, civil and military communications and guidance systems, maximum takeoff weights, cargo capacity, maintenance requirements and compliance with military and civil certification regulations and requirements.

2.51  Depending on the nature of the information, it can be classified for national security reasons and therefore may not be publicly available.

2.52  The fourth snapshot, Figure 15, sets out the DMO's assessment of the likelihood of delivering all of the key capability expected to be delivered and the project maturity score progress.80 For 27 of the 28 projects, the DMO's assessment of the likelihood of delivering capability is against the Materiel Release Milestones (MRMs) and Completion Criteria specified in each project's MAA. The exception being Wedgetail, where the assessment has been against the Supplies section of the MAA, which lists the equipment to be delivered. The project maturity score is based on the project manager's judgement at defined milestones in the project's capability development and acquisition phases.81

2.53  The complexity and type of acquisition (MOTS, AMOTS or Developmental) is a significant factor affecting the likelihood of delivering all key capabilities; the risk is appreciably higher for more developmental projects. The DMO's assessment of the likelihood of any project delivering all the key capability requirements should become better informed as a project matures along its capability development and acquisition lifecycle.

2.54  The DMO's key capability measures should be interpreted with some caution due to their lack of rigour as a data system and the high level of uncertainty in forecasting outcomes.

2.55  Overall, the DMO's assessment is that 25 of the 28 projects with key capability data in this year's MPR will deliver all their key capability requirements. The projects that are not expected to deliver all their key capability requirements are Wedgetail, MRH90 Helicopters and 155mm Howitzer.

Figure 12
Project snapshot-Key capabilities expected to be delivered and project maturity progress (percentage)

2.56  During 2010-11, 26 of the 28 Major Projects transitioned to a new MAA.82 Each of the new MAAs was signed by a delegate of the Chief Capability Development Group (as the purchaser), Chief Executive Officer (CEO) of the DMO (as the supplier), and the relevant Capability Manager (as the customer).

2.57  One change introduced by the new MAA was the inclusion of MRMs and Completion Criteria, which replaced Measures of Effectiveness (MOEs), and were designed to overcome deficiencies perceived in the MOE framework. However, the DMO has acknowledged that for one project the MRMs and Completion Criteria were poorly designed and as a result, the DMO has reverted to the MOEs framework as the capability measure for this report.83

2.58  The MRMs and Completion Criteria are designed to define at a strategic level the key elements required to achieve Initial Materiel Release (IMR) and Final Materiel Release (FMR).84

2.59  In general, the MRMs and Completion Criteria focus on the achievement of certain technical, regulatory or operational requirements. Where key requirements are not achieved, this could be expected to have a significant effect on a system's likely suitability for acceptance into operational service.

2.60  In the 2009-10 MPR, the ANAO observed that there was considerable diversity across the projects in the number, level of specification and focus for MOEs. This observation is maintained for the MRMs and Completion Criteria. The ANAO continues to observe that there is not a clear underlying consistency in the identification and articulation of the MRMs and Completion Criteria in the MAAs. For example, AWD Ships currently has four MRMs (IMR-Ship 1, MR 2-Ship 2, MR 3-Ship 3 and FMR), while 155mm Howitzer reports on 23 MRMs.

2.61  The MRMs and Completion Criteria are assessed using traffic light indicators by the DMO's project management, and reported monthly within the DMO and the Defence Organisation. The June 2011 traffic light assessment is represented in the PDSS as Measures of Materiel Capability Performance.

2.62  As the Measures of Materiel Capability Performance's assessment concerns forecasting future achievements, it has been excluded from the scope of the ANAO's formal review.85 However, the ANAO has been provided with data from the DMO's reporting systems in order to examine the accuracy of the disclosure in the PDSSs.

2.63  Due to national security considerations, only the overall status from each project's assessment of the likelihood of delivering the required materiel capability is disclosed in the 2010-11 MPR and earlier reports. Figures 16 and 17 present the DMO's assessment of the percentage of materiel capability that:

•  has a high level of confidence will be delivered (green);

•  are under threat but still considered manageable (amber); and

•  at this stage are unlikely to be met (red).

2.64  For example, Figure 14 shows the assessment for MRH90 Helicopters as at 30 June 2011, as having a high level of confidence in delivering 60 per cent of the materiel capability, while 20 per cent of the materiel capability is under threat but still considered manageable, and 20 per cent of the materiel capability at this stage is unlikely to be met. This is a reduction in the expected capability to be delivered from the 2009-10 MPR.

__________________________________________________________________________________

78  Department of Defence, Defence Capability Development Handbook 2011, p. 2.

79  Defence Instructions (General) OPS 45-2, Capability Acceptance into Operational Service, paragraph 1.

80  Source 1: As the DMO's assessment of the likelihood of delivering key capabilities involves high levels of uncertainty which may cause actual outcomes to differ materially from that stated in the PDSSs, this data and the DMO's assessment is outside the scope of the ANAO's assurance review for the 2010-11 MPR.

Source 2: The DMO have advised that the project maturity score concept was not designed for a strictly linear representation of a project's progress. However, the ANAO's analysis has utilised a linear representation for comparative purposes in Figures 3, 8, and 12.

81  This score can then be compared against an ideal or benchmark score for that milestone to indicate the project's relative performance.

82  The MAA for Bushmaster Vehicles was signed during July 2011 and August 2011. Hornet Refurb was not required to transition to the new MAA due to its expected completion date.

83  DMO have used Section 4 of Wedgetail's MAA (Supplies) to assess Section 4 (Measures of Materiel Capability Performance) of the project's PDSS.

84  IMR is achieved on the delivery of the DMO Acquisition Project supplies that have been identified within the MAA as required to support the achievement of Initial Operational Release (IOR). FMR is achieved on the delivery of all the DMO Acquisition Project supplies specified within the MAA to support the achievement of Final Operational Capability (FOC). Defence Materiel Instruction (PROJ) 11-0-001, Management of DMO Acquisition Project Schedules, Table 1.

85  See paragraph 1.9 for PDSS items out of scope for the ANAO's assurance review.