7.8.3. High development risk solutions

It has previously been observed (see Chapter 6) that the Department is incentivised to over-specify equipment in order to ensure that sufficient capability is eventually delivered despite funding constraints. Furthermore, alternative acquisition techniques which have the potential to mitigate the consequences of high technical risk (e.g., spiral / incremental / sub-system acquisition techniques) are militated against due to the relatively uncertain long-term funding environment (i.e., uncertainty over the eventual release of money for the scheduled upgrade programme).

As a result, in order to make the capability leap from "old" to "new" equipment and deliver a technical advantage, the technical risk of equipment projects embarked upon by the Department is generally very high, frequently alongside significant overall system integration risks.

The challenge that technical development presents is often not recognised in advance and is compounded by the relatively poor active management of risk during the project. As a result, at the point of Main Gate (or even contract award) the level of risk is not always fully recognised within the cost estimates.

Additionally, as is now widely recognised, the MoD is unable to divest itself fully of development risk through contract structuring because:

• either the contractors recognise the risk and the price escalates to unaffordable levels as a result; and / or

• the MoD fundamentally requires the capability and will pay to fund its development whatever the contract structure notionally in place.

The perception amongst interviewees was that the MoD endeavours to contract for procurement of equipment whilst very significant levels of technical risk remain to be explored / quantified. Some characterised this as embedding technology development within the core project, rather than before it (as is the intention of the CADMID process). The consequence of inadequate recognition of technical risk is that the duration / cost estimates established at Main Gate are generally proven to be optimistic.

The Department has an established way to benchmark the technology and system integration risk for each project: Technology Readiness Levels ("TRLs") and System Readiness Levels ("SRLs"). The TRL assessment system was originally devised by NASA, with the intention of reducing overall project risk and the likelihood of project delay due to immature technology. The system operates a 9 level grading system reflecting the technical maturity (or conversely risk) associated with a system. For further information see Appendix F.

TRLs are usually applied to individual subsystems of a project (e.g., software protocols within a hardware platform). To supplement TRLs, the MoD has devised the SRL scale which has been used on some projects to cover risks associated with integrating individual subsystems.

Some business cases going to IAB for approval provide TRL and SRL estimates. Projects seeking Main Gate approval from the IAB are recommended to have a TRL of 7, although it should be noted that a number of projects have been approved despite incorporating less mature technologies (i.e., lower TRL scores). In practice, the Review team had difficulty finding evidence for any widespread use of TRL/SRLs: fewer than 15 instances were identified across the mature project sample.

Where TRL is available, analysis has been undertaken to understand the impact of TRL on time / cost overrun (see Figure 7-15).

Forecast duration vs. TRL

Note: Projects more than 75% complete at latest forecast

Source: CMIS (Feb 2009); IPT Interviews; IAB; Review team analysis

Figure 7-15: Demonstration and Manufacture phase duration overrun vs. TRL at Main Gate

These data show that although there is a relatively weak link between TRL assessed at Main Gate and slip versus Main Gate duration estimates thereafter, projects where TRL was assessed at 8 or 9 appear to perform significantly better than those with TRLs of 7 or below.