F.1. Data sources: CMIS + interviews
A number of data sources have been drawn on to build a comprehensive view of project performance. From within DE&S these consist of CMIS data, IPT interviews and Initial and Main Gate Business Cases submitted to the Investment Approvals Board. In addition, the National Audit Office (NAO) produces an annual report containing data on the top 20 major MoD procurement projects.
The DE&S Corporate Management Information System (CMIS) is designed to allow DE&S to manage performance, cost and time data. CMIS is currently used by DE&S to capture information on Equipment Projects, Other Projects, Business Milestones and DE&S Submissions to IAB (Figure F-1).
Figure F-1: DE&S Corporate Management Information System (CMIS)
Since project performance is being examined, the "Equipments Projects" data has been the starting point and predominant data source for most analysis. CMIS was introduced 1st April 2004, at which point it became mandatory for all Category A-C equipment projects past Initial Gate or Main Gate to be tracked in CMIS.
CMIS is a web based tool that can be accessed by all and contributed to by designated users within DE&S. It is mandatory, and the responsibility of the relevant IPT, to record all project approvals in CMIS and to enter latest project time and cost forecasts on a monthly basis. The CMIS extract used was as of 1st February 2009 and consisted of 165 equipment projects, of which 117 were past Main Gate and the remaining 48 projects still in Assessment phase. Furthermore 42 out of 165 projects have been archived following either completion or cancellation.
Figure F-2: CMIS web interface
If the project is in Assessment phase forecast Main Gate date, In-Service Date (ISD), Assessment phase cost and Demonstration and Manufacture (D&M) Cost are tracked. Projects past MG record latest forecast ISD and D&M cost. Projects remain in CMIS for as long as they appear in the EPP but even once archived their historical data can still be extracted from the system.
The causes of any variations in cost and time forecasts must also be recorded and they can be categorised as shown in Table F-1. Guidelines are provided to explain the categorisations further, but responsibility remains with the IPT to ensure that variations are correctly classified.
| Category | Variations due to changes: | Impact |
Technical | Technical factors | Technical ability to deliver the project | Time and cost |
Customer requirement | Changed requirement | Customer's requirement for the equipment, flowing from operational reassessment rather than budgetary priority | Time and cost |
Changed budgetary priorities | Customer's requirement for equipment, flowing from changed budgetary priorities | Time and cost | |
Economic conditions | Inflation | Inflation assumptions | Cost |
Exchange rate | Exchange rate assumptions | Cost | |
Management | Receipts | Expectations of receipts, e.g., liquidated damages, commercial exploitation levy | Cost |
Contracting process | Includes time taken in contract negotiations and placing contracts, international contract negotiations and effect of contractor bids compared to estimates | Time and cost | |
Procurement strategy | Impact of overall procurement strategy (e.g., change to collaborative options), or from competitive to single source | Time and cost | |
Reporting conventions | Accounting adjustments | Reflecting no substantive change, including imported or exported costs arising from changes to accounting rules and adjustments to reflect changes in the definition of term | Time and cost |
| Risk differential | Amount of risk allowed for in the approval - difference between the 50% 'most likely' estimate and the 90% 'not to exceed' estimate at the point of Main Gate approval | Time and cost |
Associated projects | Change in associated project | Associated project changes, e.g., availability of equipment from another project for trials | Time and cost |
Table F-1 : CMIS forecast variations categories
In addition, there are the following non-mandatory tabs that contain useful data: Contractor, KURs and UPC/Quantity.
As part of the review process an extensive programme of IPT interviews were carried out and in support of this IPTs were asked to complete a data
template. This has been valuable in expanding the sample of projects for which unit data is available at Initial Gate, Main Gate and Latest forecast. It has also proved to be a key source of TRL (Technical Readiness Level) data.
The final source of data from within the organisation has come from reviewing Initial Gate and Main Gate Business Case submissions to the IAB. The business cases are able to provide both IG and MG forecast data and add to the unit and TRL data. They also provide a very good insight into the project scope and help to understand any significant changes that might have occurred between IG and MG