2.62 On completion of a project, the Inland Revenue business unit tests whether the information technology system developed by EDS meets its requirement and, if it does, takes ownership of the system for live running. As part of this process, it should agree key criteria with EDS for evaluating the system's performance. Where a project is an enhancement of an existing system, the existing key performance criteria should be amended. Thereafter EDS should provide the Inland Revenue Contract Management Team with monthly data on performance of the live system against the agreed criteria. If the agreed performance criteria are not met, this may trigger financial remedies.
2.63 Project managers are also required to conduct a project evaluation review at the conclusion of each project. Its purpose is to confirm whether the original project objectives have been achieved and whether the customer has taken delivery of the final product. It should evaluate the effectiveness of the project management, provide assurance that all project documentation is complete and up to date, and set out lessons learnt, including any issues outstanding.
2.64 In some cases, the project's sponsor, usually a senior manager, may decide to commission an independent post implementation review three to six months after implementation or, if a full year's operating data are required, up to 24 months after implementation. Such reviews build on the project evaluation review and are conducted only where there is a clear likelihood of useful knowledge being gathered. Their key objectives are to establish actual and estimated outcomes, for example costs and performance, identify outstanding issues and proposals for their resolution, and to communicate the key lessons learnt from the project.
2.65 Figure 10 illustrates the extent to which the Department has evaluated the projects in the sample which had been completed at the time of our review.
Extent of project completion reporting on sample projects |
|
|
|
|
| |
Figure 10 |
|
|
|
| ||
|
|
|
|
| ||
| Project evaluation report | Post implementation review | Key performance criteria | |||
| Computerised environment for self assessment - release 2 | Yes | Yes1 | Yes | ||
| Payroll and personnel management information system | Yes | No2 | No | ||
| Business continuity implementation | Not due | Not due | Yes | ||
| Integrated repayments system | Yes | Yes | Yes | ||
| Repayment of interest on PAYE | Yes | No | Yes | ||
| RPC rationalisation - Phase 2 | Yes | No | Yes | ||
Source: Lorien analysis of Inland Revenue data | Notes: | 1. A full post implementation review of CESA was carried out following completion of the whole project, including release 2. 2. The post implementation review of PPMIS will start in Spring 2000. It was delayed because not all of the functionality of the project was initially available. | ||||
2.66 Our examination of project evaluation reports identified a number of weaknesses in approach.
■ While project evaluation reports summarised the outcome of a project, they varied in quality and did not, for example, include comparisons of estimated and actual costs and the functionality delivered compared with the original specification.
■ The reports did not analyse what had gone well and what had gone less well, so that lessons could be learnt for the future.
■ In five of the six completed projects, the Department had agreed key criteria with EDS for judging the performance of the system. In one case, however, the payroll and personnel management information system, no key performance criteria had been established some 18 months after implementation. The system, a commercial software package, was selected by the Department, not by EDS; EDS did not take responsibility for the information technology aspects of the project until after it had started.
■ In one project where the Department had carried out a post implementation review, the integrated repayments system, it had not quantified the extent to which the project had delivered the anticipated staff and monetary savings, despite the fact that the system was delivered in April 1997.
2.67 The Department told us that it now had a clear process for project evaluation reports and post implementation reviews, documented in a Project Reviews Guide. The process was overseen by a formal review board. This board ensured that:
■ reviews were planned in advance and terms of reference were agreed;
■ reviews were completed to an acceptable standard; and
■ evaluation reports were analysed and lessons learned were taken into account for the future.
Since May 1999, 15 project evaluation reports and four post implementation reviews had been reviewed by the board and, in addition to this, a further ten project evaluation reports and five post implementation reviews were imminent or already under way.
2.68 The Inland Revenue's risk management plan has identified the need to define key performance criteria and keep them up to date. The Department told us that although it was not always possible to define these at the outset, it was taking action to review all major new developments which had gone live over the past year to ensure that key performance criteria were in place. It would also be reviewing all services due to go live in the next twelve months to ensure that such criteria were developed.