Design, development and testing

2.24  The Agency's operational imperative to start the roll-out to offices in October 1998 required all parties to deliver the new systems on time. Working to an agreed overall project plan, the contractors were each responsible for designing and testing their own systems. Apart from some initial difficulties with on-site printers in Liverpool, there were no significant difficulties with the development and implementation of the new secure printing system; the rest of this section therefore focuses on the new computer system developed by Siemens in support of the initial processing and examination of applications.

2.25  Both the design and development stages, which overlapped to some extent, took longer than expected. The design stage was completed some four months later than expected and only six months before the 'go live' date (Figure 15). During the design phase, Siemens had to gain a detailed understanding of the Agency's existing processes and to put forward a solution that would meet the Agency's requirements. Discussions between the Agency and Siemens took longer than expected, as each party sought to clarify the other's intentions and to reach agreement on the system's specifications, for example to enable the system to retain electronic images of application forms, and to capture additional information from forms such as personal details of the counter-signatories

appearing on passport applications. The design phase was on evolutionary process with several versions of design documents as the new system was defined and redefined by the Agency and Siemens. This process also prolonged the subsequent development.

2.26  As a result of delays at the design and development stages, testing started almost four months later than planned. Throughout the test phase, the Agency, advised by consultants from the Central Computer and Telecommunications Agency, had access to test records kept by the contractors, and representatives from the Agency witnessed most of the tests.

2.27  The first stage of factory-based testing, which involved testing the new software, took place at Siemens' testing centre in July 1998. Staff from the Agency were brought to help test the various elements of the software. Further testing, including on-site testing in Liverpool, took place in August and September. By the time the system went live in October most of the system's core functions had been delivered on schedule, although some parts of the system, for example management reports, were not fully developed. Taken as a whole, the Central Computer and Telecommunications Agency advised that the system had been well-designed. In its view, the Liverpool office should go live as a "controlled pilot site" although there might be a need to extend the pilot because reliability had been a problem during testing.

2.28  The Agency's internal audit service, supplied by Home Office Internal Audit, examined the introduction of the new system during September and early October and concluded that: "Given the tight timescales under which the project has been developed it has been well managed with sound controls in place to ensure the quality of the end products. However, there has been a compression of testing timescales…… (We) understand the compelling reasons for wanting to meet the 5 October 1998 pilot start date and acknowledge that the functionality of the (new) system was largely proven. However, the risk is that the pilot may need to have its timescales extended to ensure that robust operational and user procedures and controls are in place."

2.29  Due to time pressures, the test programme did not extend to thorough testing of the system's impact on productivity. During the factory acceptance tests conducted at Siemens' testing centre in July 1998, the Agency had become aware that the examination stage of passport processing was taking much longer under the new system. The Agency had intended to conduct further tests on productivity on-site at Liverpool in September prior to going live but, although terminals had been set aside for this to happen, the tests did not take place due to shortage of time.

2.30  The clerical procedures that would support the new system had not been tested and had not been fully documented by the time the training courses for the Agency's staff at Liverpool started in September. All Agency staff were given basic training by the Agency, together with further training specific to the role which they would be undertaking. Siemens trained the Agency's trainers and prepared the course material relating to the use of the computer system. Siemens took responsibility for training those staff who were due to transfer to it. A review of training for Agency staff, conducted by the Agency in October and based on feedback from staff, found that the courses had been well delivered and had met their objectives. However, the review had noted that the training objectives had related primarily to the use of the new computer, and less to the clerical procedures needed to support the new issuing system.

The Agency's roll-out timetable was short and left it with little room for manoeuvre once the problems emerged at Liverpool. The roll-out to Newport took place despite the continuing problems at the Liverpool office.

2.31  The Agency's implementation plan (Figure 16) envisaged a staged roll-out of the new system between October 1998 and February 1999, starting with Liverpool and then Newport. The Agency's aim was to install the new system before the seasonal rise in applications early in the New Year. At each office, existing computer systems would be removed; building works undertaken where necessary and office layouts re-configured, with space set aside for Siemens' staff; the new system installed and tested; and staff trained. The Agency expected each office to run down its existing workload prior to its roll-out date and transfer its incoming work to other offices that would be working normally. The aim was to maintain normal service to the public.

2.32  The roll-out at Liverpool started on time, on 5 October. The Agency's timetable left it with little time to make a final decision about the roll-out to Newport, which was due to take place on 16 November. Preparations in the Newport office, including reorganisation of the building, had begun before Liverpool started live running, and a final decision on Newport had to be taken within five weeks of Liverpool going live.

2.33  The Agency's implementation plan originally envisaged that roll-out to the other regional offices would only take place once the Liverpool office had achieved a capability of 30,000 issues per week - its normal output under the old system. The Agency's plan had assumed this target would be achieved within six weeks of going live, that is by 16 November.

2.34  Output at Liverpool did not pick up as quickly as expected. There were two principal bottlenecks:

■ Siemens' method of capturing information by scanning application forms, followed by correction by its operators, was contributing to delay. The Agency had been given advance warning of this, firstly during the evaluation of the bids when a consultant commissioned by the Agency expressed concern that there would be a "high error rate in reading hand-written letters (and) numbers…(which) would lead to a high correction level and all the resulting problems". The trials of the scanning equipment, using application forms completed by the public specifically for the trial, had also indicated that there might be problems although Siemens and the Agency thought this was because the forms had not been completed with the same care as real applications. Another problem, affecting a small percentage of applications, was the variability in the ink density and print positioning on the application forms, designed by the Agency in conjunction with Siemens and Security Printing & Systems Ltd, which caused difficulties for the scanning equipment.

■  A loss of productivity at the examination stage due to examiners' lack of familiarity with the system, the need to correct errors in the data scanned and transferred by Siemens in a small percentage of cases and the large number of key-strokes and on-screen operations to be completed before a decision on eligibility for a passport could be taken and recorded. This stage, which was designed by Siemens according to specifications set by the Agency, records the checks carried out by examiners, thereby providing a more secure and accountable examination process.

2.35  The other main factors that lay behind the failure of the Liverpool office to meet output targets were:

■  Delays in providing a counter service to personal callers at the Liverpool office - urgent applications are normally dealt with by passports printed on site. The new on-site printers provided by Security Printing & Systems Ltd initially did not work as well as expected. In addition, Siemens were not always able to respond to urgent scanning requests as quickly as the Agency wanted.

■  There was insufficient support available to the Agency's examining staff to answer their queries about the new system and working arrangements.

2.36  By 14 October, senior managers within the Agency judged that the 30,000 target for Liverpool would not be deliverable within the six weeks. At this point, the roll-outs to Newport and Peterborough were at risk of being delayed, the latter possibly well beyond Christmas. The Agency therefore decided to revise its roll-out criterion - it decided that the roll-out to Newport should proceed as long as there were no outstanding serious problems and that output in Liverpool was likely to be near 10,000 at the four week point and 20,000 after six weeks.

2.37  The final decision to roll-out to Newport was taken by the Agency's Management Board on 9 November. Output at Liverpool had reached 6,200, the latest figures available, compared to the lower target of 10,000. At this stage, the Agency had not had sufficient time to judge the rate at which output might improve; although output in Liverpool was far below the expected level, it had shown some improvement (Figure 17).

2.38  Despite some serious reservations, the Board nonetheless decided to roll-out to Newport on schedule. The Board felt that the Liverpool Office had not been fully prepared for the scale of the change it had undergone and that the Agency had not been fully prepared for its relationship with Siemens. The Board was concerned about Siemens' ability to deliver the required output in Liverpool and had serious doubts about whether the Agency would have the ability to carry on with the roll-out programme without adversely affecting service to applicants. However, the on-site testing at Newport had gone well and the Agency believed that there were a number of arguments in favour of pressing ahead:

■  The Newport office had completed almost all the tasks required of it before going live, the accommodation had been reconfigured and re-cabled, and staff had been trained - to revert to the old system at this stage would, in the Agency's judgement, not have been easy and would have taken some time.

■  The Agency wished to test the new system on more than one site, in particular to test the central printing facility at higher volumes.

■  Delay at this stage, only five weeks into the roll-out programme, would almost certainly have meant delaying the remainder of the roll-out programme until the following Autumn. This had significant cost and operational implications, including the extra cost of maintaining the old system for another year and of making it millennium compliant.

■  The Agency's contract with Security Printing & Systems Ltd had guaranteed a minimum volume of two million of the new digital passports a year. The Liverpool office alone would have generated only half of this total, and if Newport did not go ahead the Agency might have been liable to pay for the full two million passports - an additional cost of over £3 million.

Output at the Liverpool and Newport offices remained below expectations until June 1999. If the planned output at these offices had been achieved according to the original timetable, the Agency would have issued around 400,000 additional passports, enough to avoid the serious backlogs that occurred.

2.39  On 18 November, as a result of experience at Liverpool and Newport the Project Steering Committee chaired by the Agency's Chief Executive decided to defer the roll-out to Peterborough, due to take place on 7 December, until revised performance criteria for the two pilot sites had been met. The revised criteria included a requirement to prove a minimum output of 30,000 issues per week from both Liverpool and Newport and a need for the systems to have run for five consecutive working days without a failure involving loss of service.

2.40  Output at Liverpool and Newport continued to fall short of expectations for some time. Overall, it took nearer six months than six weeks to attain previous levels of output in the two offices, although output at Newport picked up more quickly. Greater experience of using the system, improved working procedures, extra staff and overtime working at both pilot sites helped to increase outputs during the Summer period to levels comparable to those achieved by the previous system. Liverpool and Newport met the target output of 30,000 each in June 1999 (Figure 18).

2.41  We estimate that if output in the first half of 1999 at Liverpool and Newport had been at the 30,000 per week anticipated in the Agency's 1998-99 business plan, the Agency's backlog of applications would have been reduced to around 100,000-except for an increase in June to nearly 200,000-comparedtothe actual backlog of 565,000 (Figure 19). In total, around 400,000 more passports would have been issued. This calculation assumes the Agency received the same monthly profile of applications it received in the first half of 1999 and that the Agency's other offices performed at the levels they actually achieved during this period.