3.8 Pathway told us that the emergence of complexity in the service requirement after the contract was signed caused them acute problems in two main areas:
■ temporary tokens - the arrangements that apply when someone collects a benefit (prior to issue of a Card) or on behalf of another person at short notice on a temporary basis; and,
■ extended verification processes - through which the Department could change the targeting of identity checks on particular groups of payees. Payees were asked to verify their identity at the counter by giving personal details in answer to computer-generated questions when presenting their payment card. This important anti-fraud feature is explained further in Figure 13.
Figure 13 |
|
A case study in emerging complexity: Arrangements for verifying the customer's identity | |
| |
Source: Project documents reviewed by the National Audit Office | |
Pathway felt that complexity lay not so much in each individual area but in the way they combined across the different types of benefit recipient (normal beneficiaries, permanent agents, temporary agents, and alternative payees). Different rules for different benefits needed to be reconciled. Complexity thus related to a small minority of payment collections (estimated to be between one and five per cent), often where agents collected more than one benefit. Pathway told us that the Department appeared inflexible in simplifying the application of rules to lessen the complexity of the resulting system. We noted that the Department had attempted simplification, but were constrained by legislation and by the need to meet benefit recipients' expectations in this difficult area, (paragraph 3.11).
3.9 In accordance with Private Finance practice, the Department's part of the requirement was specified mainly in terms of the performance they required, rather than how the service was to be delivered technically. Pathway, like the other bidders, responded with their own technical solutions. Tokens and extended verification processes were elements of Pathway's proposed solution. At the time the contract was signed there was not a detailed design that demonstrated how the solutions in each of these areas would work in practice. Pathway's position is that their solution was robust enough at that time to be developed to meet the full requirement. This was also the view of the Department.
3.10 The Department gave us their own perspective of how complexity emerged in areas identified by Pathway. On the issue of verification procedures, the Department felt that they had sought to preserve their requirement for flexibility to target verification procedures in response to changing patterns of fraud. They had agreed to Pathway developing initial software using "hard coding", but had required the full version to be written using "soft-coding", to ensure verification questions could be targeted effectively in practice.
3.11 Pathway also noted that at the time the contract was signed there was no authoritative document which collated and reconciled all the different rules which governed payment of different benefits. The rules differed between benefits partly for historic reasons, and partly because the differing circumstances of different types of claimants determined the extent of their reliance on emergency payments and on agents to collect benefit on their behalf. This proved to be a very difficult area to resolve. It required the Department and Pathway to identify and harmonise historically divergent payment practices between different benefits, tracing these back to often conflicting regulations and legislation, to produce a single set of rules that could be programmed into the Card system. Pathway produced the first version of the new document which documented these rules in 1996 and submitted it to the Department for agreement. Five subsequent iterations followed, during which time the Department examined at length the need to change legislation, which proved to be difficult. The document was not finally agreed until February 1999, just before the Card project was cancelled. Pathway continued designing and redesigning software while this reiteration continued. The Department told us that they had provided a statement of rules they had considered definitive as early as October 1996, and had worked hard with Pathway on the correct interpretation of these rules.
3.12 Pathway told us that they had expected more scope to redesign the Department's business rules and requirements so that they could use software more closely based on the solution that they had installed in the initial ten post offices in September 1996. This initial go-live was always intended to be an interim system. It did not offer full functionality, and included some extemporised processes which would have to be replaced with permanent software for the subsequent roll-out to 200 post offices. It was a development based on software running live in post offices in Ireland. The purchasers had, during the procurement phase of the project, identified as an issue the extent to which Pathway would have to change software running on the system in Ireland.
3.13 The project initially proceeded on the basis of proposals from Pathway that it would involve mainly the integration of existing software packages. In the event, the greater than expected complexity of the service requirement obliged Pathway to undertake much more development of new software than they had planned for in their winning bid. This had major implications for the degree of difficulty of the project since software development is a complex and high-risk activity, especially against a requirement specified largely at high level. The most comprehensive, published research known to us in this area is that undertaken by the Standish Group into the outcome of software development projects in the United States. The key findings are shown in Figure 14.
The outcome of software | Figure 14 |
|
| ||
| Project failure rates in software development projects across private and public sectors are high, and more severe still in larger organisations | |
|
| In small | In medium | In large |
| Project successful: Completed on time, on-budget, with all features and functions as initially specified | 28 | 16 | 9 |
| Project "challenged": Completed and operational but over budget, or over the time estimate, and offers fewer features and functions than originally specified | 50 | 47 | 62 |
| Project cancelled: at some point in the development cycle | 22 | 37 | 29 |
| Notes: 1. The Standish Group research also reported that for projects that were "challenged" or cancelled, average time overruns were 230 per cent in large companies and the average cost overrun was 189 per cent of the original cost budget. 2. For the purpose of this survey, large companies were defined as those having an annual turnover greater than $500 million, and a medium-sized company as those with turnover of $200 million to $500 million. Source: The Standish Group, based on 1995 survey responses from 365 IT executive managers in the USA across major industry segments including finance, manufacturing, retail, services and local, state and federal government institutions. Some 8380 development projects were represented. | |||
3.14 An increased need for bespoke software development also places additional demands on a supplier's resources. ICL is primarily a software integration company and most of its staff work in this area. Department of Social Security officials told us that they felt Pathway had had difficulties mobilising and managing sufficient resources to deliver the revised solution. This was supported by the conclusions of a review in 1997 by PA Consulting (Figure 19), which concluded that Pathway did not initially have the resources in place to complete the development work required. The Department understood that the full range of software to be integrated was not delivered to Pathway by its sub-contractors until late 1997. Pathway's position is that the speed of this mobilisation was never a cause of project delay and that their system integration staff had the necessary skills for system development and testing.