The purchasers decided to accept the lower confidence of delivery that they had in Pathway's proposed solution compared to other bidders', because only Pathway's bid was clearly compliant with the Private Finance Initiative
3.33 The purchasers evaluated bidders' technical proposals on a scale of zero to ten against the eleven criteria shown in Figure 15. Zero represented an unacceptable proposal with serious shortcomings and ten an outstanding proposal. Most scores under each criteria were in the range four to six, where four indicated that the proposal was viable but raised some outstanding issues that were relatively minor or manageable; and six that the proposal was sound with no significant issues. The evaluation team consistently assessed Pathway's proposal as being less strong overall than those of the other two bidders under most of the eleven criteria.
3.34 The technical evaluators' overall assessment of the credibility and viability of the Pathway solution was that while acceptable, they ranked behind the other two consortia. The final evaluation identified and confirmed a significant number of risks inherent in the Pathway solution, many of which were potentially high probability and high impact risks and which were factors in the later problems in the project.
3.35 Pathway's proposed solution had some attractions because it was based on a simpler benefits payment system already working with the Post Office in the Republic of Ireland. The solution featured a "distributed architecture" in which each post office held its own database of payments due. In principle this offered advantages of improved availability and shorter transaction times, an important factor for Post Office Counters Ltd. It contrasted with the other two bidders' proposals which were based on centralised databases, as used by UK banks for credit card systems. Neither of these bidders had already developed a working social security benefit payments system. The risks associated with the Pathway architecture were seen to be in terms of scaling up of the system, the software required to support it and the changes needed to cope with the full service requirement.
3.36 The purchasers' evaluation team reported their strong concerns about the reliance on unproven third party software to support Pathway's solution for post office automation and the benefits payment card system. This matter, they said, generated little confidence in Pathway's ability to deliver to time and quality and left a question mark over the delivery of the service.
Figure 15 |
|
How the purchasers assessed the bidders' proposals in April 1996 | |
Pathway's solution was ranked behind its competitors' in terms of delivery, but was deemed acceptable and was the only one that was clearly compliant with the Private Finance Initiative. The lowest evaluation in each of the criteria is shaded. | |
Criteria | Weighting, | Cardlink | IBM | Pathway |
Price (£ million)1 |
| 1257 - Third | 1026 - Second | 1005 - First |
|
|
|
|
|
Compliance with the Private Finance Initiative (See Figure 16) |
| Probably compliant | Not compliant | Compliant |
| ||||
External factors affecting potential success2 |
|
|
|
|
Acceptability of the system to the customers of the Department of Social Security and post offices. | 30 | 5.8 | 5.3 | 5.3 |
|
| |||
Acceptability of the system to Department of Social Security & post office staff and agents. | 15 | 6.0 | 5.0 | 5.5 |
| ||||
Flexibility, in terms of reacting to external change. | 20 | 4.7 | 4.2 | 4.4 |
| ||||
Reliability and support in terms of helping ensure continuity of service. | 20 | 5.3 | 5.2 | 4.5 |
| ||||
Innovation | 15 | 5.3 | 5.8 | 5.4 |
| ||||
Viability factors, in terms of service delivery2 |
|
|
|
|
Fraud free method of payment for benefits payments. | 30 | 5.5 | 5.4 | 4.0 |
| ||||
Credibility to manage and deliver the start up of services. | 15 | 5.4 | 5.1 | 4.4 |
| ||||
Credibility of service delivery once implemented. | 20 | 5.7 | 5.5 | 4.1 |
| ||||
Management capability of the bidding team shown in the demonstrator phase. | 20 | 4.7 | 4.5 | 3.7 |
| ||||
Stability and coherence of the bidding organisation, including the prime contractor and other consortium members. | 15 | 6.2 | 6.0 | 4.4 |
| ||||
Notes: 1. Price is shown as Net Present Value, discounted at six per cent per annum, for the duration of the contract. The values given are for the most likely outcome. Total prices could be higher or lower depending mainly on the volume of transactions and on inflation over the life of the contract 2. In this scoring system a score of "4" meant "Ordinary" - that the proposal was viable but nothing special, and raised some outstanding issues that were considered relatively minor or manageable; and "6" was "good" - that the proposal was a basically sound solution with no significant issues. Source: National Audit Office | ||||
3.37 Foreseeing the areas of difficulty that were to arise after the contract was signed, the evaluation team reported that they had a low belief in Pathway's appreciation of, and empathy with, either of the purchasers' business requirements or ability to deliver a service which would meet the required service levels. They reported that they considered that Pathway had weak authentication procedures, the weakest security proposals and that Pathway had manifest lack of understanding of the management of fraud risk.
3.38 The evaluation team reported that Pathway's performance in managing the demonstration phase of the procurement had been substantially below those of the other two service providers and raised concerns about Pathway's lack of internal controls. They reported that Pathway had not shown itself to be adept at foreseeing and preventing problems.
3.39 The results of the financial evaluation of bids showed that the Cardlink consortium's bid was the most expensive. The IBM and Pathway prices were very close together.
Figure 16 |
|
How the purchasers assessed the bidders' proposals for compliance with the Private Finance Initiative | |
Only Pathway's bid offered the extent of risk transfer that would ensure compliance with the Private Finance Initiative and enable the investment in the system to be off the public sector's balance sheet. | |
The risks the purchasers wished to transfer to the supplier | Cardlink's proposal | IBM's proposal | Pathway's proposal |
Liability for up to £200 million of fraud over the life of the contract, should it continue | Onus of proof on the purchasers, no acceptance of cardholder verification fraud | Liability limited to £10 million a year, no acceptance of cardholder verification fraud1 |
|
That the system would be delayed or not work |
|
|
|
That the volume of transactions would be lower than expected | guaranteed volumes to be set at end of first year of roll-out | 92 per cent of revenue to be guaranteed |
|
That inflation would increase the supplier's costs | Charges to increase at inflation less 1 per cent | Charges to increase in line with inflation |
|
That the system would not be operated to agreed standards | Financial limits of liability set per year and per event |
|
|
Note: 1. The purchasers estimated the risks accepted by Pathway in the area of fraud at a value to them some £34 million higher than the risks accepted by IBM. Source: National Audit Office assessment of the purchasers' bid evaluation | |||
3.40 Whilst acknowledging the implications of selecting Pathway, the evaluation team considered this preferable to awarding the contract to IBM. IBM's bid would be unlikely to be PFI compliant and accepted less risk than Pathway's in terms of having to pay penalties if they failed to prevent fraud. The group therefore unanimously recommended to the Evaluation Board that any contract award should be made to Pathway. On 29th April 1996 the Project Board accepted that to proceed with Pathway implied a degree of risk but concluded that such risks were manageable and were outweighed by the relative merits of other aspects of their bid.