4.11 The Department encountered difficulties in obtaining commercially sensitive price and cost information from other organisations. This experience is not unusual and reflects the fact that quantitative benchmarking often requires:
■ access, on an anonymous basis if required, to data sources;
■ facilities for secure storage and handling of the data;
■ knowledge of the quality and meaning of the data and the skills and experience to manipulate and interpret the data;
■ the ability to demonstrate these attributes to potential providers of comparative data.
4.12 We consider that it is unlikely that these requirements can be met economically, if at all, by the Department. There are, however, commercial research organisations which specialise in providing benchmarking services to information systems and information technology users and providers. There are also groups of users and providers who share information of mutual interest and value, known as benchmarking clubs.
4.13 If the Department established long-term relationships with commercial research organisations and benchmarking clubs it could concentrate its internal resources on:
■ identifying areas where there is a need for information on the external market;
■ specifying the study requirements; and
■ interpreting the study results and translating them, when appropriate, into action plans.
4.14 The effectiveness of the Department's benchmarking of hardware and infrastructure costs was also handicapped by the fact that, in some cases, EDS was prevented by contractual arrangements with its suppliers from sharing information on prices with third parties, in view of the commercial sensitivities involved. The Department told us that, subsequent to our examination, it had established some procedures and EDS was participating actively in further software benchmarking.
4.15 The Department also told us that it believed that the benefits of benchmarking lay more in providing information which it could use in its price and performance negotiations with EDS than in providing absolute assurance about where the Department stood in relation to other information technology service providers. On the cost of development work, for example, there were questions whether information supplied by third parties included the cost of work on abortive systems development as well as successful development work. It was considering how best to take its benchmarking work forward with the assistance of external advisers.