1.37 Although the College's evaluation of its training compares well against good practice, we consider that there are a number of improvements which the College could make. The College should be in a stronger position than other organisations to evaluate its training as there is the expectation that its graduates will continue their Service careers. The College therefore has, in many respects, a "captive" population which should enable it to establish robust tracking arrangements.
1.38 The evaluation of the students' performance in the medium and long term relies on the use of questionnaires (Figure 5), feedback from the College's governance structure and informal discussions with senior officers and other key stakeholders when visiting the College to lecture (paragraph 1.31). In our opinion, the College should augment the above with other methods, such as follow-up interviews with individuals, or focus groups with a sample of graduates and their line officers. We pilot-tested the use of focus groups by holding one such group, composed of graduates of the early Advanced courses. We found that this forum provided a useful opportunity to explore in depth issues such as students' motivations and perceptions. We also consider that the College should put its discussions with visiting lecturers on a more formal basis by, for example, giving them a list in advance of the questions it wishes to discuss with them.
1.39 The College's evaluation is currently confined to obtaining feedback from the graduates and their line managers (both military officers and civil servants). We consider that there would be benefits if the College extended this evaluation to include gathering feedback from the graduates' peers (those with whom they work) and from other key stakeholders who would be in a position to provide an informed assessment on the level of jointery generally demonstrated by the College's graduates. Such key stakeholders would include senior civil servants and NATO and UN commanders. Given that such feedback would be of a general nature, we consider that obtaining this would not contravene the Department's current policy of not using 360 degree appraisal to assess the performance of individuals. We also recognise that many of these stakeholders, but not all, will now be contacted under the College's planned improvements to its evaluation (paragraph 1.30).
1.40 In identifying the impact of its training on its students' performance in the medium and long-term the College makes no use of the performance information generated by the single Services' personnel systems. In our opinion, the College should discuss with the Services' personnel sections the possibility of obtaining access to this management information in such a way that the confidentiality of the contents of individual performance appraisal reports is maintained. Such access would help the College to identify career progress and performance after attendance on the course. This should be more easy to do now as the College has harmonised its end-of-course appraisal report with the Services' performance report. This harmonisation should help the Services' personnel officers gain a clearer understanding of the final reports the College produces on its graduates. It should also make comparisons between the students' final marks and their subsequent performance easier.
1.41 Access to the single Service's management information and to the graduates over a long period of time should assist the College in assessing the continuing relevance of the training and the competencies that it delivers. It should also enable the College to identify more objective measures of training effectiveness by analysing trends in such matters as the length of service of graduates and the speed of their promotion, although other factors in addition to training also affect these trends. We recognise that there can be no control group of non-graduates against which comparisons can be made as all of the most talented officers attend the Advanced course or an equivalent overseas course.
1.42 In order to identify the value added by the College's training, it is not sufficient to measure the quality of the student at the end of the course. It is also necessary to establish this at the start of the course in order to measure the improvement over the course. The information already exists within the College to enable it to do this. For example, the College could plot participants' marks during the course to identify value added by attendance. The College could then use these marks to identify group trends (such as whether weaker performers improve and whether initially strong performers remain static) and to compare performance on a course year-on-year.
1.43 Currently the College does not set itself firm targets for the levels of satisfaction to be achieved in any of the evaluation methods and tools. In our opinion, the College should monitor satisfaction and other performance ratings (for example, percentages of graduates appointed to senior posts) to identify trends over a number of courses. Significant upward or downward movements should act as triggers for follow-up investigation to identify changes needed, and best practice and lessons to be learned.