As stated in Chapter 2 (see '2.3.4 Method'), the process of using semi-formal instruments for data acquisition should begin with design, followed by pilot testing.
As a means of commencing and pre-testing the design, the findings of the initial review of literature and case studies (Chapters 4 and 5) were first discussed with professional contacts comprising three executives and two senior managers from the Victorian State Government, each with extensive PPP experience. These interviews, conducted between June and July 2011, yielded informal feedback and guidance on the issues identified for partnership, risk and performance management in PPPs. A copy of the literature review findings (presented as structured questions and their justification) is appended as Attachment C.
The pre-testing discussions then led to the development of a data collection instrument comprising three sections, each containing structured questions / prompts for a more formal pilot study. These are presented in Tables 8.1 to 8.3 below. It was intended that a number of unstructured topics would be covered in addition, depending on the responses received. Thus, while the data collection instrument followed a structured design, it was intended to be administered in a semi-structured fashion.
Table 8.1 shows the material pertaining to PPP operational partnership management.
Table 8.1 Pilot Study Protocol: Partnership Management Issues.
Partnership management questions | |
| Organisational culture |
1 | Do you think the organisational culture of the public partner influences Value-for-Money outcomes during PPP operational phases? |
2 | Can: a) personality traits of individual public partner project team members b) the effectiveness of public partner project teams c) incentives provided to the public partner project team ...contribute to the development of effective organisational culture, leading to improved Value-for-Money outcomes during PPP operational phases? |
3 | In your experience, what other factors contribute to the achievement of an effective organisational culture? |
4 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve organisational culture in PPP projects? |
| Management commitment and support |
5 | Do/does: a) negotiated outcomes between public partner decision-makers and the private partner over service delivery standards and outputs b) the acquisition and allocation of additional resources for under-resourced public partner project teams ...always lead to the achievement of Value-for-Money outcomes during PPP operational phases? |
6 | In your experience, what other actions taken by public partner decision-makers lead to effective commitment and support during PPP project operations? |
7 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve existing levels of management commitment and support in PPP projects? |
| Employee capability and expertise |
8 | How does: a) clear definition of public partner project team member roles and responsibilities b) the development of subject matter knowledge and the way this knowledge is applied by public partner project team members ...contribute to the development of employee capability and expertise, leading to improved Value-for-Money outcomes during PPP operational phases? |
9 | In your experience, what other factors influence the improvement of employee capability and expertise with respect to the public partner during the operational phase of PPPs? |
10 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve employee capability and expertise in PPP projects? |
| Clear and open communication |
11 | Is the development of: a) shared understanding between public and private sector partners b) trusting relationships between public and private sector partners ...essential to clear and open communication, leading to the achievement of Value-for-Money outcomes during PPP operational phases? If yes, in which way(s)? If no, why not? |
12 | In your experience, what other factors influence to clear and open communication during operations? |
13 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve the clarity and transparency of communication in PPP projects? |
| Relationship continuity |
14 | Can: a) personal and professional influence exerted by public partner project team members on the private partner ...contribute to relationship continuity, leading to improved Value-for-Money outcomes during PPP operational phases? If yes, in which way(s)? If no, why not? |
15 | In your experience, what other factors influence relationship continuity during operations? |
16 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve relationship continuity in PPP projects? |
| Conflict management |
17 | Can effective management of conflict, arising from: a) service delivery under-performance b) differences in organisational cultures between the public and private sector partners ... contribute to the achievement of Value-for-Money outcomes during operational phases? |
18 | In your experience, what other factors lead to conflict during operations? |
19 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve conflict management in PPP projects? |
| Other (un-identified) factors |
20 | Are there any other partnership management issues that have not been identified that could impact on PPP operations? |
21 | If yes, what: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve partnership management? |
22 | If you had to pick one partnership management experience that is most meaningful for you to share in context of this research, what would it be? |
Table 8.2 summarises the PPP operational risk management issues.
Table 8.2 Pilot Study Protocol: Risk Management Issues.
Risk management questions | |
| Implementation of transition plan |
1 | Can: a) project / integration challenges during transition ...impact on the achievement of planned operational phase of Value-for-Money outcomes? |
2 | In your experience, what factors typically impact upon implementation of transition plans at the commencement of operational phases? |
3 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve the transition of implementation plans in PPP projects? |
| Contract variation |
4 | Can: a) modification of existing services b) re-allocation of risk c) business continuity planning modification ...contribute to the achievement of Value-for-Money outcomes during operational phases? |
5 | In your experience, what other factors can lead to contract variation during operations? |
6 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve the contract variation process in PPP projects? |
| Change of consortium members / change to public partner's agency authority |
7 | Do changes to consortium members or to the public partner's agency authority lead to: a) exposure to new risks for the public partner in the operational phase of PPP b) changes in the management of risks by the public partner c) re-allocation of new risks If yes, in which way(s)? If no, why not? |
8 | In your experience, what other factors can: a) impact upon the change of consortium members b) lead to changes to the public partner's agency authority ...during PPP operations? |
9 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve the transition process of consortium members in PPP projects? ...could be implemented by the public partner in making changes to its agency authority? |
| Contract termination |
10 | Does: a) failure of a service provider ...always lead to poor Value-for-Money outcomes? |
11 | In your experience, what other factors can lead to contract termination during operations? |
12 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to reduce the impact of contract termination in PPP projects so that Value-for-Money outcomes are still achieved? |
| End of concession hand-over |
13 | Can: a) asset monitoring b) transfer of project documentation / knowledge c) orientation / up-skilling new employees ...contribute to the achievement of effective end of concession hand-over, leading to improved Value-for-Money outcomes towards the end of PPP operational phases? If yes, in which way(s)? If no, why not? |
14 | In your experience, what other factors can impact upon end of concession hand-over during operations? |
15 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve end of concession hand-over in PPP projects? |
| Reputation damage |
16 | During PPP operations, can the public partner's reputation be damaged by: a) issues / omissions that the private partner is legally accountable for resolving b) the failure to realise project benefits / criticism by service users c) a public perception that service operators are treating their customers unfairly d) criticism from public commentators due to the protection of sensitive and commercial information through 'commercial in confidence' arrangements What other situations / circumstances can lead to the public partner being criticised during operations? |
17 | Can: a) the implementation and enforcement of governance, probity and compliance frameworks b) the use of confidentiality agreements for public partner project team members c) 'commercial in confidence' agreements d) responsive management of un-anticipated / un-intended events ...reduce the impact of potential reputation damage to the public partner, therefore contributing to the achievement of Value-for-Money outcomes during PPP operational phases? |
18 | In your experience, what other factors can reduce the impact of reputation damage for the public partner during operations? |
19 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to reduce reputation damage for government in PPP projects? |
| Other (un-identified) factors |
20 | Are there any other risk management issues that have not been identified that could impact on PPP operations? |
21 | If yes, what: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve risk management? |
22 | If you had to pick one risk management experience that is most meaningful for you to share in context of this research, what would it be? |
Table 8.3 reflects the PPP operational performance management issues.
Table 8.3 Pilot Study Protocol: Performance Management Issues.
Performance management questions | |
| Performance management systems modification |
1 | Is: a) the ability to capture, monitor and measure all performance outputs and outcomes through performance management systems b) testing public partner project team members' knowledge / skills in line with performance systems upgrades and associated documentation ... essential to the achievement of Value-for-Money outcomes during operational phases? |
2 | In your experience, what other factors can affect Value-for-Money outcomes during transitioning of performance management systems? |
3 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve PPP performance management systems? |
| KPI modification |
4 | Does: a) annual KPI review b) ongoing KPI review ... always contribute to the achievement of Value-for-Money outcomes during operational phases? |
5 | In your experience, how often should KPIs be reviewed during operations? |
6 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers when deciding when to review and adjust KPIs in PPP projects? |
| Availability and integrity of performance data and metrics |
7 | Is the: a) availability of performance data and metrics b) integrity of performance data and metrics ... essential to the achievement of Value-for-Money outcomes during operational phases? If yes, in which way(s)? If no, why not? |
8 | In your experience, what factors can impact upon availability and integrity of performance data and metrics during operations? |
9 | What: a) policies / principles b) actions ...could be applied / taken to provide assurance that the availability and integrity of performance data is / remains intact? |
| Performance monitoring and adjustment |
10 | Does: a) performance evaluation b) management reporting c) managing performance shortfalls d) opportunity (risk) implementation ... always contribute to the achievement of Value-for-Money outcomes during operational phases? If yes, in which way(s)? If no, why not? |
11 | In your experience, what other factors can impact upon performance monitoring and adjustment during operations? |
12 | What: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve performance monitoring and improve the likelihood that corrective actions will lead to enhanced Value-for-Money outcomes in PPP projects? |
| Penalties and abatements |
13 | Does /do: a) applying penalties and abatements b) decisions taken to relax the application of penalty / abatement regimes ...always lead to the achievement of Value-for-Money outcomes during operational phases? If yes, in which way(s)? If no, why not? |
14 | In your experience, what factors should form the basis of decision-making leading to abatement / non-abatement for service under-performance during operations? |
15 | What: a) policies / principles b) actions ...could be applied / taken by the public partner when deciding not to abate that will increase the likelihood that private partner performance will improve in the future? |
| Other (un-identified) factors |
16 | Describe any other performance management issues that have not been identified that could impact on PPP operations? |
17 | If yes, what: a) policies / principles b) actions ...could be applied / taken by public partner decision-makers to improve performance management? |
18 | If you had to pick one performance management experience that is most meaningful for you to share in context of this research, what would it be? |
In the pilot study, participants were asked to assess the pilot study protocol design (Tables 8.1 to 8.3) against the criteria outlined in Table 8.4. The document referred to as the 'Research summary', which appears as a footnote under Table 8.4, is appended as Attachment D.
Table 8.4 Pilot Study Assessment Criteria.
Pilot Study Assessment Criteria |
Are the questions aligned with the stated aim (purpose) and objectives (significance) of the research?* |
Are the individual questions supported by the main research question?* |
Are the questions relevant, clear, unambiguous and effective? |
Are the questions presented in a logical order? |
Have the 'right' number of questions been asked i.e. not too many / few? |
Is the timeframe allocated for interview appropriate? |
Do gaps / omissions exist in the line of questioning? |
Do you have any other comments about the proposed method of data collection or the data collection instrument? |
*As stated in the 'Research summary' document: Attachment D.