a) Evaluation Criteria
The Evaluation will mainly use as reference guides the Development Assistance Committee (DAC) Principles for Evaluation of Development Assistance, 37 the DAC Quality Standards for Development Evaluation,38 and the Good Practice Standards of the Evaluation Cooperation Group39 in conducting the assessments at the various levels. Specific guiding principles to include gender equality and disadvantaged groups into this approach are inclusion, participation and fair power relations. Field Discussions with CSOs, community groups and beneficiaries will help assist in implementing these guiding principles.
The following Evaluation Matrix presents the main Evaluation questions and criteria related to the achievement of development outcomes and the management of Bank's PPP interventions as well as the data sources and data collection methodology. It will serve as an umbrella for the different levels of assessment: project lending or non-lending interventions or country levels, and aggregation (sector and overall results).
Table 8: Evaluation Criteria, Questions, Sources and Data Collection Methods
| Evaluation Criteria/Issues | Evaluation questions | Sources of Data/data collection methods |
| 1 - Achievement of Development Results | ||
| PPP Relevance: | To what extent Bank assistance to PPPs and interventions are relevant? | Country Case Studies field-based or desk work based PRAs. Other Field and desk work; Interviews with Actors and Stakeholders |
| Strategic alignment to Bank's policies and strategies including MTS and TYS | Are Bank assistance and PPP interventions aligned to Bank corporate, sector and thematic policies? | CSPs, CSPEs, Bank corporate, sector and thematic policies and strategies Country case studies Interviews of actors and stakeholders |
| Alignment to country policies, strategies | Are Bank assistance and PPP interventions aligned/appropriate to country development priorities and strategies, policy environment and development needs (fulfilling the infrastructure gaps)? | Country development plans, infrastructure investment plans, PPP policies and strategies Country case studies Interviews of actors and stakeholders |
| Relevance of Objectives and Quality of the design including risk analysis and mitigation | How relevant are the PPP interventions objectives? How is the quality of the design of Bank assistance compared to alternatives or other options based on fiscal sustainability, risk pricing and sharing, etc. | Structuring and due diligence reports, PEN, PCN and PAR, ADOA notes, Credit Notes Bank interventions ToCs and risk analysis. Country Value for money (VfM) analysis Country case studies PPIAF & EIU documentation Other donors documents |
| PPP Effectiveness (See Theory of Change40 and Annex-7 for details): | To what extent Bank assistance in PPP projects and interventions are effective and yield development results? | CSPEs; Cluster Evaluations; XSR; XSREN; Country Case Studies; field-based or desk-work based PRAs. Other Field and desk work; Interviews with Actors and Stakeholders |
| Sustainability | Are Bank PPP assistance and PPP interventions sustainable and will continue after the support of the Bank or other donors? | Country case studies Field-based or desk-work based PRAs. Other Field and desk work; Interviews with Actors and Stakeholders CBAs, Socioeconomic analysis Interviews with actors, stakeholders and beneficiary surveys |
| PPP Cross-cutting issues | To what extent Bank PPP assistance and interventions have contributed or are likely to contribute to inclusive growth? | Country case studies Field-based or desk-work based PRAs. Other Field and desk work; Interviews with Actors and Stakeholders |
| PPP Strategic Framework | What is the strategic framework guiding the Bank's PPP engagement? | CSPs, CSPEs, Bank corporate, sector and thematic policies and strategies |
| Selectivity | Was Bank PPP engagement selective based on comparative or competitive advantage and strategic (consolidatioof Bank positioning in the infrastructure sector and the country) | CSPs, CSPEs, Bank corporate, sector and thematic policies and strategies Country case studies Interviews of Project assessment teams, actors and stakeholders |
| Quality of front-end work and additionality | Are Bank interventions well structured with quality due diligence, assessment of development outcomes and additionality? | Structuring and due diligence reports, PEN, PCN and PAR, ADOA notes, Credit notes Country case studies and PRAs Other donors (PPIAF etc…) guidelines and benchmarking studies |
| PPP operational directives and guidance | How operational directives and guidance for screening, structuring, due diligence, and approval including ex ante additionality & development outcomes assessment are effective and efficient as compared to good practices and other MDB operational processes? | Bank Manuals and guidelines for screening and structure infrastructure projects and PPP Specific guidelines for PPP procurement and contract management ALSF guidance notes on PPP agreements ADOA guidelines and specific templates for PPP interventions Bank corporate, sector and thematic policies and strategies Other donors (PPIAF etc…) guidelines and benchmarking studies Country case studies Interviews of Project assessment teams, actors and stakeholders |
| Efficient Use of resources | Are Bank PPP interventions efficient and contributed to ensure an efficient use of resources including financial, economic and institutional efficiency? | Country case studies Field-based or desk-work based PRAs. Other Field and desk work; Interviews with Actors and Stakeholders CBAs, Socioeconomic analysis Interviews with actors, stakeholders and beneficiary surveys |
| Bank's role and contribution to leverage, partnership and coordination | What is the role of the Bank and to what extent was it effective and efficient in ensuring leverage, partnership and coordination? | CSPs, CSPEs, Bank corporate, sector and thematic policies and strategies Country case studies Interviews of Project assessment teams, donors, actors and stakeholders |
| Policy Dialogue, ESW, Advisory services, analytical capacities and institutional strengthening | How effective and efficient are advisory services and analytical work (ESW), institutional capacity building and technical assistance provided within PPP interventions? | CSPs, CSPEs, Bank corporate, sector and thematic policies and strategies Country case studies Interviews of OPSD, OSGE, ALSF, ORPF, EDRE, Project assessment teams, donors, actors and stakeholders |
| Innovation and Scaling Up | Has the Bank provided solutions adapted to country and project contexts including innovative approaches? | CSPs, CSPEs, Bank corporate, sector and thematic policies and strategies Country case studies Interviews of OPSD, OSGE, ALSF, ORPF, EDRE, Project assessment teams, donors, actors and stakeholders |
| Contribution to Managing for Development Results | To what extent have Bank's PPP interventions contributed to managing for results within the Bank and to RMCs? | CSPs, CSPEs, Bank corporate, sector and thematic policies and strategies Country case studies Interviews of OPSD, OSGE, ALSF, ORPF, EDRE, Project assessment teams, donors, actors and stakeholders |
| Factors of Success or Failures | What are the critical factors of success or failure | CSPs, CSPEs, Bank corporate, sector and thematic policies and strategies Country case studies Interviews of OPSD, OSGE, ALSF, ORPF, EDRE, Project assessment teams, donors, actors and stakeholders |
| Overall Assessment, Conclusions, Lessons and Recommendations | Overview of findings, conclusions and Lessons to be learned |
|
Specific and detailed evaluation questions, criteria and data collection methods, derived from the main evaluation matrix are developed at project level (Annex 7) and country level assessments (See Annex 10) of volume 2 of the Inception Report. Annex 8 presents the PPP Project Evaluation Guide and Template while Annex 9 presents the Project Results Assessment (PRA) Rating Guidance Notes. Detailed evaluation questions, criteria and data collection methods for non-lending TAs, ISPs and ESW are also developed (Annex 11). In addition to the guidance, a quality assurance (QA) will be implemented internally. The QA will be guided by a form and involve a concurrent review of PRAs by typically two (2) IDEV staff, followed by a comparison/discussion to qualify each PRA (as meeting the minimum quality threshold for inclusion in the synthesis or to be reviewed or rejected).
A PPP portfolio review will also be conducted to generate standard portfolio Key performance indicators (KPIs) such as the disbursement ratio, average size, composition (green-field, brown-field projects), quality at entry and at exit, potentially problematic and project at risk, and evolution over time by sector, regions, income countries etc., and will respond to the following questions:
• To what extent were PPP operations aligned to the Bank strategic vision (PSD and other high level strategic objectives in priority areas? How has the Bank deployed its instruments to support PPP? To which extent the PPP portfolio Pipeline is aligned to the strategic objectives of the actual Bank strategies and the "High 5s"? To what extent have PPP operations contributed to enhanced Bank's visibility, institutional effectiveness and efficiency, and development effectiveness (including sustainable development or SDGs)?
• To what extent do the quality and performance of PPP portfolio adhere to established quality at entry criteria, timelines; cost estimates, and implementation performance, financial performance (nonperforming loans) including the financial returns it has generated for the Bank (Investments profitability)? How is the Bank managing PPP investment risks (assessing and pricing investment risk)?
• What are the underlying causes for PPP interventions financing effectiveness and disbursement delays? To what extent does quality-at-entry default have a cost and/or negative impact on the PPP portfolio performance?
• To what extent ADOA ratings and implementation contribute to desired outputs and outcomes?
• How effective was portfolio management at HQ and field offices in reducing PPP projects at risk and operations cancellation? What is the quality at exit of matured operations based on XSR and XSREN and what is the Net disconnect of project ratings? What are the critical factors of success or failure?
The Guiding Template of the Portfolio Review is presented in Annex 12 of Volume 2 of the Inception Report. Furthermore, due to the particular importance of the PPP-related ISPs, Policy Dialogue, ESW, Advisory services, analytical capacities and institutional strengthening initiatives, specific questions, criteria and data collection methods were developed to carry-out an in-depth assessment. Specifically, the review will answer the following questions:
• To which extent PPP TAs, ISPs, Budget support components (General or Sectoral), Capacity strengthening and ESW are consistent and aligned to the Bank and country Strategic Objectives?
• What is the quality of Bank policy dialogue, coordination and partnerships activities on legal and institutional framework, advisory and investment services (structuring, contractual arrangements): Bank strategic positioning, effective Bank PPP hubs, and representation, support of country analytical, knowledge products and dissemination, capacity strengthening and resource mobilization for PPP preparatory work, institutional and regulatory framework, and PPP implementation?
• To which extent the generated analytical work, advisory services and knowledge products have been used by countries and Bank policymakers, project managers and other Stakeholders?
• To which extent Bank non-lending activities (PPP TAs, ISPs, Budget support components (General or sectoral), Capacity strengthening and ESW) were harmonized with other donors and actors (avoiding duplication) to promote the PPP enabling environment, regulatory, legislative and institutional framework enhancement (PPP law and PPP unit establishment)?
• To which extent the Bank (through its hubs or other instruments such as budget support) has focused on strengthening national/regional capacities in contractual management and procurement systems, monitoring and evaluation (M&E) and results orientation of PPPs?
• What are the factors that have permitted/contributed or limited to Bank non-lending activities (TAs, ISPs, BSO components etc.) achievements?
• What has worked or has not worked and why? What are the lessons to be learned in future Bank non-lending activities?
b) Sources of Information
The evaluation will rely on internal and external documentation, triangulated statistical data at project level, country or development agencies, and will comprise an in-depth literature review of various available sources particularly of the MDBs. Official country statistical and administrative data will be, to the extent possible, corroborated and/or triangulated with other available sources, particularly from the Bank, IMF, WB, IFC, and UN agencies with specific reference to the various study reports or data analyses. The Bank PPP hubs, RECs and Country offices, clients, financiers or counterparts will also be an important source of information.
PPP projects documentation such as PCN, PAR, ADOA, CSN, PSR, PCRs, XSRs as well as country documentation (CSPs, CSPE, CPPRs,..), ex ante social impact study and PRAs conducted within the Country Strategy and Program Evaluations (CSPEs) under the Comprehensive Evaluation of Bank assistance 2004-2015 will be used as background documentation for this evaluation. Country sources of information will be related to economic and social data, regulatory and legislative framework, private investment attractiveness, fiscal sustainability, inclusiveness, environmental and social safeguards as well as PPP funding mechanisms and modalities (eg. tariffs, user fees, maintenance fees, budgetary allocations, other stakeholder contributions, aid flows), Value for Money analysis, procurement, contract management and implementation of the Government PPP Unit. These will be the central and sector ministries, private sector companies, local authorities, beneficiaries, CSOs, donors (MDBs, DFID, USAID, AFD, ALSF), PPP agencies/regulators.
Other sources of secondary qualitative and quantitative data and/or perceptions will be gathered through interviews, mini-surveys or surveys with relevant counterparts, government officials, beneficiaries, investors, industry associations, civil society organizations, academia, and other suitable stakeholders, complemented by data gathering and site visits (See the detailed evaluation output templates in the technical Annexes (Volume 2 of the Inception Report).
____________________________________________________________________________________
37 OECD (1991). Principles for Evaluation of Development Assistance. Development Assistance Committee; Paris.
38 OECD (2010). Quality Standards for Development Evaluation. Development Assistance Committee, Paris.
39 Of particular relevance to this Evaluation are the Good Practice Standards for Country Strategy and Program Evaluations (2008); the Good Practice Standards for the Evaluation of Public Sector Operations (February 2012), and the Good Practice Standards for the Evaluation of Private Sector Investment Operations, Fourth Edition. November 2 and the ECG Harmonized Evaluation Criteria and Rating, 2013.
40 See Table 7 and Annex-6 for details.