6.4 Evaluation and Data Collection methodology

The chosen methodology is one of mixed methods with the utilization of participatory processes at project/intervention (lending or non-lending), country and sector levels. The combination of the application of quantitative and qualitative methodologies will allow the identification of good practices, lessons learned and recommendations on the way forward. More importantly, it will make use of:

•  A retrospective results framework approach and Contribution Analysis (CA) (see above) using multiple data sources and triangulation.

•  Discussions with key stakeholders (country officials, CSOs and beneficiaries) within the framework of semi-structured or field surveys will favor their enrolment in the process while seeking their views on key findings and conclusions

•  Cost-benefit analysis (CBA) of infrastructure PPPs to determine the socio-economic benefits.

•  An inclusive and gender-responsive approach (mainly at project level).

•  Structured surveys of Bank Task Managers and co-financiers or partners (such as ALSF, PPIAF, etc..) will also implemented, particularly through the benchmarking exercise.

•  An analysis of external factors and unintended effects to sustain the Attribution/Contribution Analysis.

a) Project Level Assessment (Lending Activities)

The evaluation will mainly focus on the infrastructure projects (power and transport) as they constitute 94% of total PPP portfolio (see section 5). The detailed project level assessment or Project Result Assessment (PRA) will be conducted by either the Consultant/ Consulting firm hired to support the PPP evaluation process, or internally. Sixteen (16) PRAs will be field-based for totally disbursed or completed and close to completion projects and desk work-based for purposively selected active or completed projects to complement the country case and sector studies. It is to be highlighted that simplified PRAs will be prepared for on-going or active projects which will focus on actual implementation performance and likelihood of potential results. Detailed guidance is provided in order to minimize risks of non-consistent assessment across countries.

b) Assessment of Non-Lending Activities

The assessment will be conducted at all ISPs (18 interventions) as well as all identified non-lending activities and will be aggregated at country, sector and overall levels.

c) Country Level Assessment

Nine (9) Country Case Studies will be carried out to answer the questions provided for in the Evaluation Results Matrix. The purpose is to assess the quality of Bank assistance in supporting RMCs PPP agenda and implementing the PPP financed transactions as well as contributing to achieving high development results in terms of inclusive accessibility, affordability, poverty reduction, gender, youth and regional disparities. Country case studies will also assess how well the Bank has managed PPP interventions in terms of the work quality, additionality, policy dialogue, ESW, Advisory services, analytical capacity and institutional strengthening, work coordination, leverage and scaling up.

Country Case Studies will be based on an in-depth analysis of Bank's assistance strategies to develop infrastructure through PPP modalities using Bank's financing instruments (investment lending, guarantees, partial guarantees, equity participation, risk management, etc…) as well as non-lending activities such as ESWs and knowledge products, ISP, TAs, policy dialogue, donor coordination and leverage. The in-depth analysis will also look for variation factors and contrast findings across countries/RMCs taking into account the contextual effects. The country case studies will seek to:

•  Collect data on long term performance of AfDB supported PPP interventions;

•  Collect feedback from country stakeholders, including government, implementing partners and beneficiaries;

•  Verify the results achieved through Bank's support to PPPs in the selected countries using comparison analysis;

•  Assess the relevance, alignment and selectivity of Bank's contribution including its additionality through policy development, economic and sector work, and institutional capacity building of countries utilizing PPP to promote hard and social infrastructure;

•  Test the Results Framework of Bank interventions which involve support to PPPs;

•  Identify the factors which contribute or limit the achievement of sustainable development results (including fiscal and institutional) through PPPs;

•  Provide descriptive examples to support the broader evaluation, both in terms of good practices that require improvement.

The individual countries are selected through application of a stratification procedure in order to obtain a good coverage of the Bank's support to PPPs in terms of:

•  Geographic distribution (North, South, Central, East, West);

•  Level of sophistication of the regulatory environment and institutional framework41 in developing sustainable and efficient PPP projects;

•  Sectoral distribution of Bank's support (energy, transport, social and lending/non-lending, policy dialogue and towards PPP development);

The purposive sample of countries has the following properties:

a.  It covers 20 projects representing 68% of total PPP portfolio as well as of the net commitments;

b.  It represent 75% of the totally disbursed loans for completed projects;

c.  It represents the bulk of the regulatory environment and institutional frameworks in Africa (Emerging and Nascent) (See Table 1 in Section 4).

This will allow for a rich and diverse learning experience with a differentiation regarding countries' PPP design and implementation agenda as a strategic framework42 to fulfill/close the infrastructure gaps and market failure while adopting the right regulatory and legislative environment for PPP infrastructure. The proposed purposive sample of country case studies is set as follows:

Table 9: Selected Countries as Case Studies:

Source: BDEV and 2015 EIU PPP business environment scores and ranking

d) PPP Institutional Arrangements in MDBs - A Benchmarking

As the majority of MDBs and bilateral development agencies have full-fledged work programs on PPPs, an analysis of the strategic relevance of PPPs across these agencies, the nature of their support, their organizational and institutional arrangements and solutions to deliver on their respective PPPs will help draw useful lessons to the Bank. This analysis will benchmark their experience with implementing as well as identifying the emerging issues in managing PPPs. This will include questionnaires and when possible visits to selected MDBs (World Bank/IFC/MIGA, EIB, EBRD, IaDB, AsDB and AFD) 43 . The specific questions/issues to be covered by the benchmark analysis are the following:

1.  Strategic relevance of PPPs

a.  Specific PPP policy and strategy guiding MDB/Bilateral agency PPP financing and support

b.  Specific Operational work plan and guidance for upstream (ESW, AAA, capacity building) and/or downstream support (investment services and support for implementation and exit)

c.  Knowledge generation and management (feedback, dissemination and use)

2.  Organizational and institutional arrangements

a.  Institutional arrangement (corporate policy, PPP unit/hub and/or focal points, Strategy/Operational Unit within the institution, decentralized delivery)

b.  Organizational arrangement: Mandate: authority and responsibility, staffing and budget resources;

c.  Financing instruments (lending and non-lending, dedicated trust fund)

d.  Relationship with internal policy and operations in charge of private sector (or infrastructure) development

e.  Relationship and coordination with other donors in joint activities

3.  Active and completed portfolio and development results

a.  Volume and number of financed PPPs during the last decade, by region and sector (Trend analysis); active vs completed;

b.  Processes and operational guidelines

c.  PPP Portfolio management: performance and efficiency (quality at entry, quality of supervision, time length for structuring including origination and due diligence, effectiveness, implementation and completion)

d.  Co-financed PPP operations and support

4.  Achievement of development results (Development effectiveness)

a.  Satisfactory performance of PPPs (achievement of outputs, outcomes and impacts)

b.  Sustainability of outcomes and impacts

c.  End-users/beneficiary satisfaction and ownership

5.  Emerging issues, barriers, enabling factors affecting performances, solutions delivered and future plans

a.  Emerging issues from past experience of PPP support

b.  Barriers, Enabling Factors affecting performances

c.  Solutions delivered

d.  Planned evaluations of PPP policy, strategy, operations and support e.  Future plans towards PPP operations and support

e) Aggregation/Synthesis of the Various Level Assessments

The aggregation/synthesis of the various level assessments will be for the non-lending interventions, at the sector level, and overall.

Non-lending Review Synthesis - The Purpose of this Synthesis is to summarize the findings and conclusions of all the Non-lending activities, including policy dialogue, ESW, TAs, Advisory Services and Institutional Support (ISPs) and Capacity strengthening. The Summary guiding note and proposed note outline is presented in Annex 11 in Volume 2.

Sector Synthesis (Power, Renewable Energy, Transport)44 - The purpose of the Sector Case Studies is to assess the quality of Bank assistance in supporting RMCs PPP agenda and implementation of the PPP sector financed transactions and non-lending as well as its contribution to achieving high development results in terms of inclusive accessibility, affordability, poverty reduction, gender, youth and regional disparities.

The Sector Reviews will also assess how well the Bank has managed PPP interventions in a particular sector in terms of Bank work quality, additionality, as well as policy dialogue, ESW, Advisory services, sector analytical capacity and institutional strengthening, work coordination, leverage and scaling up. The aggregation/synthesis of the evaluation results drawn from PRAs, Country Case Studies and ISP reviews will help identify what has worked and what has not worked at the sector level and why based on a comparison of PPP in the sector to derive the specific drivers of success and failure of PPP interventions at a sector level. It will also analyze how the Bank has really made a difference by assessing its contribution to the sector development effectiveness by closing the infrastructure and inclusiveness gaps, for example.

The Guiding Template and proposed outline is presented in Annex 13 of Volume 2 of the Inception Report.

Overall Synthesis/Aggregation: A Synthesis/Aggregation of the evidence-based findings and conclusions triangulated through the various sources of evaluative information such as the PRAs, country case studies, sector reviews, benchmarking analysis, interview notes will be carried out. It will help assist in drawing conclusions on the quality of Bank assistance in supporting RMCs PPP agenda and implementation of the PPP financed transactions as well as its contribution to achieving its corporate goals and mandate in terms of inclusive accessibility, affordability, poverty reduction, gender, youth and regional disparities, covering the High Fives' topics. It includes the analysis of specific drivers of success and failure of PPP interventions at a PPP sector/country and thematic level, and how the Bank has made or will make a difference in contributing to RMCs sustainable development goals by closing the infrastructure and inclusiveness gaps, for example. The Synthesis will also draw conclusions on the Bank strategic fit and institutional effectiveness in assisting RMCs in creating the PPP enabling environment and appropriate investment climate, the Bank's contribution to development results and management of its PPP interventions. The proposed outline for the CODE Synthesis report in presented in Annex 14 of Volume 2 of the Inception Report.

____________________________________________________________________________________

41  This is based on the Economic Intelligence Unit (EIU) Benchmarking Index Scoring

42  PSD policy and strategy, Industrialization strategy

43  These institutions have extended PPP portfolio and strong experience in funding PPPs

44  These sectors represent the bulk of the portfolio. It is suggested to have a specific sector review report to the PPP in the renewable energy sector due to its potential development in Africa.