2006 annual report on operations evaluation - isbn:...

THE WORLD BANK

2006 Annual Report onOperations Evaluation2006 Annual Report onOperations Evaluation

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

WORKING FOR A WORLD FREE OF POVERTY

The World Bank Group consists of five institutions—the International Bank for Reconstruction and Development(IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), theMultilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of InvestmentDisputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their envi-ronment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public andprivate sectors.

THE WORLD BANK GROUP

ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION

The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses onassessment of IFC’s work toward private sector development, and IEG-MIGA evaluates the contributions of MIGAguarantee projects and services. IEG reports directly to the Bank’s Board of Directors through the Director-General,Evaluation.

The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of theBank Group’s work, and to provide accountability in the achievement of its objectives. It also improves Bank Groupwork by identifying and disseminating the lessons learned from experience and by framing recommendations drawnfrom evaluation findings.

THE INDEPENDENT EVALUATION GROUP

2006 Annual Report on

Operations Evaluation

2006The World Bank

Washington, D.C.

W O R L D B A N K I N D E P E N D E N T E V A L U A T I O N G R O U P

http://www.worldbank.org/ieg

©2006 The International Bank for Reconstruction and Development / The World Bank

1818 H Street NW

Washington DC 20433

Telephone: 202-473-1000

Internet: www.worldbank.org

E-mail: [email protected]

All rights reserved

1 2 3 4 5 09 08 07 06

This volume is a product of the staff of the International Bank for Reconstruction and Development / The World Bank. The

findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive

Directors of The World Bank or the governments they represent.

The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors,

denominations, and other information shown on any map in this work do not imply any judgement on the part of The World

Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries.

Rights and Permissions

The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission

may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank

encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly.

For permission to photocopy or reprint any part of this work, please send a request with complete information to the

Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-

4470; Internet: www.copyright.com.

All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher,

The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: [email protected].

Cover illustration credit: Images.com/Corbis.turbines. ©Royalty-Free/Corbis.

ISBN-10: 0-8213-6845-1

ISBN-13: 978-0-8213-6845-9

e-ISBN:0-8213-6846-X

DOI: 10.1596/978-0-8213-6845-9

Library of Congress Cataloging-in-Publication Data have been applied for.

Printed on Recycled Paper

World Bank InfoShop


Telephone: 202-458-5454

Facsimile: 202-522-1500

Independent Evaluation Group

Knowledge Programs and Evaluation Capacity

Development (IEGKE)


Telephone: 202-458-4497

Facsimile: 202-522-3125

v Acronyms and Abbreviations

vii Acknowledgments

ix Executive Summary

xv Summary of Management Response to IEG Recommendations

xix Chairman’s Summary: Committee on Development Effectiveness (CODE)

3 1 Introduction3 Using M&E Information for Results4 Overview of the 2006 AROE

7 2 Monitoring and Evaluating Results in Bank Operations7 Recent Developments in Monitoring8 How a Results Chain Works9 Possible Uses of Results Frameworks10 Current Status of Monitoring Results14 Challenges to Developing Results-Based Monitoring17 Self-Evaluation and the Bank’s Results Agenda 19 Implications for Future Work

25 3 Improving IEG’s Effectiveness25 IEG’s Results Framework26 IEG’s Outputs: Increasing the Focus on Results and Impact in

Independent Evaluation 27 Strengthening the Focus on M&E in IEG Evaluations29 IEG Initiatives for Applied Learning and Real-Time Use of Evaluation29 Deepening Strategic Partnerships 30 Strengthening IEG’s Results Framework31 Quality and Relevance of IEG Outputs32 Communicating Knowledge from IEG Evaluations33 IEG Intermediate Outcomes: Evaluation Recommendations Incorporated

into Bank Operations and Policy34 IEG Intermediate Outcomes: Use of Evaluation Findings by Bank Staff at

the Operational Level

Contents

437 Conclusions and Recommendations39 Recommendations for Management39 Recommendations for IEG-WB

41 Appendixes43 A: Overview of Monitoring and Evaluation in the World Bank51 B: Implications of the World Bank’s Results Agenda for Monitoring and

Evaluation59 C: Review of CAS Results Frameworks69 D: Focus Group and Individual Interviews with CODE Members,

Country Directors, Sector Managers, and Task Team Leaders79 E: Client Survey Results93 F: Overview: Changes in IEG’s Dissemination Methods 97 G: Accelerating the Results Agenda101 H: World Bank Management Action Record105 I: Management Response

109 Endnotes

113 Bibliography

Boxes12 2.1 What Constitutes a Strong Results Framework?15 2.2 What Are We Hearing from Managers and Staff?18 2.3 GTZ’s e-VAL System: Using Computer-Supported Evaluation Tools

for Rapid Progress Reviews 20 2.4 Integration of Objectives in Results Frameworks21 2.5 Reducing the Gap Between Country- and Project-Level Outcomes29 3.1 Real-Time Use of IEG Knowledge—Accountability through

Education Indicators in Peru

Figures8 2.1 Sample Results Chain for Agricultural Development13 2.2 Results Chains Better Developed in IDA Countries than in IBRD or

Blend Borrowers13 2.3 Country Assistance Strategies Tend to Lack Adequate Baselines and

Targets13 2.4 Precision of Milestones Could Be Improved13 2.5 Larger Programs Tend to Have More Indicators, Outliers May Need

Review26 3.1 IEG Results Chain27 3.2 Uses of Evaluation Knowledge32 3.3 Bank Staff, Executive Directors, and Advisors Generally

Satisfied with Quality of IEG Evaluations34 3.4 Level of Adoption of IEG Recommendations Similar to That in

Previous Years

Table16 2.1 Regional Client Services

i v

2 0 0 6 A N N UA L R E P O RT O N O P E R AT I O N S E VA L UAT I O N

v

ACRONYMS AND ABBREVIATIONS

AAA Analytical and advisory activitiesACS Activity Completion SummaryADB Asian Development BankAIS Activity Initiation SummaryAROE Annual Report on Operations EvaluationARPP Annual Report on Portfolio PerformanceBP Bank procedureCAE Country Assistance EvaluationCAS Country Assistance StrategyCASCR CAS Completion ReportCBD-CDD Community-based and -driven developmentCC Country coordinatorCD Country directorCGAP Consultative Group to Assist the PoorCODE Committee on Development EffectivenessCOMPAS Common Performance Assessment SystemCPPR Country Portfolio Performance ReviewCPR Country program reviewDAC Development Assistance Committee (Organisation for Economic Co-operation

and Development)DEC Development Economics Vice PresidencyDGF Development Grant FacilityDIME Development Impact Evaluation InitiativeDPL Development Policy LoanECD Evaluation capacity developmentECG Evaluation Cooperation GroupED Executive directorEI Extractive industriesESW Economic and sector workGMR Global Monitoring ReportGPP Global program and partnershipGPR Global Program ReviewGRM Grant reporting and monitoringGTZ Deutsche Gesellschaft für Technische ZusammenarbeitIBRD International Bank for Reconstruction and DevelopmentICR Implementation Completion ReportIDA International Development AssociationIEG Independent Evaluation Group (World Bank)IFC International Finance CorporationISR Implementation Status and Results ReportLICUS Low-Income Country Under Stress

LOC Line of creditM&E Monitoring and evaluationMAR Management Action RecordMDB Multilateral development bankMDG Millennium Development GoalMIGA Multilateral Investment Guarantee AgencyNGO Nongovernmental organizationOED Operations Evaluation Department (now IEG)OD Operational directiveOP Operational policyOPCS Operations Policy and Country Services (World Bank)PAD Project Appraisal DocumentPDO Project development objectivePPAR Project Performance Assessment ReportPSR Project Supervision ReportPRS Poverty Reduction StrategyQAG Quality Assurance GroupRBCAS Results-Based Country Assistance StrategyRMLS Results Monitoring and Learning SystemRMS Results Measurement SystemRSG Results Steering GroupSAP Systems Applications and ProductsSM Sector managerSSIU Sector Strategy Implementation UpdateSPC Strategy and Performance ContractSTATCAP Statistical Capacity Building Multicountry Adaptable Program LoanTFSCB Trust Fund for Statistical Capacity BuildingTTL Task team leaderVPU Vice presidential unit

OED changed its official name to the Independent Evaluation Group (IEG) in December 2005. Thenew designation “IEG” will be inserted in all IEG’s publications, review forms, databases, and Websites.

v i


v i i

Acknowledgments

This report was prepared by a team comprisedof Janardan Prasad (J.P.) Singh (task manager),Jan Rielaender, and Yoshine Uchimura. The peerreviewers were Sid Edelman (IndependentEvaluation Group [IEG]-International FinanceCorporation), Gregg Jackson (George Washing-ton University), and Alfred Nickesen (WorldBank Central Operational Services Unit), whoprovided invaluable advice and guidance inpreparing the report.

Janet Mancini Billson and Martin Steinmeyer fromGroup Dimensions International conductedfocus group interviews for this report. SusanCollins and Howard Smith from the AdvancedPerformance Consulting Group evaluatedCountry Assistance Strategy results matrices.

Thanks also go to Forum One Communicationsfor support in implementing and analyzing theclient survey, as well as to all those who respondedto the survey and participated in focus groups.

Many IEG staff, including Nils Fostvedt, PatrickGrasso, Keith Mackay, and Ray Rist, providedextensive support and contributions to allphases of this report. Aida Tapalova providedresearch assistance.

Others who have provided valuable input andassistance include Vivian Jackson, Pierre-JosephKingbo, Princess Moore-Lewis, Maria GabrielaPadrino, Juicy Qureishi-Huq, and Barbara Yale.Julius Gwyer, Maria J. Mar, Alex H. McKenzie, andYang Soo Yoon offered computer and technicalsupport.

Document production support was provided byWilliam Hurlbut. Caroline McEuen edited thereport.

The report was prepared under the direction ofKlaus Tilmes, Manager, Knowledge Programsand Evaluation Capacity Development.

Director-General, Evaluation: Vinod Thomas

Director, Independent Evaluation Group–World Bank: Ajay Chhibber

Manager, Knowledge Programs and Evaluation Capacity Development: Klaus Tilmes

Task Manager: Janardan Prasad (J.P.) Singh

i x

Executive Summary

The 2002 International Conference on Financing for Development in Mon-terrey emphasized the need to measure, monitor, and manage for re-sults and concluded with joint statements issued by the leaders of the

multilateral development banks. The World Bank began implementing itsManaging for Development Results action plan in 2003. The plan calls for stepsto (a) strengthen countries’ capacity and demand to manage for results; (b)enhance the Bank’s relevance and effectiveness in contributing to results; and(c) harmonize results-based approaches across development agencies.

The Bank’s results agenda shifts attention fromoutputs of development activities, such asdollars lent or reports written, toward countryand sectoral as well as global outcomes, such asthe Millennium Development Goals, economicgrowth, and environmental and social sustain-ability. This shift has implications for monitoringand evaluation (M&E). In a results framework,monitoring includes tracking outcomes as wellas outputs, while evaluation assesses the contri-bution of Bank operations to achieving results.

Managing for results requires better informationon program performance at the project, sector,country, and global levels to guide decision makingand reporting. The focus on results is a jointresponsibility of countries and developmentagencies and requires parallel efforts to strengthendemand and the capacity to make use of informa-tion. The results agenda combines heightenedaccountability at the country and development

agency levels with learning to understand betterwhat works, what does not, and why.

The 2006 Annual Report on Operations Evalua-

tion (AROE) updates the actions taken since the2004 and 2005 AROEs to strengthen the resultsfocus in M&E. First, it analyzes the implicationsof managing for results on Bank operations.Second, it assesses the extent to which theBank’s M&E systems provide staff with theinformation they need to better manage forresults. Third, the report evaluates products andservices of the Independent Evaluation Group-World Bank (IEG-WB) as part of a continuousprocess of self-evaluation and assesses theirquality, influence, and use among both internaland external audiences. The report also raisessome unresolved issues for further considera-tion. Finally, it makes recommendations on howM&E can be strengthened to increase the WorldBank’s effectiveness.

World Bank Progress in Implementingthe Results Agenda

The Bank has made progress in putting in place anarchitecture for managing for development results,but more work is needed to extend this frameworkbeyond country programs and projects.

Policies and procedures have been revised tomanage better for results. Results-BasedCountry Assistance Strategies (RBCASs),Country Assistance Strategy (CAS) CompletionReports, and IEG validation of the completionreports provide the basis for managing forcountry-level results. The results frameworks areintended to clarify the Bank’s contribution tothe achievement of country development goals.This addresses a weakness of previous CASs,which discussed country goals and Bankoperations, but did not link them systematically.

Investment Loans and Development PolicyLoans now have formal results frameworks thatlink individual Bank projects to countrydevelopment goals. Even if more difficult to doin many respects, similar efforts are needed insectoral program areas such as infrastructureand health initiatives and in global initiatives,whether in education or the environment. TheBank needs to assess more systematically andstrategically what has worked well in differentsectors, as is being currently attempted forhealth and for infrastructure.

Some Bank operational teams are using self-evaluation to improve performance, but these effortsare not yet systematic.

While still at a pilot stage, evaluative methods suchas rapid results approaches, deployed duringimplementation, can improve the effectiveness ofBank projects and programs. Impact evaluationshave been recognized as a formal analytical andadvisory activity product, and the DevelopmentImpact Evaluation Initiative has identified some 60ongoing impact evaluations, mostly in social sectorand poverty programs. A variety of approaches canbe used to assess program impact, including well-focused objectives-based assessments, rates of

return analysis, retrospective impact evaluation,and randomized experiments.

The Bank’s country partners are still at an early stagein adapting to a results focus.

Lack of capacity and the additional cost of datacollection are seen as constraints to M&E in theBank’s client countries. A few countries, particu-larly those preparing Poverty Reduction Strate-gies, are receiving Bank support to strengthentheir national statistical systems that supportimproved poverty monitoring. Some countries,most visibly in Latin America, are working withBank support to strengthen systems for M&E,both governmentwide and at the sectoral level.Engagement with both the country and multilat-eral and bilateral partners is essential in theseefforts.

The Bank’s approach to the results agenda needs tobe supported by complimentary action on severalfronts where early resolution could enhance theeffectiveness of the results agenda.

The AROE analysis confirmed the importance offocusing on country outcomes and aligningBank operations around them, often withinmultisectoral settings. Yet incentives for cross-sectoral teamwork remain weak. Also, withgrowing globalization, the potential role ofcross-country interventions—and the Bank’scontribution to them—is likely to grow.

Yet inadequate measurement of the effects ofsuch interventions in areas such as health,water and sanitation, and the environmentholds back current or future support for them.The Bank is encouraging staff to take ondifficult but important challenges, such asissues of governance or the environment orworking in tough country circumstances, suchas those of fragile states. In these situations, thefocus on measurable and achievable resultsneeds to be factored into the complexity of theprocess and the difficulty of outcomes so thatattention and incentives do not shift exclusivelytoward the outcomes that are easiest to obtainor measure.

x


Are M&E Systems Providing What IsNeeded to Manage for Results?

The use of the results framework as an effectivemonitoring and management tool remains inquestion.

An AROE review of 25 CAS results frameworksprepared during fiscal 2005–06 found that aboutone-third of the CASs had weak resultsframeworks that made it impossible to follow thelinks between country goals, CAS outcomes, andBank activities. In addition, more than half didnot include baselines and did not set specific,time-bound targets. The absence of performanceindicators with baselines and targets reduces theusefulness of the results frameworks.

In cases where indicators are present, there arecommonly too many, which makes the frameworkan unwieldy tool. At the project level, the QualityAssurance Group found that the clarity andrealism of development objectives could havebeen improved in about 40 percent of projectsreviewed in fiscal 2004–05. In addition, 40–50percent of the projects reviewed lacked well-articulated results frameworks with baselines andsystems for tracking and reporting progress. AnOperations Policy and Country Services (OPCS)review of the quality of results frameworks inImplementation Status and Results Reports (ISRs)that was prepared in fiscal 2005 found that 40percent did not contain any outcome indicatorswith baseline data for either the project develop-ment objective or intermediate outcome indica-tors, and were not considered acceptable.

Disconnects in outcome ratings between countryand sector policy outcomes and project outcomespoint to a gap in understanding the factors thatcontribute to results.

An IEG retrospective of Country AssistanceEvaluations (IEG 2005d) found that in a third ofthe country programs reviewed, aggregate projectoutcomes were satisfactory, but the overallcountry assistance programs were unsuccessful.Similar disconnects have emerged at the sectorlevel. While the Bank’s primary education portfo-

lio performed above Bank average and met statedexpansion targets, most such programs failed tolook beyond increased enrollment and comple-tion rates to how well students were learning. AnIEG evaluation of the Bank’s trade assistance (IEG2006a) found that Bank advice and lendingcontributed to systematic reductions in protec-tion and in opening up economies. But the impacton employment and poverty reduction was lessthan expected, and the Bank did not conductsufficient analysis to inform its policy and lendingin this area. Systematic evaluation during and oncompletion of Bank activities could help improveunderstanding of factors critical to success thatcan be used to refine the results framework at thecountry, sector, and project levels.

Bank managers and operational staff encounter avariety of challenges in managing for results.

Focus groups and interviews conducted for thisAROE indicated that managers and operationalstaff struggle to link higher-level goals to Bankoperations and to identify performance indica-tors. The possible uses of performance informa-tion remain unclear to many staff, and this actsas a disincentive to using monitoring to achieveoutcomes. Competing pressures in the Bank,combined with the lack of clarity about theutility of M&E, relegate performance informa-tion to a lower priority.

The Bank has made progress in putting formalpolicies and procedures in place, but those changeshave not yet translated into improved incentives andconsistently applied operational practices.

The challenge remains to change the Bank’sculture and the staff perceptions that serve asinformal incentives and disincentives to managingfor results. A particular issue worthy of furtherwork is to incorporate the relative difficulty orease of success into the results frameworks, andto find the appropriate balance between measuresof success and measures of risk.

There is a particular need for such balance inconsidering and encouraging Bank engagementin low-income countries under stress, for

E X E C UT I V E S U M M A RY

x i

example, or on difficult thematic issues such ascorruption or environmental degradation. Therecommendations of the 2004 AROE remainvalid: the Bank needs to maintain and communi-cate its vision for a results-oriented institution,strengthen incentives, and develop a cohesiveand fully funded action plan.

The Bank has outlined what needs to be done tomanage for results during the next stage, but lack ofprioritization and funding raise concerns about thelikelihood of success.

The 2006 OPCS results progress report (WorldBank 2006b) identified possible actions toaccelerate the implementation of the agenda butdid not identify priority areas and fundingsources. M&E improvement initiatives havefailed in the past because of fragmentation ofefforts, resource constraints, and lack of staffcapacity. These lessons need to be considered.

How Is IEG Contributing to Meeting theResults Agenda?IEG’s mandate is to assess whether the WorldBank Group’s programs and activities areproducing the expected results. While IEG isfully independent, it is placed within the WorldBank Group to provide strategic decisionmakers and operational staff with knowledgethat helps them work more effectively.

IEG is reinforcing its focus on M&E and is deepeningits work to develop evaluation capacity among Bankclients.

IEG is strengthening its assessment of M&E in itsevaluations. Country, sector, and project evalua-tions now give more attention to M&E and theeffectiveness of results frameworks, but moreremains to be done. IEG is increasing its support ofresults-oriented M&E capacities in client countriesand refocusing its high-intensity support in a fewtargeted countries, while maintaining less-intensive support to a broader range of countries.

Greater emphasis is needed on applied learning andreal-time use of evaluation findings to improve Bankperformance.

The AROE survey queried clients about thequality, use, and influence of IEG evaluationsand found that IEG provides good services tothe Board. However, task team leaders are theleast likely of the groups queried to report thatthey incorporate evaluation findings intoplanning, designing, and implementing Bankprojects and programs. IEG needs to find moreeffective and innovative channels to bring itsfindings to task teams in a way that would helpimprove design of future operations, and workwith Bank management to ensure relevantevaluative lessons are reflected in futureprograms.

Continued emphasis is also needed on thedevelopment of the newly established initiativeof providing quick-turnaround products for just-in-time knowledge to decision makers at thestrategic and operational levels.

IEG is increasing the follow-up, monitoring, andaccessibility of the Management Action Record, itsprimary tool for tracking management progress inimplementing IEG recommendations.

One of the ways IEG can influence the Bank’seffectiveness is through the recommendations itmakes to management in its evaluations. TheManagement Action Record (MAR) tracksprogress in incorporating findings andrecommendations in Bank operations andpolicy. The 2006 MAR contains 79 recommenda-tions. IEG rated the level of adoption of itsrecommendations by management “high” or“substantial” for 66 percent of the recommenda-tions. IEG and management agreed on the ratingof adoption for 65 percent of the recommenda-tions. A summary of the 2006 MAR can be foundin appendix H.

IEG has a critical role in the workings of the resultsagenda in the World Bank Group.

The AROE includes a results framework for IEGthat identifies its contribution to the Bank’sresults agenda. Within that framework, IEG’soutputs are the findings, lessons, and recommen-dations from its evaluations and evaluation

x i i


capacity support in client countries. Outcomesare twofold. First, for accountability, the evalua-tions are used by the Board to fulfill its oversightfunctions. Second, Bank management—as wellas countries—are to use the recommendationsfrom evaluation in policies and procedures.

Even if they are difficult to measure and target,greater attention needs to be given to theseoutcomes of evaluative work. There has beenprogress on this count. For example, earlyfindings from the recent evaluation of the Bank’sprimary education assistance (IEG 2006c) wereincorporated in the 2005 Education SectorStrategy Update. The trade assistance evaluation(IEG 2006a) is being used to recalibrate theBank’s approach to trade, with pilot initiativesgetting under way in several Regions. The recentevaluation of natural disaster assistance (IEG2006d) made special efforts to provide usefulfindings to operational staff dealing with currentevents as well as to inform emerging interna-tional responses.

Reviews of Country Assistance Strategy Comple-tion Reports (CASCRs) could provide inputs intothe preparation of new CAS. For learning, IEGproducts are aimed at improving the Bank’spolicy advice and program and project design:all parties can do much more to strengthen thislearning agenda. The final outcome for IEGoutputs is the use of its evaluative knowledge toimprove the effectiveness of Bank operations.

Recommendations for Management� Building on progress achieved so far in ad-

vancing the results agenda, agree on a three-year action program supported by appropriatefunding for the implementation of the nextstage, differentiating new allocations and re-deployed resources. Critical action items wouldbe the following:• Support country directors and country teams

in their efforts to refine and use resultsframeworks at country and sectoral levels to

manage country programs. Similar effortsneed to be made to strengthen the resultsframeworks of thematic and global programs.

• Assess the effectiveness of (self-) evalua-tion approaches during project and coun-try program implementation and provideguidelines to staff on their use.

• Strengthen incentives and accelerate aresults-oriented training and communica-tions program for management and staff toencourage use of M&E information.

• Identify and support in-depth learning op-portunities to develop and use results-basedapproaches with task teams, particularly inchallenging country environments and incomplex, multisectoral settings.

� Provide support to task teams and technical ad-vice to countries that intend to institutionalizeM&E systems to strengthen the collection anduse of performance information in order to en-hance the countries’ capacity and demand tomanage for results.

Recommendations for IEG-WB� To further strengthen IEG’s contribution to

the workings of the results agenda in the WorldBank Group, IEG should continue to follow itsown results framework and monitor it throughthe AROE. Its focus on the usefulness of eval-uation findings for its core audiences shouldbe enhanced: for the Board for oversight, formanagement through the incorporation ofrecommendations into Bank policies and strate-gies, for Bank staff through the use of evalu-ation findings for policy advice to countrypartners and in project design, for external

partners through the use of evaluation find-ings to improve their programs and policies,and for the countries more broadly. In play-ing this role, IEG should: • Improve the timeliness of its evaluations.• Strengthen the operational relevance of the

findings.• Increase access to and exchange of the

lessons.

E X E C UT I V E S U M M A RY

x i i i

x v

Summary of ManagementResponse to IEG Recommendations

IEG recommendation

1. Building on progress achieved so far in advancing the results agenda,

agree on a three-year action program with a corresponding budget to

fund implementation of the next stage, differentiating new allocations

and redeployed resources. Critical action items would be:

• Support country directors and country teams in their efforts to re-

fine and use results frameworks at country and sectoral levels to

manage country programs. Similar efforts need to be undertaken

to strengthen the results frameworks of thematic and global

programs.

• Assess the effectiveness of (self-) evaluation approaches during

project and country program implementation and provide guidelines

to staff on their use.

• Strengthen incentives and accelerate a results-oriented training and

communications program for management and staff to encourage

use of M&E information.

• Identify and support in-depth learning opportunities to develop and

use results-based approaches with operational teams, particu-

larly in challenging country cases and in complex multisectoral

settings.

Management response

At present, management does not agree that new budget allocations are

required to implement the results agenda. A results focus is the responsi-

bility of country teams and task teams and is to be included in the formu-

lation of RBCASs and task preparation and task supervision budgets.

Management regularly monitors the progress of the work on the results

agenda and will continue to formally report on progress to CODE every two

years. The most recent progress report includes a specific action plan that

is now being implemented. A key step forward has been the formation of

a Bank-wide Results Steering Group (RSG), which is now functioning. The

RSG is responsible for tracking progress on the action plan articulated in

the March 2006 progress report and for promoting communication and co-

ordination across network and Regional level efforts. An early task for the

RSG is to review and assign priorities for implementing the action plan and

to work with operational management teams to ensure that these priori-

ties are included in Regional and network budgets and work programs. If

this process identifies major budget pressures that go beyond what can be

reasonably funded through reallocation, management would discuss this

issue with executive directors using established budgetary procedures as

outlined in the recent budget document approved by executive directors.

Management is pleased to report progress on other critical actions iden-

tified in this recommendation:

• Regional Quality Teams are supporting country directors and country

teams in their efforts to refine and use results frameworks at country

and sectoral levels to manage country programs. As an example, the

Latin America and Caribbean Region together with OPCS carried

(Continues on the following page.)

IEG recommendation Management response

out during fiscal 2006 a regional portfolio review of results orienta-

tion in operations that was backed by regional management and pre-

sented to country directors and country and sectoral teams. Results

frameworks at thematic and global program levels are also being

strengthened. Six new results frameworks are being developed as part

of either sector strategy updates or the 2006 Sector Strategy Imple-

mentation Update: health, nutrition and population, environment,

forestry, social protection, finance, and transport. Results frameworks

in global programs are also being strengthened. Each vice presiden-

tial unit is now explicitly accountable for periodically reviewing and

adjusting global programs to ensure that they focus on results. New

procedures will allow the Bank to track its global programs from their

start through to evaluation and impact assessment.

• The Country Director/Country Management Peer Learning Event in

September 2006 will include a specific discussion of the results agenda

to identify methods for accelerating progress in enhancing staff un-

derstanding of how to construct appropriate results chains at the coun-

try level. Management will work to establish an analogous peer learning

structure for sector directors and sector managers during fiscal 2007.

• Management does not see the need to issue additional guidance to as-

sess the effectiveness of (self-) evaluation approaches during project

and country program implementation, as there are mechanisms and

processes already in place for this purpose. The Quality Assurance Group

reports on effectiveness of quality of supervision during project im-

plementation, and CAS Progress Reports report on evaluation of coun-

try program implementation. The self-evaluation in a CASCR is now

validated independently by IEG. In addition, country program reviews

(CPRs) have become another key instrument to assess the effectiveness

of (self-) evaluation approaches during project and country program im-

plementation. They also strengthen incentives and have helped to ac-

celerate a results-oriented training and communications program for

management and staff to encourage use of M&E information. During

fiscal 2006 OPCS issued a review note of some 50 country CPRs pro-

viding guidance to teams on modalities and current practice of results-

oriented country program reviews. The note examines the evolution of

Country Portfolio Performance Reviews (CPPRs) and their growing im-

portance as a management tool and a tool for policy dialog. Going for-

ward OPCS will help country teams strengthen CPRs through guidance

to individual teams and facilitating cross-country learning.

• Management is assessing the experience to date with several inno-

vations in the application of the rapid results approach to accelerating

implementation of the results agenda. That assessment, to be completed

by December, will then feed into updating the results learning program.

x v i


IEG recommendation

2. Provide budgetary support to task teams and technical advice to coun-

tries that intend to institutionalize M&E systems to strengthen the col-

lection and use of performance information in order to enhance the

countries’ capacity and demand to manage for results.

Management response

Management will consider its agreed actions complete with the functioning

of the RSG (accomplished), the implementation of the changes noted

above for global programs (in progress), and the successful implementa-

tion of the planned learning activities in fiscal 2007. Management will re-

port on the completion of these activities in the context of the regular

progress report to CODE in early calendar year 2008.

Management does not agree to provide special budgetary support to task

teams (see above) but does agree that greater country support will be use-

ful. Management is working to identify ways, likely through partnerships

with other donor agencies that are facing analogous challenges, to pro-

vide greater funding support to countries that are working to improve their

capacity to manage for results. Management will report on progress in the

context of informing Executive Directors on the upcoming Roundtable on

Results to be held in Hanoi in February 2007.

S U M M A RY O F M A N AG E M E N T R E S P O N S E TO I E G R E C O M M E N DAT I O N S

x v i i

x i x

Chairman’s Summary:Committee on DevelopmentEffectiveness (CODE)

On July 19, 2006, the Committee on Develop-ment Effectiveness (CODE) considered the 2006

Annual Report on Operations Evaluation

prepared by IEG and the draft managementresponse.

BackgroundThe 2005 AROE assessed the use and usefulnessof monitoring and evaluation information toimprove results and enhance Bank effectivenessat the country level. In the Management ActionRecord (MAR), one of the recommendations formanagement was to establish clear lines ofaccountability for the follow-up and oversight ofadopted recommendations and to make manage-ment responses more action specific to includetimelines for implementation. For IEG, therecommendation was to institute a formal systemfor tracking and managing its recommendations.The Committee supported the proposedchanges to the MAR, including a pilot three-yearexpiration criterion for MAR recommendationsfrom fiscal 2006 onward. On March 8, 2006,CODE discussed two draft documents preparedby management that are relevant to the AROE:Accelerating the Results Agenda: Progress and

Next Steps (World Bank 2006b) and “Updating theBank’s Operational Policy on Monitoring andEvaluation” (World Bank 2005d).

Report FindingsThe 2006 AROE found that progress had beenmade on the 2003 action plan. Procedures werein place and results frameworks were being usedin projects and RBCASs, but their effectiveness asa monitoring tool remained in doubt, and morework was needed at the sector and globalprogram levels. Bank managers and staff facedchallenges in managing for results such asconceptual difficulties in developing resultschains, lack of client capacity and interest,competing priorities in the Bank, and lack ofappreciation for the use of performance informa-tion and evaluations for management decisionmaking. Moreover, disconnects in outcomeratings between country and sector policyoutcomes and project outcomes pointed to a gapin understanding contributing factors to results.The AROE recommended establishing fullyfunded priority actions. Assessing IEG’s effective-ness, the report recommended that IEG’s resultsframework should continue, and even enhance,its focus on the use of evaluation findings.

Management ResponseManagement broadly agreed with the mainthrust of IEG recommendations, which are inline with its own findings and the priorities foraction set out in its February 2006 update to

CODE, with one major exception. It noted that aresults focus was the responsibility of countryteams and task teams and needed to be includedin the formulation of RBCASs and in taskpreparation and task supervision budgets.

Overall ConclusionsSpeakers welcomed the opportunity to reviewmanagement’s and IEG’s actions to strengthenManaging for Development Results. They notedthe findings that the Bank has strengthened theresults focus of its operations, while majorchallenges lie ahead. The Committee wasgratified to observe a broad agreement betweenIEG and management on the main findings andrecommendations, which it also generallyendorsed. It also accepted IEG’s clarificationthat it was not suggesting additional budgetaryallocations but rather stronger prioritization indeployment of resources, a position closer towhat was enunciated by management in itsresponse. There were also a few questionsregarding specific statistics reported, includingsome from the client survey and the MAR.

Several members concurred with IEG thataccountability was as important as incentives andresources are to promoting the results agenda butalso to achieve better results. Members alsoendorsed greater IEG efforts to enhance theusefulness and application of IEG products, partic-ularly by operational staff. In this regard, the timingof Country Assistance Evaluations (CAEs) vis-à-visthe CAS was cited as an area where some improve-ment might be possible. A number of speakersraised questions about the effectiveness of theresults architecture being put in place, includingprioritization of areas and statistical capacity. Therole of the Board in raising results issues whendiscussing Bank’s operations was highlighted.

Next StepsThere was agreement about a two-year cycle forAROE; the report will be considered by theCommittee, followed by a Board discussion.Given the likely Board agenda in August, theCommittee recommended that the Boardconsider this year the AROE and the draftmanagement response on a streamlined

procedure (that is, without discussion). (Initiallya Board meeting was scheduled for August 22).

The following main issues were raised duringthe meeting:

Results architecture and results framework. TheCommittee welcomed the progress of the Bankin implementing the results agenda. Yet severalspeakers had questions about the effectivenessof the results architecture being established. Amember expressed concerns about possibledistortions favoring more measurable activitiesand against more complex priority areas such asgovernance or rural development. The robust-ness of the system of indicators, given weakstatistical capacity in many countries, was alsoquestioned. A member stressed that countrieswere at the center of the effort to strengthen thefocus on results and that a substantive demandfrom governments in this respect was instru-mental. Management noted that there was

limited demand from governments due in part

to data gaps at the country level (yet these gaps

would not be so expensive to fix) and lack of

relevance in the decision-making process.

Management found that there was a risk of

distortions to only wanting to do things that are

measurable.

Some speakers commented on “disconnects”between country and sector policy outcomesand project outcomes. In this regard, efforts tofocus on results were perceived as uneven andinconsistent among and within countries,Regions and networks, rendering it difficult toassess the Bank’s overall contribution todevelopment results. Two members noted thatframeworks appeared to be better developed inInternational Development Association coun-tries than in International Bank for Reconstruc-tion and Development countries, althoughissues of quantity and quality of data were raised.IEG noted that some reasons for “disconnects”

and perception of inconsistency were the

difficulty of indicators and measurement due

to the political economy of reform or the

problem of attribution of a single straightfor-

ward intervention on a range of outcomes. In

x x


response to comments about the importance ofharmonization, management noted that the

multilateral development banks were entering

the second year of doing a comparative assess-

ment of their respective works on results

(COMPAS). One member urged management toredouble their efforts to monitor and evaluateresults of trust fund activities. Management

stated that they are working with trust fund and

global programs to increase their results focus.

Members stressed the need for staff to betterunderstand the results framework as a tool tolink Bank operations with outcomes in countryprograms and projects, and to strengthen clientcapacity to manage for results, particularly low-income countries. Further analyses on projectoutcomes, baselines, and targets, especially inRBCASs and Poverty Reduction Strategies, weresuggested. In this regard, consideration shouldalso be given to time (time needed toimplement the results framework and the timelag between project completion and outcome).Staff from the Africa Region provided informa-

tion about efforts in promoting the results

agenda and building capacities on results.

Organizational issues and incentives. Memberscommented on the need to change the organiza-tional culture, align incentives and rewardstructure, and disseminate successful experi-ences for learning purposes. Management

replied that the current culture, in which staff

perceive greater incentive to report up than

across, will be addressed by the development of

the Results Monitoring and Learning System. Theneed for a better communication strategy wasalso mentioned. More analysis was sought aboutoverload, time pressures, availability of resources,and difficulties in developing the results chain.Speakers and management shared the Director-General, Evaluation’s views regarding the key roleof accountability in implementing the resultsagenda. One member sought information onrecent activities of the RSG. Management

responded that the group has started to function

and is expected to play a key role in increasing

attention Bank-wide to results and in communi-

cating and consolidating gains across the Bank.

Funding priorities. IEG proposed designatedfunding for implementing the results agendadifferentiating new allocations and redeployedresources. Some members commented onspecial budgetary support to task teams. TheCommittee took note of IEG’s clarification thatit was not asking for additional budgetary alloca-tion but rather stronger prioritization in deploy-ment of resources.

Recommendations for IEG. The Committeewelcomed the AROE’s recommendation onthe role of IEG in promoting the resultsagenda. There were other comments on theneed to improve timeliness of IEG evaluations,including timing of CAE and CASCRs; toanalyze the relative strength of resultsframeworks by region and sector, particularlyto help operationalize the findings; and totrack and publish the results of sectoral,Regional, or project analysis as well as theirtrends to facilitate learning. One member feltIEG needed to focus on new and ongoingoperations. Another member commended IEGfor the efforts to develop its own resultsframework and noting the difficulties ofconstructing measures that link final outcomesto IEG interventions. Regarding the clientsurvey, one member expressed concern aboutthe perceived decline in the quality and depthof IEG’s analysis and recommended that IEGanalyze the trend. IEG clarified that surveys

did not show such trends in regard to external

clients but that this was an internal issue that

had to be sorted out.

Regarding the content of the AROE itself, amember made some suggestions to add someflesh to the report’s substance and stressed theneed to use the AROE to help promote “out-of-the-box” thinking on the results focus by explor-ing such topics as how to better link theevaluation findings to administrative budget andhuman resources management and how theincreased use of innovative work and experi-mentation—for example, conditional cashtransfer—can foster the results agenda. IEG

promised a more specific AROE in two years

and took note of the suggestions.

C H A I R M A N ’ S S U M M A RY: C O M M I T T E E O N D E V E L O P M E N T E F F E C T I V E N E S S ( C O D E )

x x i

Management Action Record. A member expressedconcern about the statistic that the adoption ofsix percent of IEG’s recommendations wasnegligible. This member also warned against thethree-year expiration criterion acting as anincentive for delaying implementation of the

recommendations. IEG explained that this

figure reflected the relatively high number of

new recommendations entered in the MAR in

2006. Members took note that a summary of theMAR will be posted in the Bank’s external Website for disclosure.

x x i i


Pietro Veglio

Chairman

Chapter 1: Evaluation Highlights

• The Bank has strengthened the results focus of its operations.• This strengthening has implications for the way the Bank monitors

and evaluates its operations.• Results frameworks at the project and country levels form the basis

for monitoring and evaluating results.• Independent Evaluation Group evaluations have strengthened their

assessments of results-oriented monitoring and evaluation.

3

Introduction

The Annual Report on Operations Evaluation (AROE) fulfills the man-date of the Independent Evaluation Group-World Bank (IEG-WB) to as-sess the progress, status, and prospects for monitoring and evaluating

the development effectiveness of World Bank activities. The objective of the2006 AROE is to assess the extent to which the Bank’s monitoring and evalu-ation (M&E) systems provide Bank staff with the information they need to bet-ter manage for results.

11

Using M&E Information for ResultsThe Bank has been strengthening the resultsfocus of its operations in recent years. The 2002International Conference on Financing forDevelopment in Monterrey and the jointstatement issued by the heads of the multilat-eral development banks (MDBs) highlighted theneed to better measure, monitor, and managefor results.

In 2003, the Bank started implementation of itsManaging for Development Results Action Plan.Within the Bank, a key element is the Results-Based Country Assistance Strategy (RBCAS),which was piloted and then mainstreamed in 2005.The RBCAS includes a results framework that linkscountry development goals with Bank operationsand forms the basis for Country AssistanceStrategy (CAS) M&E. The Bank also introducedresults frameworks into projects and sector strate-gies: staff outline their expected outcomes andexplain how they are to be achieved.

The introduction ofresults frameworks at theproject and countrylevels has implicationsfor the way the Bankmonitors and evaluatesits operations. Monitoring would includeoutcomes as well as outputs and inputs. Thepurpose of these results frameworks and M&Esystems is to provide information that can drivemanagement decisions. That is, what effect is theproject or country program having on theplanned outcomes? This would affect the way theBank collects, analyzes, and uses monitoringinformation.

IEG’s evaluations have been strengthening theirassessments of results-oriented M&E. More tar-geted attention is beinggiven to building evalua-tion capacity among in-

The results-based Country

Assistance Strategy has

been mainstreamed since

2005.

Results frameworks are

now used at the country

and project levels.

ternal and external clients. Effective communi-cation of IEG’s knowledge is a critical factor inincreasing the use and usefulness of IEGproducts, and IEG is making progress in increas-ing awareness of its products.

Overview of the 2006 AROEThe 2006 AROE assesses the Bank’s results focus inM&E. Chapter 2 discusses the implications of theBank’s results agenda on monitoring and self-evaluation of Bank operations. Chapter 3 reviewsIEG-WB’s effectiveness, presents a results frameworkfor IEG, and examines how IEG can increase the useand usefulness of its evaluations. Chapter 4summarizes the findings and presents recommen-dations for Bank management and IEG-WB.

The 2006 AROE draws on reports fromOperations Policy and Country Services (OPCS)and the Quality Assurance Group (QAG), as wellas IEG evaluations and a review of recentlyapproved RBCAS. The evaluation team alsoconducted focus group interviews with Banksector managers, task team leaders, and countrycoordinators; structured interviews with countrydirectors and members of the Board’s Commit-tee on Development Effectiveness (CODE); anda client survey of key internal and externalaudience groups for IEG’s recently completedevaluation studies. The findings are included asappendixes to this report.

4



• RBCASs include country results frameworks that clarify the Bank’scontribution to achieving country goals.

• Results frameworks are being used in projects and RBCASs, but theireffectiveness as a monitoring tool remains in doubt.

• Challenges to results-based monitoring include the following:– Perception that monitoring is about inputs and outputs rather than

results– Conceptual difficulties in developing results chains– Lack of client capacity and interest– Competing priorities in the Bank– Lack of appreciation of the utility of performance information

and evaluations in management decision making.• A greater focus on results increases the need for self-evaluation

during implementation.

7

Monitoring and EvaluatingResults in Bank Operations

This chapter reviews the implications of the Bank’s results agenda on mon-itoring and for the self-evaluation of Bank operations. A descriptiveoverview of the Bank’s evolving M&E framework at the activity (lend-

ing and analytical and advisory activities), country, sector, and global levels isprovided in appendix A. This chapter focuses on Bank lending operations andthe CAS—two areas where the Bank has systematically introduced a strength-ened results focus over the past few years.

To the extent that global programs with Bankinvolvement produce results at the countrylevel, they can be considered one of the instru-ments in the Bank’s country programs. Evalua-tions are done both by Bank operational staff(self-evaluations) and by IEG. IEG reviews andvalidates the Bank’s self-evaluations andconducts its own independent evaluations. Thischapter covers the Bank’s self-evaluation; IEGactivities are discussed separately in chapter 3.

Recent Developments in MonitoringThe Bank has taken a number of steps over thepast two years to strengthen the links betweencountry development goals and individual Bankoperations. At the project level, the Bankintroduced results frameworks for its investmentprojects in 2004. A results framework focuses onthe project development objective and interme-diate outcomes, both of which are to besupported by performance indicators. Theseindicators would be used to track progress

toward meeting thedevelopment objectiveand to make changes inthe project, if necessary,during implementation.The Bank introducedresults frameworks into its adjustmentoperations when it issued its new policy onDevelopment Policy Loans in 2004.

The Implementation Status and Results (ISR)Report was introduced in 2005 to replace theProject Supervision Report (PSR). One of theobjectives in introducing the ISR was tostrengthen the focus on outcomes. Monitoringis now given more prominence by making theperformance indicators part of the main report;previously it was an annex.

Task teams are required to specify the expecteddevelopment outcomes for economic andsector work (ESW), nonlending technical

22

A results chain links

inputs to outputs and

then to intermediate and

final outcomes.

assistance, and other analytical and advisoryactivities (AAA), although they are not requiredto establish results frameworks for them.

At the country level, RBCASs were adopted aspolicy in 2005. Critical elements of the RBCASare: (a) the country results framework, whichspecifies country development goals andoutlines the results chains that link individualgoals with CAS outcomes, intermediateoutcomes and milestones, and Bank operations;

and (b) the CAS Comple-tion Report (CASCR), aself-evaluation of theCAS that is validated byIEG. The country resultsframeworks attempt toclarify the Bank’s contri-bution to achievingcountry development

goals, thereby addressing a weakness found inprevious CASs, which discussed the country’s

objectives and Bank operations in detail but didnot establish the links between the two.

How a Results Chain WorksFigure 2.1 shows a sample country-level resultschain for agricultural development, outlining theexpected links between Bank projects, projectoutputs, intermediate outcomes, final outcomes,and the country development goal.1 It also showsthe results chain for individual operations.

For example, Bank funds channeled through arural development project (input) wouldproduce improved or new rural roads (output).This, in turn, would improve farmers’ access tomarkets and suppliers (intermediate outcome1), increase profitability (intermediate outcome2), and improve rural household incomes (finaloutcome). The results chain outlines how theproject is expected to contribute to the overallgoal of improving rural incomes and reducingpoverty (the country development goal).

8


Results frameworks focus

on objectives and

intermediate outcomes

and are supported by

indicators that track

progress.

Figure 2.1: Sample Results Chain for Agricultural Development

Rural households increase incomes

Farmers, smalland medium-size enterprises gain

access to working capital and

investment funds

Credit linesfor small andmedium-sizeenterprises,

farmersestablished

Rural roads and otherrural transportation

infrastructureimproved orconstructed

Irrigationsystems

improved orconstructed

Technicalassistanceoutreachprograms

for farmersput in place

Farmers receiveland titles Farmers, small and

medium-size enterprisesgain access to

markets and suppliers,reduce transport costs

Farmers increase productivity (yields)and diversify production

Inputs

Outputs

Final outcome

Government and Bank funding and support

Land Administration

LIL

• Rural Development I • Rural Finance

• Social Fund • Rural Development 1 • Rural Infrastructure

• Social Fund• Rural Development 1

• Rural Infrastructure

• Rural Development 1

• Rural Infrastructure

Farmers, small and medium-size enterprises increase profitability

Intermediateoutcomes

Governmentagriculture

pricingpolicies

improved

• Agriculture sector study

Studies andsystems

prototypes

Govt. capacityto issue land

titles improved

Source: Adapted from material from the Bank East Asia and Pacific Region.

Note: LIL = Learning and Innovation Loan.

The example in figure 2.1 shows that a singleproject could have several outputs andcontribute to multiple intermediate outcomes.For example, the same rural developmentproject could include a financing componentthat would provide farmers and small andmedium-size rural enterprises with access tocredit through formal financial institutions atterms more favorable than they would havebeen able to obtain from informal lenders.

Two or more projects could share commonoutputs and outcomes. For example, the Bank’scountry portfolio may include a rural creditproject that would address wider issues offinancial sector reform and financial institutionbuilding nationwide. The rural credit and ruraldevelopment projects may or may not overlap,because the latter would be focused on specificareas of the country. But it does point to theneed to ensure that both projects are wellcoordinated—for example, by sharing commonoutcomes and performance indicators. It alsoprovides opportunities to compare approachesand learn about what works and what does notand to identify “good practices.”

The example in figure 2.1 also shows possiblesynergies between Bank projects thatcontribute to a common intermediate out-come. For example, financial institutions mayrequire a formal title as collateral to providefinancing to farmers and small and medium-size enterprises. The land administrationproject, which aims to strengthen the govern-ment’s capacity to issue titles, would comple-ment the rural finance operations. Alter-natively, if it proves difficult to provide titles,other options for rural financing may need tobe explored.

Figure 2.1 shows another case of synergieswhere different projects provide target benefici-ary communities (in different geographic areas)with access to support for infrastructuredevelopment and extension services. Togetherwith an agriculture sector study, they wouldcontribute to increasing farmer productivity(one of the intermediate outcomes).

Developing results chainsfor individual operationsoften proves to be acomplex undertaking,because there may bemultiple causal chainsand sequencing patterns between Bankoperations and CAS outcomes, with manyintermediate steps. But simplified country-levelresults chains, such as the example shown, allow acountry team to identify synergies and gaps in itsoperations and to establish common intermediateoutcomes to which team members may jointlycontribute. Where other development partnersmay be involved, the results chains also helpidentify synergies and areas of common interest.

Possible Uses of Results FrameworksCountry directors and task team leaders (TTLs)can use the results frameworks to establish theextent to which theBank and individualBank operations can beheld accountable forproducing results. Thatis, the country directorsand TTLs make their accountabilities explicitthrough their choices of CAS outcomes andintermediate outcomes in the CAS resultsframeworks and the project developmentobjectives and intermediate outcomes in theproject results framework.

There are trade-offs. If the outcomes set are toomodest, country directors and TTLs may be seenas setting the bar toolow. If the outcomes aretoo ambitious, they maybe criticized for lackingrealism, because morefactors come into play inachieving the expected outcome, which makesattribution and Bank accountability unclear.Country directors and TTLs have to balance theease of achieving successful outcomes with therisks involved in addressing important butcomplex issues, such as governance, or workingin weaker policy and institutional environments,such as fragile states.

M O N I TO R I N G A N D E VA L U AT I N G R E S U LT S I N B A N K O P E R AT I O N S

9

Results chains can be

used to identify synergies

between Bank projects.

The frameworks can be

used to establish

accountability.

Nonlending activities also

define expected outcomes

but are not required to

have results frameworks.

The results frameworks include performanceindicators—with baselines and targets—for eachoutcome that can be used to monitor progress andassess success or failure on completion of the Bankoperation or country program. At the project level,the Bank has increased the importance and visibil-ity of these performance indicators by giving themmore prominence in the ISR (which replaced thePSR).2

The focus on intermedi-ate and final outcomesincreases the amount ofinformation to be col-lected during implemen-tation and to be used for

M&E. In addition to inputs (such as Bank loandisbursements) and outputs (such as the lengthof roads constructed), the borrower will berequired to monitor progress toward achievingagreed outcomes through the resultsframeworks (for example, the effect the roadshave had on farmers and small and medium-sizerural enterprises). While information on inputsand outputs may be available in house, collect-ing outcome information may require additionalexpenditures for surveys or other methods toassess the effect of the project on beneficiaries.

Information on what is happening or nothappening to expected outcomes would helpthe government and Bank managers and staffdetermine where to focus their efforts.Decision making in these situations is oftencomplex.

A lack of progress could lead to the provision ofadditional resources to correct problems, or to areduction in resources to cut losses. Faced withsuch challenges, managers and staff may need touse evaluative approaches, such as rapid assess-

ment methods, to get abetter understanding ofwhy progress towardoutcomes is not beingachieved as planned. Thiscould lead to more

decisions, such as to restructure a project orprogram.

In sum, managers and staff in the borrowercountries and the Bank could use the resultsframeworks as a management tool to do thefollowing:

• Outline how their operations would contributeto achieving country-level objectives.

• Communicate priorities and clarify expecta-tions and accountabilities with superiors andexternal stakeholders.

• Motivate their staff and teams by setting chal-lenging but realistic goals.

• Improve coordination by identifying com-mon outcomes to which other Bank opera-tions and development-partner programs couldcontribute.

In addition, Bank country directors could useresults frameworks to identify synergies andgaps in the Bank’s country program. They couldexplore better integration between country andproject results frameworks by sharing commonoutcomes and performance indicators whereappropriate.

With baselines and targets, the resultsframeworks would provide a monitoring systemthat could act as an early warning system to alertmanagers and staff to the risk of not achievingproject and country outcomes. When expectedoutcomes are not being achieved, self-evaluations could be used to explore thereasons for problems and lead to correctiveactions. The implications of the Bank’s resultsagenda for M&E are discussed in more detail inappendix B.

Current Status of Monitoring ResultsResults frameworks are being incorporated intoprojects and RBCASs, but their effectiveness as amonitoring tool remains in doubt. At theproject level, QAG found that consistency withcountry and sector strategies was strong inprojects approved in fiscal 2004–05, but therewas scope for improving the clarity and realismof development objectives in about 40 percentof the projects reviewed (World Bank 2006d). Inaddition, many projects (40–50 percent of thesample) lacked well-articulated results frame-

1 0


The focus on outcomes

increases the amount of

information needed for

M&E.

Such information is useful

in determining where to

focus additional effort.

works with baselines and systems for trackingand reporting progress.

Similar weaknesses were observed in the

results frameworks of ISRs. An OPCS reviewof the quality of results frameworks in ISRsprepared in fiscal 2005 found that 40 percent didnot contain any outcome indicators with baselinedata for either the project development objectiveor intermediate outcome indicators, and werenot considered to be acceptable (World Bank2006b). The review noted that there was littleevidence that the ISR is used as a document ofrecord for implementation or results reporting.A review of ISRs by selected networks found thatin fiscal 2005 many lacked baseline data foroutcomes—only 52 percent in rural develop-ment had such data, 50 percent in public sectorgovernance or rule of law, 47 percent in health,and 32 percent in water supply and sanitation—which limited their usefulness for monitoringprogress. The ISR was introduced in 2005, sothese issues may reflect problems of the transi-tion from the PSR to the ISR.

At the country level, 25 CAS resultsframeworks prepared during fiscal 2005–06were reviewed in preparation for this AROE toassess their quality and usefulness as a tool tomanage Bank country programs. A four-pointrating system, with level 0 the weakest and level3 the strongest, was used to assess:

• The strength of the links among country de-velopment goals, CAS outcomes, milestones,and the Bank program

• The usefulness of the performance measuresfor M&E (see box 2.1 for an example).

The analysis confirmed weaknesses similar tothose at the project level: poorly articulated resultschains and lack of indicators with baselines andtargets. These weaknesses would lessen the useful-ness of the results frameworks as country programmonitoring and management tools. The results ofthe analysis are presented in appendix C.

The review found that results chains are betterdeveloped in International Development Associ-

ation (IDA) countriesthan in InternationalBank for Reconstructionand Development(IBRD) countries. Over-all, about one-quarter(24 percent) of the CASsreviewed have strong (level 3) and another 44percent have moderate results chains (level 2).While 85 percent of CASs in IDA countries havemoderate to strong overall results chains (levels2 and 3), only 50 percent of CASs for IBRDcountries do (figure 2.2).

Countries with stronger results chains have CASoutcomes and milestones that are better formu-lated. One factor that may contribute to this differ-ence is that IDA countrieshave experience workingwith Poverty ReductionStrategies (PRSs), wherecountry-level perform-ance measures are set and tracked as part of theoverall development assistance program. In IBRDcountries, where the Bank may account for asmaller share of the overall investment toward acountry goal, it may be more difficult to articulatethe Bank’s contribution in a results framework andto set performance measures and milestones. Itshould be noted that better plans (resultsframeworks) do not necessarily translate intobetter achievement of results without governmentbuy-in.

Lack of adequate baselines and targets contin-ues to be a weakness, as identified in last year’sAROE.3 Only 9 percent of the CASs reviewed hadbaselines for more thanhalf of their CAS out-comes (levels 3 and 4;figure 2.3). Just underhalf (45 percent) had settargets for more than half of their CASoutcomes. This raises questions about theavailability of data and the usefulness of theseCAS outcomes for M&E.

The Bank does better at identifying milestones4

that are relevant to achieving CAS outcomes,


1 1

Frameworks with

baselines and targets

could act as an early

warning system for

projects.

The effectiveness of

results frameworks

remains in doubt.

Frameworks could be

improved, and indicators

are often inadequate.

although their precision could be improved(that is, they could describe more clearly what isto be achieved; see figure 2.4). Only 36 percentof the CASs, however, had specific, time-boundtargets for more than half of their milestones.

The number of indica-tors remains high. Onaverage, there were 24country developmentgoals, 41 CAS outcomes,and 60 milestones percountry. Although thereis no correct number of

indicators, large numbers pose problems for datacollection and analysis. A smaller number ofindicators would not only be more manageable

but could also help focus the dialogue with thegovernment on the critical elements of thecountry program.

A comparison of the number of CAS outcomeswith the fiscal 2006 country budget (as a proxyfor program size) showed that the largerprograms had more indicators (figure 2.5). Butit also identified some outliers that would becandidates for review and simplification. A largenumber of CAS outcomes relative to the overallsize of the country program may indicate a needfor greater selectivity.

When results frameworks are developed, eitherat the project or the country level, a conflict mayarise between managing for results, which

1 2


A strong country results framework, which would score at least3 on the four-point rating scale for all criteria, would establisha robust results chain, identifying clear links among:

(a) Specific country development goals (b) CAS outcomes (identifying the nature and extent of the

Bank’s contribution to achievement of the goal) (c) Intermediate outcomes, actions, or outputs (that is, mile-

stones) that serve to mark progress toward achievementof the CAS outcomes

(d) Bank inputs (its lending and nonlending services).

In addition to these identified links, country developmentgoals and CAS outcomes would be accompanied by performanceindicators that are measurable and include baselines and spe-cific, time-bound targets in a strong results framework. Milestonescould be actions (such as passage of key legislation), in whichcase they should be defined in such a way that their relevanceto the CAS outcome is clear. Finally, the CAS outcomes and in-dicators would allow the government and the Bank to focus onkey elements of the Bank’s country program and conduct a reg-ular dialogue on progress made, based on the performanceindicators.

The review identified several CASs with well-developed resultsframeworks, although none scored high in all categories.

The Burkina Faso CAS results framework, for example, had astrong results chain that described how the CAS outcomes weresupported by the activities. It also had a few key milestones to allowfor monitoring and management against each CAS outcome.

Performance measures were included for country goals andCAS outcomes. Most were well constructed and included base-lines and targets. For example, a CAS outcome of “reduced riskand increased revenues for rural households” is supported byfour performance indicators, one of which is “land tenure: 50 per-cent of male applicants and 50 percent of women applicants ob-tained land titles or equivalent in six pilot provinces.” A milestoneof land tenure is “revised implementing decrees for property man-agement are adopted.”

However, the Burkina Faso CAS results framework had 54country goals and 65 CAS outcomes. These numbers could limitits usefulness in managing the country program.

Poor results frameworks—those scoring at level 1 in many ofthe categories—had poorly developed results chains. For exam-ple, milestones were included as country or CAS outcomes or, con-versely, higher-level CAS outcomes were included as milestones.Country and CAS outcomes were poorly defined and not measur-able, such as “involve all sectors at all levels to scrutinize gov-ernment projects.” Where performance indicators were identified,many did not include baselines or targets, so they could not be usedto track progress.

Box 2.1: What Constitutes a Strong Results Framework?

Large numbers of

indicators, currently

common, pose problems

for data collection and

analysis.

Source: IEG data.


1 3

Figure 2.2: Results Chains Better Developed in IDA Countries than in IBRD or Blend Borrowers

Source: IEG data.

Note: CAS = Country Assistance Strategy.

Source: IEG data.

Figure 2.3: Country Assistance StrategiesTend to Lack Adequate Baselines and Targets

Figure 2.4: Precision of Milestones Could Be Improved

Source: IEG data.

Source: IEG data.

Figure 2.5: Larger Programs Tend to HaveMore Indicators, Outliers May Need Review

emphasizes learning on what results are beingachieved and why and using this information fordecision making, and accountability forresults, where performance information isused to hold agencies, managers, or staffaccountable (Binnendijk 2001). A balance needsto be struck between risk taking—tacklingcomplex, challenging issues such as gov-ernance—and the likelihood of success, that is,focusing on outcomes that are achievable.

Similarly, too strong an emphasis on measure-ment may lead to projects and country programsfocusing on areas that can be measured. Forexample, the ready availability of infant mortalitydata should not be a determining factor inwhere the Bank focuses its intervention in acountry’s health sector.

These challenges tend to be country-specific. Dataavailability would differ significantly between amiddle-income country and a fragile state.Similarly, what could be considered an easilyachievable outcome in a middle-income countrymay pose a significant challenge in a fragile state.This reconfirms the importance of the country asthe unit of account in Bank operations and theneed to strengthen CAS results frameworks.

Challenges to Developing Results-BasedMonitoringThe 2006 AROE conducted focus groups andstructured interviews with 66 Bank countrydirectors, country coordinators, sector man-agers, and TTLs to examine why Bank staff find itdifficult to develop a results-based monitoringsystem (results framework) and use it to managefor results. The key findings are summarized inbox 2.2, and additional information on the focusgroups and interviews is available in appendix D.

Focus group participants reported difficulties indeveloping results chains. Often they setambitious project development objectives and

had difficulty linkingthem to the projects:“Once you are unrealis-

tic in project conception,

everything that follows

from it is also overstated.” Participants also feltoverburdened by the emphasis on quantitativedata: “There is a tyranny of the measurable here

that we have to be very careful of.” QAG alsonoted that M&E problems are frequently relatedto difficulties in defining and monitoring a set ofoutcome/impact indicators.5

Participants in the 2006 AROE focus groupsmentioned lack of resources as a constraint todeveloping M&E systems during project prepara-tion, establishing baselines, and systematicallycollecting data during project implementation.Insufficient budgets were often cited as an issue.In a previous QAG assessment of supervision infiscal 2001–02, only 31 percent of TTLs respond-ing felt that they had sufficient resources toadequately implement an outcome-focusedsystem for supervision (World Bank 2003b).

A review of Regional budgetary expenditures forfiscal 2002–05 found that Bank budget expendi-tures for supervision within the Regions hadincreased in constant terms in line with theincrease in the Regional budget (table 2.1).Supervision received more resources thaneither lending or country and ESW andaccounted for about 27 percent of total clientexpenditures within the Regions. Supervisionexpenditure per project increased in real termsduring the period.

This suggests that the issue may not be the lack ofBank budget for M&E per se, but the manydemands on both financial resources and theTTL’s time and attention. Participants explainedthat they often postponed the preparation of M&Earrangements until after project approval becauseof the need to reduce project preparation time:

You need to do it (develop monitoring

and evaluation system) quickly, you need

to do it before (appraisal), and then you

are running out of time because there are

all these pressures to deliver.

Some focus group participants noted thatmonitoring gets less priority duringimplementation:

1 4


Focus groups reported

difficulties in developing

results chains.

The rationing of resources begins to

happen here [during implementation].

Monitoring and evaluation is an

expensive thing, so the monitoring and

evaluation guy gets bumped off the

train.

The Bank’s fiscal 2005 budget document6 identi-fied safeguards and fiduciary requirements ascost drivers that lead to increased resourceneeds within the Bank.

A closely related issue is the lack of capacity andthe cost to the borrower of developing systemsto collect the necessary performance data andthe borrower’s reluctance to bear that cost.Increasing the focus on monitoring outcomesduring implementation may require new ways ofcollecting data and add to borrower costs.

Participants in the focusgroups saw lack ofborrower buy-in as one ofthe key constraints todeveloping and using aneffective outcome-based monitoring system:

All clients want is to get the job done. They

should do monitoring and evaluation,

but it is not their priority because they all

know that their job ends with the success-

ful completion of the project, and then

they want to go to the next project.

Participants also recog-nized the need to createdemand for monitoringinformation and to buildborrower capacity:


1 5

Managing for Results and MonitoringJudging by the focus groups, managers and staff seem to un-derstand the overall objectives of the Bank’s Managing for De-velopment Results agenda and its implications for Bank operations.Managing for results means aligning project and country programoutcomes with country objectives, focusing on outcomes ratherthan outputs, and measuring and being accountable for outcomes.

Based on interview findings, policy makers (Board members,country directors) see the results frameworks as a way of setting over-all strategic directions and monitoring progress toward these strate-gic goals. TTLs accept the results frameworks as part of their work,but disagree about their usefulness. Country directors, sector man-agers, and TTLs struggled with technical aspects of managing forresults—developing results chains, positioning Bank operations ap-propriately in the chain, and coming up with performance indicators.

The focus group identified the following challenges to managingfor results:

• Perceptions: Many people still see implementation and mon-itoring as being about inputs and outputs. That is, results willcome at the end of an operation. M&E is still seen as gen-erating data, especially outcomes, whose use and usefulnessare not necessarily clear or seen as a high priority.

• Conceptual difficulties: Managers and staff struggle to de-velop results chains— that is, to link higher-level objectivesto Bank operations and identify appropriate performance in-dicators. They also have difficulty positioning a project withinthe chain. They may come up with overly ambitious projectdevelopment objectives (PDO) or position a PDO too high upthe results chain, and then struggle to develop a resultsframework around that PDO. They also had difficulty comingup with numerical indicators, commenting that not all im-portant results could be measured.

• Client capacity and buy in: Lack of client capacity and in-terest is often given as a constraint to carrying out M&E. Itis linked with lack of clarity on the use and usefulness of mon-itoring data, especially of outcomes, during implementa-tion. This has led to difficulty in collecting data, includingbaseline data.

• Bank pressures: TTL overload, time pressures, and lack of re-sources were cited as constraints. TTLs tend to see M&E asa lower priority than operational activities (and also moreproblematic, given difficulties with results chains, indica-tors, lack of clarity on use, and client buy-in). When devel-oping a project, M&E was often left until the last minute,given the pressure to reduce project preparation time.

Box 2.2: What Are We Hearing from Managers and Staff?

Source: Focus groups and structured interviews (appendix D).

Focus group participants

pointed to a lack of

resources as a constraint.

The issue may be more a

matter of competing

demands on staff time

and attention.

I have managed to convince clients to

monitor. I have one client who became

creative about coming up with new indica-

tors and using them to convince the prime

minister to back her on certain things.

Finally, lack of incentives is often given as aconstraint to managing for results. The 2004AROE (IEG 2004b) noted that “changes inguidelines and templates are important, but notsufficient for ensuring a systematic adoption of aresult-oriented approach in an organization.” Itrecommended, among other measures, thatBank management “review and strengthenincentives such as managerial recognition, careeradvancement, and reward systems to promoteresults-oriented monitoring and evaluation,managing for results and lesson-learning.”

The Bank has takensteps to incorporate re-sults explicitly into itsmanagement policies.The new staff-promotioncriteria for level GH

require a “sustained record of achievingoperational impact/results on the ground.”Performance on portfolio implementation is anelement in the evaluation of all operationalmanagers. The achievement of results is one ofseveral criteria that go into assessing perform-ance. Further analysis is required to establish theextent to which these policies have affected staffand manager behavior.

While formal incentives may be in place,informal incentives may wield strongerinfluence. Focus group participants mentionedthe pressures to deliver projects as quickly aspossible. OPCS also noted that “the consistentmessage to task teams is that delivery impliesdelivering a project to the Board rather thandelivering results for clients” (World Bank2006b).

Messages from Bank senior management are stillseen as reinforcing this perception. Forexample, an internal Bank news article on fiscal2005 operational results highlighted theincrease in IBRD/IDA loans, grants, and guaran-

1 6


Fiscal year2002 2003 2004 2005

Amount (US$ millions, constant fiscal 2005 prices)

Project supervision 153.5 159.7 163.3 168.7

Lending 124.2 118.6 151.8 138.9

Country, economic and sector work 131.6 140.0 148.2 149.1

Other country services 59.0 61.7 69.9 66.0

Sector and global services 81.5 87.0 91.9 91.6

Total client services 549.9 566.8 625.2 614.2

Share (percent)

Project supervision 27.9 28.2 26.1 27.5

Lending 22.6 20.9 24.3 22.6

Country, economic and sector work 23.9 24.7 23.7 24.3

Subtotal 74.4 73.8 74.1 74.4

Supervision/projecta (US$ thousands) 75 78 79 82Source: World Bank data.

Note: Direct costs. The Bank moved to Systems Applications and Products, a new accounting system, in fiscal 2000, which resulted in some data problems for fiscal 2000 and 2001, so

only four years (fiscal 2002–05) are presented.

a. Supervision cost for IBRD/IDA and special (West Bank and Gaza, East Timor) projects. Excludes Global Environment Facility and other projects.

Table 2.1: Regional Client Services

Formal incentives to

pursue outcomes have

improved, but more could

be done informally.

tees (which were 10 percent higher than in fiscal2004 and reached a six-year high) and comparedthe numbers of ESW products delivered andtechnical assistance activities completedbetween fiscal 2004 and 2005.

In addition, a lack of understanding of thepossible use and usefulness of performanceinformation may be acting as a disincentive touse monitoring to achieve project and programoutcomes. Many focus group participants stillsee results as something that comes at the endof an operation, and implementation asprimarily about project administration (orfocusing on ensuring that inputs are made toproduce outputs according to the agreedschedule). To many, the Bank’s Managing forDevelopment Results agenda is seen as an add-on or “overhead” that requires the generation ofmore information, and the utility of thatinformation is unclear:

Some things are not easy to measure, and

we try to force things into a measurement

box that isn’t always appropriate, so it

becomes the “feeding the beast”

syndrome—you are doing it because the

institution demands it.

Self-Evaluation and the Bank’s ResultsAgenda Traditionally, monitoring has been concernedwith implementation, compliance withschedules, and the tracking of inputs andoutputs, while evaluation has focused on assess-ing whether the project achieved its statedobjectives. Focus groups held as part of the 2006AROE found that many Bank staff share this viewof M&E. Many equate monitoring with the ISRand evaluation with the ImplementationCompletion Report (ICR)—in other words, anevent that is scheduled at the end of projectwhen the “results” are clear.

The increasing focus on intermediate and finaloutcomes could change the way the Bank viewsand conducts self-evaluations. Bank managers andstaff could use self-evaluations to betterunderstand why progress may not be made in

achieving certain projector program outcomesduring implementation.Evaluative approachessuch as program chainlogic assessment7 couldalso be carried out prior to the start of a project orprogram to determine the strength and logic ofthe underlying conceptual model.

Although self-evaluations would continue to becarried out at the completion of a project, it maynot be sufficient to examine whether a projectwas able to meet its objectives. It may also benecessary to address the broader question ofwhat factors contributed to changes at thebeneficiary level and to what extent thosechanges are attributable to the project. Thiswould require more sophisticated methods,such as impact evaluation.8

Some teams have begun to use a heightenedfocus on project outcomes coupled with self-evaluation to improve performance duringproject implementation. A Rapid ResultsApproach, for example, calls for the formation ofmultistakeholder teams to agree on and achievecritical outcomes within a short (usually 100-day) period. Thisapproach has led to asignificant increase inthe use of a VoluntaryCounseling and TestingCenter in Eritrea. In anAgricultural TechnologyProject in Nicaragua, aRapid Results Approach led to a tripling in theproduction of high-grade milk over a shortperiod in one municipality (with the approachreplicated in others) and to the jump-starting ofa program to promote agricultural productivityin another; this latter result was accomplishedby the signing of 180 contracts between farmersand technical service providers within 100 days.

Other donor agencies are using evaluativemethods to improve implementation of theirprograms. For example, the independent evalua-tion unit of the Deutsche Gesellschaft für


1 7

Many see the results

agenda as an add-on,

and its usefulness is not

fully understood.

Some project teams have

begun using self-

evaluation to improve

performance during

implementation.

Technische Zusammenarbeit (GTZ) has de-veloped an electronic interview and evaluationprocedure, e-VAL, that allows evaluators toobtain and analyze stakeholder assessment onthe likely success or failure of a project orprogram and to provide quick feedback to teams

and stakeholders, who can then take actionwhere necessary (box 2.3).

The Bank is working to assess the effect of itsinterventions on beneficiary outcomes throughimpact evaluations. Impact evaluation gained

1 8


In 2003, the GTZ signed a new performance contract with the Ger-man Federal Ministry for Economic Cooperation and Develop-ment (BMZ) that strengthens results orientation by puttingoutcomes at center stage.

Under this contract, GTZ assumes joint responsibility with thenational partner to achieve outcomes. In turn, implementation ismade significantly more flexible: the project contract specifiesbudget, outcomes, and expected impact, but neither activities noroutputs. This flexibility requires continuous results-oriented ad-justments of activities and outputs—and places a premium onthe collection and use of real-time evaluation information for de-cision making.

GTZ’s evaluation system is based on the two pillars of self- andindependent external evaluation and fosters both accountabilityand learning.

GTZ’s evaluation approach complements the standard Orga-nization for Economic Cooperation and Development–Develop-ment Assistance Committee (OECD-DAC) criteria for assessingdevelopment results against objective indicators with a newcomputer-supported evaluation tool (e-VAL). e-VAL captures keyqualitative dimensions of ongoing projects/programs, such asownership, participation, and target group orientation. Speciallytrained interviewers conduct interviews with the project team,government partners, and civil society organizations or targetgroup representatives.

The computer-supported evaluation procedure processes

these interview data and provides quick feedback at relatively lowcost. The interviewees are encouraged to describe how they see“success,” “failure,” and the current situation of the project/pro-gram against a set of defined elements. The e-VAL software thenconverts these interview findings into a graphic summary to as-sess and compare the impressions among the various groups.

Sharing these results in real time increases ownership by fos-tering a common understanding of the achievements and failuresof the program. Moreover, by aggregating the (anonymous) inter-view data at Regional and sectoral levels, e-VAL allows instantcross-cutting analysis beyond the level of an individual programand enables GTZ’s management to take corrective actions. Todate, GTZ has carried out e-VAL assessments in almost 500 proj-ects and programs worldwide.

The e-VAL tool was used as part of the assessment of the Ar-menia PRS in 2005. Representatives from the Armenian government,the donor community, and nongovernmental organizations (NGOs)were interviewed using e-VAL. The assessment found disagree-ment among the three groups on the achievements being madeunder the PRS. The government was most optimistic and the NGOsmost pessimistic, but common ground was found on elements re-lated to the implementation of the PRS. These evaluation findingswere immediately fed back into the design of the next PRS stage,prompting a rethinking of the roles of the government, donors,and NGOs; greater participation by local NGOs; and emphasis onthe social dimensions of poverty reduction.

Box 2.3: GTZ’s e-VAL System: Using Computer-Supported Evaluation Tools for Rapid ProgressReviews

Source: Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ) GmbH.

Note: BMZ = German Federal Ministry for Economic Cooperation and Development.

By BMZ

e-VA

Le-

VAL

(2–5 years after theend of promotion)

e-VA

L

e-VA

L

By auditor onbehalf of BMZ

Evaluation of ongoing project/program

Final evaluation

Ex post evaluation

Project/programProgress Review

Final reporting

Independent evaluation Self-evaluationGTZ evaluation system

By institutes on behalf of GTZ

formal recognition through its establishment as anew AAA product in 2005.9 The DIME (Develop-ment Impact Evaluation Initiative) by the Bank’sDevelopment Economics (DEC) Vice Presidencyand Poverty Reduction and Economic Manage-ment Thematic Group for Poverty Analysis,Monitoring, and Impact Evaluation provideOperations with training and advice on impactevaluations. An ongoing stocktaking of Bankimpact evaluations by DIME has identified 60ongoing evaluations. Of these, close to two-thirdsare in social sector and poverty programs. TheAfrica and the Latin America and the CaribbeanRegions represent 80 percent of the evaluations.

It would be helpful to clearly define the intendeduses of impact evaluations, which range fromongoing evaluation during project implementa-tion to extensive ex post meta-evaluations, anddeepen the understanding of which interven-tions work under what circumstances at thebeneficiary level. Impact evaluations can be costlyand complex to administer. Alternative evaluativeapproaches, such as cost-benefit analysis, couldbe complimentary to impact evaluation in inform-ing decisions. IEG evaluations have found thatthere is a need for more—and more rigorous—project economic analyses.10 Further researchcould be done to differentiate which methodscould apply in what circumstances.

In sum, the increasing focus on projectoutcomes during implementation is expected toincrease the need for self-evaluation during (andeven before) implementation. Some parts of theBank and other donor agencies are alreadytaking this approach, but it is not yet widely usedwithin the Bank.

Implications for Future WorkThe Bank is taking steps to strengthen resultsframeworks at the project level. QAG and OPCScontinue to review and point out issues withproject results frameworks. Regions are takingsteps to help their staff increase the resultsorientation of their work.

The Africa Region established an M&E teamwithin its quality assurance unit, and other

Regions are bringingstaff who are knowledge-able about managing forresults into their qualityassurance functions(World Bank 2006b).The Latin America andthe Caribbean Regionwas awarded an IEG Good Practice Award in2006 for its efforts to promote managing forresults by disseminating good practices,conducting impact evaluations in severalsectors, and providing technical assistance andsupport to project teams in designing andimproving M&E of project components.

There are opportunities to further strengthenthe CAS results framework as a managementtool. The framework could be a useful way forthe country director to identify and exploitsynergies across Bank operations and managethe country program. That is, country directorscould look to align projects and AAA aroundcommon CAS outcomes through the CAS resultsframework (the vertical alignment in box 2.4).Bank TTLs would continue to focus on how welltheir project or program is progressing, but withmore attention to outcomes.

A recent review reached a similar conclusion. Itrecommended, among other measures, that theBank organize client work into programs(clusters of tasks) focused on country ormulticountry results. But incentives continue todiscourage working across sectors. The reviewidentified the current emphasis on own-unitdeliverables and higher budget predictability forown-managed deliverables as key disincentivesto working in multisector teams and re-commended replacing the Bank’s task-basedapproach to budgeting and management.

A stronger results focus would require strength-ening teamwork within the country team.Country team members would need to cometogether, identify and agree on a common CASand intermediate outcomes to serve as the focusof their efforts, determine how best to measureand monitor progress toward these shared


1 9

Country directors could

make more use of the CAS

results framework to

identify and exploit

synergies among

activities.

One way to address the differences between country- and project-level outcomes would be to focus more onthe integration of project objectives with country objectives through the results framework. The diagram belowoffers a schematic of how this might be done in the CAS.

outcomes, and set up a system to collect thenecessary performance information. Forexample, progress toward a health outcome mayrequire not only health projects, but alsotransport projects to improve access to healthfacilities, water and sanitation projects toprovide access to safe water and a healthierhome environment, and stronger communityoutreach to improve sanitary habits. Operationsthat share a common outcome may considersharing performance indicators.

A focus on country-level outcomes would addressconcerns about disconnects in outcome ratingsbetween country and sector policy outcomes andproject outcomes. An IEG evaluation of Bankcountry programs (IEG 2004c) found that in a thirdof the country programs reviewed, aggregate

project outcomes weresatisfactory, but theoutcomes of the overallcountry assistance pro-grams were unsuccessful(box 2.5).

Similar disconnects have emerged at the sectorlevel. For example, most primary educationprojects met their stated expansion targets butfailed to look at how well students werelearning. Bank advice and lending for tradecontributed to systematic reductions in protec-tion and in opening up economies, but had lessimpact on employment and poverty reductionthan expected. And, finally, a fragmentedapproach to capacity building may have resultedin missed opportunities for more systematicimprovements. These disconnects could beaddressed by conducting evaluation during andupon completion of Bank activities to improveunderstanding of critical factors contributing tosuccess. This understanding should help refinethe results framework at the country, sector, andproject levels.

Buy-in by the client will be essential for betteroutcome-based M&E at both the project andcountry levels. The countries see lack of capacityand the additional cost of data collection at thebeneficiary level as constraints to monitoring.

2 0


XStudy?Y

XTechnicalassistanceactivity

XXXProject?C

XStudy?X

XXProject?B

XXProject?A

Country?outcome

4

Country?outcome?

3

Country?outcome??

2

Country?outcome

1

Efficiency (inputs and outputs) will continue to be a key concern. But more periodic assessment of outcomes is needed.

Focus will be on effectiveness (progress toward meeting CAS objectives and outcomes). More collaboration is needed across taskteams and with borrowers and other development partners to develop M&E systems and monitor progress toward country goals.

Box 2.4: Integration of Objectives in Results Frameworks

More teamwork within

the country team would

also help the results

focus.

Source: IEG data.

A 2004 IEG review of the PRSs (IEG 2004d)found that the development of country-specificindicators and monitoring systems to track themis still at an early stage in most countries, and theavailable information is generally not linked todecision making. The study noted that PRSsmust be more effective in enabling countries toselect and monitor relevant indicators that cansupport domestic decision making on asustained basis. An IEG self-evaluation of itsEvaluation Capacity Development (ECD)Program noted that strong demand for M&E,across all levels of government, is critical tosustaining ECD work (IEG 2004c).

An IEG review concludedthat substantive demandfrom the government is aprerequisite to buildingeffective M&E systems(Mackay 2006). It alsoobserved that there was a“chicken-and-egg” problem:

A lack of government demand for

monitoring and evaluation because of

lack of understanding of monitoring and

evaluation and what it can provide; a

lack of understanding because of lack of


2 1

Several IEG evaluations found that satisfactory outcome ratings atthe project level do not necessarily add up to satisfactory out-comes at the sector and country levels. This finding points to a gapin understanding the factors that contribute to country and sectoroutcomes. To overcome this problem and improve its developmenteffectiveness, the Bank needs to focus more on country-level out-comes and how its interventions align with these outcomes and con-duct systematic evaluation during and upon completion of projectsto better understand the critical factors contributing to success.

IEG documented the gap between aggregate project outcomesand overall country outcomes in 2004. It found that for 33 percentof Country Assistance Evaluations (CAEs) prepared between fis-cal 2001 and 2003, the aggregate rating of all project outcomes ina country was satisfactory, while the outcome of the CAS was ratedunsatisfactory (IEG 2005d).

An update of this analysis for CAEs from fiscal 2001–06 con-firmed the earlier findings with a gap of 32 percent. CAEs assessthe overall country outcome against the objectives stated in theCAS, including the relevance of the strategy in the country con-text. Projects are much narrower in scope and are evaluated asfree-standing products. The discrepancy indicates that the Bankfaces challenges in translating good project outcomes into suc-cessful country strategy outcomes.

At the sector level, IEG has produced similar findings:

• Its power sector evaluation found that sector outcomes havefared worse than project outcomes (IEG-World Bank, -IFC, -MIGA 2003).

• Its evaluation of natural disaster assistance found that sat-isfactory outcome ratings for disaster response projectsconcealed shortcomings in the achievement of durable out-comes and long-term vulnerability reduction at the countrylevel, because not enough attention was paid to prevention(IEG 2006d).

• A review of the primary education portfolio found thatmost projects failed to look beyond increased enrollmentand completion to how well students were learning, al-though the Bank’s primary education portfolio performedabove Bank average and met stated expansion targets(IEG 2006c).

• An IEG evaluation of trade found that Bank advice and lend-ing contributed to systematic reductions in protection and inopening up economies. But the impact on employment andpoverty reduction was less than expected, and the Bank didnot conduct sufficient analysis to inform its policy and lend-ing on this issue (IEG 2006a).

• Its evaluation of capacity building in Africa found that mostsupport for capacity building is designed and managed op-eration by operation. Overarching capacity-building objec-tives were left unarticulated, and most projects embeddedcapacity-building activities in other program componentswithout specifying the capacity-building objectives. The lackof integration made it difficult to capture cross-sectoral is-sues and opportunities and to learn lessons across opera-tions (IEG 2005b).

Box 2.5: Reducing the Gap Between Country- and Project-Level Outcomes

Government demand for

M&E is often inhibited by

lack of understanding

regarding its use and

importance.

2 2


experience with it; and lack of experience

because of weak demand.

Demand for performance information may comefrom government agencies involved with coregovernmentwide functions such as planning,budgeting, or government administration. It mayalso come from within government agenciesinterested in strengthening their performance.

The Bank has been supporting governments’efforts to strengthen data collection. The TrustFund for Statistical Capacity Building (TFSCB)provides grant funding to build statisticaldevelopment strategies and master plans that

lay out the overall goals, actions, and resourcesneeded to strengthen the national statisticalsystems in the recipient countries.

These master plans provide a basis to mobilizedonor support and funding through the Bank’sStatistical Capacity Building MulticountryAdaptable Program Loan (STATCAP). BothTFSCB and STATCAP focus on national statisticalsystems. While individual task teams have beenworking with line agencies to develop outcome-based M&E systems, often with funding from theInstitutional Development Fund, no systematicsupport is currently available outside thenational statistical agencies.


• IEG evaluations have strengthened their assessment of results-oriented M&E at the country and project levels.

• IEG is giving greater emphasis to applied learning and real-timeuse of evaluation.

• IEG has targeted ECD to focus on specific countries and has takena lead in harmonizing evaluation practices.

• IEG’s results framework should continue, and even enhance, itsfocus on the use of evaluation findings.

• The communications and outreach strategy is increasing the aware-ness of IEG evaluations, but clients want more summaries of findings and greater accessibility of reports online.

• The Board and higher-level staff report substantial use of IEG products, while TTLs make less use of them.

• More operations-oriented IEG products are needed.

2 5

Improving IEG’sEffectiveness

This chapter provides an overview of IEG’s role in advancing the Bank’sManaging for Development Results agenda. It includes a results frame-work that links IEG’s mandate and objectives to its operations. Within

this framework, the chapter updates IEG’s efforts to increase its evaluation focuson results. The chapter includes findings from the annual client survey, an up-date on the communications and outreach strategy, and the status of theManagement Action Record (MAR).

IEG’s Results FrameworkIEG has three functions in the Bank’s resultsframework. First, it provides accountability byindependently reporting on the results achievedby Bank operations. Second, it distills the Bank’soperational experience into knowledge of what

works and why and makes that knowledgewidely available to the Bank and to the globaldevelopment community. Third, it supportsclient governments with its technical knowledgeof M&E through its ECD activities.

IEG has a mandate to assess “whether the WorldBank Group’s programs and activities are produc-ing the expected results, including global, regional,and other programs in which the World BankGroup is a participant” (IEG 2005f). By reportingthe results of its evaluations to the Board ofDirectors and communicating the findings andlessons from its work to Bank management,operational staff, and the developmentcommunity, IEG expects to increase the Bank’s

effectiveness and to influence Bank and clientcountry decisions on policies, programs, andprocedures. While fully independent,1 IEG isplaced within the Bank to maximize its operationaleffectiveness and to provide operational staff andstrategic decision makers with knowledge thathelps them work more effectively.

Figure 3.1 summarizes IEG’s results chain andrelates it to measures of performance that havebeen collected for this report. IEG’s outputs arethe findings, lessons, and recommendationsfrom its evaluations and evaluation capacitysupport in client countries. Disseminationefforts are the intermediate step betweenoutputs and outcomes.

For IEG’s accountability function, the intermedi-ate outcomes of these outputs are the Board’s useof evaluations to fulfill its oversight function andthe incorporation of IEG recommendations inBank internal policies and procedures.

33

For IEG’s function as a knowledge provider, theintermediate outcome of its outputs is theinfluence and use of these outputs to improvethe Bank’s policy advice and program andproject design. It also includes use of theseoutputs by external partners to improve theirdevelopment work. Figure 3.2 illustrates theuses of evaluations by IEG’s key clients andstakeholders.

The final outcome for IEG outputs is the use ofIEG knowledge about what works and why tolead to improved effectiveness of Bankoperations in reducing poverty. For example,early findings of the recently completed evalua-tion of World Bank support to primaryeducation (IEG 2006c) have been incorporated

in the 2005 EducationSector Strategy Update.The trade assistanceevaluation (IEG 2006a) isbeing used to recalibratethe Bank’s approach totrade, and pilot initia-

tives are getting under way in several Regions.The Bank’s operational staff, management, theBoard, and external clients use IEG’s outputs tostrengthen actions taken in client countries.Measures that attribute achievement of the finaloutcome to IEG, however, are difficult toconstruct. The remaining sections of thischapter describe each of these links in IEG’sresults chain.

IEG’s Outputs: Increasing the Focus onResults and Impact in IndependentEvaluation Focusing on results has several implications forIEG’s work. IEG can contribute to the Bank’sefforts by strengthening the focus on M&E in itsevaluations. It can also improve the use andinfluence of evaluations by producing quick-turnaround products for applied learning anddecision making. Outside the Bank, IEG canaffect results by continuing the development ofevaluation capacity and providing leadership ininitiatives to harmonize evaluation standardsamong MDBs and bilateral donors.

2 6


Government institutions, policies,programs made more effective

IEG-WB budgetary resources (staff, consultants, and so forth)

• IEG-WB evaluations—Findings, lessons, and recommendations• Support for evaluation capacity development (knowledge, training, advice)

Bank effectiveness improved

Accountability through use byBoard and senior management• Use of evaluation findings by

Board members to provideoversight function

• Evaluation recommendationsincorporated into Bankoperations and policy

Learning from evaluation byoperational staff•Use of evaluation findings by Bank

staff to improve policy advice toclients and program and projectdesign

•Use of evaluation findings byexternal stakeholders to improve theirprograms and policies

• References to IEG in themedia

• Awareness of IEG’sevaluations (client survey)

• Management ActionRecord

• Survey of executivedirectors and their advisors

• Client survey on use andinfluence of IEG’sevaluations

• References to IEG’sevaluations in PADs andsector strategies

Data sources

Dissemination of IEG products

• Client survey on evaluationquality satisfaction

Inputs

Outputs

Final outcome


Figure 3.1: IEG Results Chain

The final outcome for IEG

outputs is the use of

knowledge it produces to

improve the effectiveness

of Bank operations.

Source: IEG.

Note: PAD = Project Appraisal Document.

Strengthening the Focus on M&E in IEGEvaluationsThe most significant implication of the Bank’sManaging for Development Results agenda isthat IEG is expected to assess the results orienta-tion of Bank operations. In the 2004 AROE, IEGcommitted itself to do the following:

Strengthen attention to results-oriented

M&E by requiring evaluations at the

country, sector, and project levels to

address systematically the three key

criteria for results-oriented M&E2 and

increase the weight that IEG ratings give

to the extent of result-oriented M&E and

managing for results in Bank programs.

IEG has been strengthening its assessment

of results-oriented M&E. At the country level,CAEs are now organized around the Bank’scountry objectives and sub-objectives and linkBank outputs with outcomes. In addition, IEGis interacting more with country teams duringthe preparation and drafting of the CountryAssistance Evaluations (CAEs). IEG’s emphasison strategic partnerships is leading to further

collaboration with partners on countryevaluations.

The adoption of CASCR Reviews3 rep-resents amajor shift in the Bank’s evaluation system,because it ensures systematic evaluationcoverage of CAS results.In CASCR Reviews, IEGlooks at the relevance ofthe CAS to the country’sdevelopment priorities,implementation of thecountry program, achievement of CAS ob-jectives, and the quality of the CASCR itself.

As of May 2006, IEG had reviewed 28 CASCRs. Itmust be noted, however, that the current processlimits the effectiveness of the CASCR Review as alearning tool. IEG receives the final CASCR forvalidation when the preparation of the new CASis nearly complete. It may then be too late for thecountry team to take advantage of IEG’s findingsby incorporating much—or perhaps any—ofthe material directly intothe CAS document. Butlessons learned on the

I M P R OV I N G I E G ’ S E F F E C T I V E N E S S

2 7

Figure 3.2: Uses of Evaluation Knowledge

Bank operational teams(M&E, results agenda, applied

learning)

International evaluationcommunity

(harmonization and joint studies)

CODE/Board(oversight)

Development partners(use of evaluation findings, ECD,

joint activities)

Bank management(use and follow-up of evaluations,

influence on Bank directions)

IEG(evaluations, M&E expertise;

self-assessment)

IEG contributes to the

Bank’s results agenda by

focusing on M&E in its

evaluations.

IEG has strengthened its

assessment of M&E.

Source: IEG.

Note: ECD = evaluation capacity development; M&E = monitoring and evaluation.

implementation of the CAS should be consideredby the country team as it moves to implement thenew CAS.

As part of IEG’s new strategic direction, the threeIEG units in the World Bank Group—in the Bank,the International Finance Corporation (IFC), and

the Multilateral Invest-ment Guarantee Agency(MIGA)—will work moreclosely than in the paston country evaluations.For example, IEG nowincludes evaluative con-tent from IEG-IFC in all

IEG country evaluations, except where there is aseparate Country Investment Review, and fromIEG-MIGA where it has conducted evaluationwork. Similarly, the ongoing evaluation ofmiddle-income countries is being carried outjointly across IEG.

Sector and thematic evaluations now have amore in-depth focus on M&E and results frame-

works. Greater emphasisis given to assessing thepolicies and programsthat have led to positiveoutcomes and derivinglessons for their replica-bility, especially in meet-

ing the Millennium Development Goals(MDGs). This emphasis can be found in IEG’srecent reviews of trade assistance, community-driven and community-based development,HIV/AIDS, and pension reform.

At the project level, ICR reviews have included aseparate section on the design, implementation,and use of M&E since August 2005. ProjectPerformance Assessment Reports (PPARs) arealso giving greater attention to M&E issues. Adesk review showed that, as of February 2006,53 percent of fiscal 2006 PPARs reviewed the

design of monitoring,which is a marked in-crease from 15 percentin fiscal 2005 and 6percent in 2004. PPAR

formats vary, however, and a separate M&Esection has yet to be uniformly adopted; only 20percent of PPARs from fiscal 2006 had a separatesection on M&E, compared with 15 percent infiscal 2005 and none in 2004.

In light of the results agenda, it is noteworthythat the language in the Bank’s project-ratingsystem differs from that of the results framework.In the results framework, outcomes can bedefined as a “change in the condition or behaviorof a target group resulting from... outputs”(World Bank 2005c). In the Bank’s projectratings, however, a broader meaning is attachedto outcome. Outcome ratings in ICRs, ICRreviews, and PPARs assess overall projectperformance, summing up the ratings forrelevance, efficacy, and efficiency; they do notdistinguish between outputs and outcomes.Moreover, neither project documents nor IEGevaluations clearly differentiate between projectobjectives that constitute outputs and those thatconstitute outcomes.4 It would be helpful for theBank and IEG to apply clearer language andsharpen the distinction between outputs andoutcomes in project ratings.

IEG is implementing the recommendation madein its 2004 evaluation of global programs. It hascompleted the first three pilot Global ProgramReviews (GPRs) and plans to undertake anadditional four to six GPRs during each of thenext two to three fiscal years. IEG will take stockof its experiences with this new product, a reviewtentatively scheduled for fiscal 2008, which willbe similar to the recently completed retrospec-tive analysis of CAEs. The lessons learned fromthe pilot GPRs have been incorporated in arevised set of guidelines for GPRs. IEG is alsoworking with international evaluation partnersto develop a common methodology for evaluat-ing global programs.

Evaluation methodologyIn addition to increasing the focus on M&E, IEGhas made refinements in the methodologies forseveral of its evaluation products. The countryevaluation methodology has been refined basedon the CAE retrospective study from fiscal 2005

2 8


Country, sector, thematic,

and project evaluations

now have a more in-

depth focus on M&E and

results frameworks.

IEG should sharpen the

distinction between

outputs and outcomes in

its project ratings.

IEG has also started

evaluating global

programs.

(IEG 2005d). IEG has worked with OPCS andQAG to harmonize the Bank’s project ratingsystems for full consistency. Finally, IEG hasdeveloped a rating system for CASCR Reviews.

IEG Initiatives for Applied Learning andReal-Time Use of EvaluationIEG is giving greater emphasis in its products toapplied learning and real-time use of evaluationto improve Bank performance. The real-timeuse of IEG knowledge in the education sector inPeru is an example (box 3.1).

IEG has also introduced a new line of quick-turnaround products that address immediateneeds for evaluative findings and lessons ofexperience, in the form of notes, presentations,or briefing papers. The quick-turnaround workwith the Pakistan Country Office on lessons fordealing with natural disasters in the wake of the2005 earthquake is an example of this increasedfocus on applied learning.

IEG’s evaluation of the LICUS (low-incomecountries under stress) Initiative (IEG 2006b)assessed implementation experience rather

than outcomes, given that many of the LICUSoperations were still under way. The focus wason how the Bank’s approach to LICUS has beenimplemented, what has been learned abouteffectiveness, and how the Bank can do better inthe future. This study represents a newapproach to providing timely assessments thatcan help inform strategic and operationaldecisions.

To create incentiveswithin the Bank forgood performance indesign, implementation,M&E, and developmenteffectiveness, IEG hasbeen giving annual Good Practice Awards tooperations that exemplify strong performancein these areas.5 In addition to providingincentives for high performance, the awardsheighten the profile of operations that offerexamples of good practices.

Deepening Strategic Partnerships IEG’s success in achieving greater focus onresults and learning will require strategic partner-


2 9

IEG is giving more

attention to applied

learning and real-time

use of evaluation.

An example of the generation and dissemination of new knowl-edge in real time can be found in the education sector. For sev-eral years, field visits during PPAR missions to primary educationprojects have found high enrollment increases but very limitedlearning outcomes. In lower-income countries, children oftenread haltingly or drop out illiterate. Also, instructional time ispoorly used. Classes are often cancelled for a variety of reasons.

Government and operations staff did not have a clear idea ofwhen children ought to read fluently and tended to attribute lowoutcomes to poverty and malnutrition. IEG staff, with the help ofDanish trust funds, surveyed the research and found studiesthat helped establish criteria and monitoring indicators for read-ing fluency. And on the basis of neurocognitive research, a ra-tionale was established to clarify the importance of instructional

time and textbook use with respect to achieving developmentgoals.

The results were put to a test in Peru, where nearly half thelower-income second graders were unable to read a single word.IEG worked with task managers in the Latin America and theCaribbean Region to outline learning parameters and M&E indi-cators that were introduced into the country dialogue and arenow being adopted by the government and donors.

One important output was the production of a public servicevideo to help Peruvian parents understand how well their childrenshould be learning and to encourage schools to set and achieveclear learning standards, particularly for the lower grades. Thevideo provided a tool of accountability that any parent canunderstand.a

Box 3.1: Real-Time Use of IEG Knowledge—Accountability through Education Indicators in Peru

Source: World Bank 2004a.

a. This research and its application by IEG and Bank staff has spawned a book by Helen Abadzi (2006) that integrates neurocognitive, educational, and social psychol-

ogy research on issues identified by IEG PPARs in education.

ships. For example, to capitalize on the impactevaluation expertise available in other groups inthe Bank, IEG has been collaborating with theThematic Group for Poverty Analysis, Monitor-ing, and Impact Evaluation and has generatedtwo joint working papers (IEG/Thematic Groupfor Poverty Analysis, Monitoring, and ImpactEvaluation 2006a, b). In addition, IEG is currentlyidentifying ways to collaborate with moregroups, such as DEC and the Bank’s networks,on impact evaluations to ensure that evaluationfindings are used.

Evaluation capacity developmentIEG is strengthening its support of results-oriented M&E capacities in client countries. As a

result of its 2005 self-evaluation of ECD, IEGhas refocused its high-intensity support in afew targeted countries.At the same time, IEGhas maintained low-

intensity support to a broader range ofcountries. These efforts have been particularlyevident in middle-income countries in LatinAmerica, where IEG, the Latin America andCaribbean Region, and the Inter-AmericanDevelopment Bank are supporting a network ofM&E system managers.

IEG will also offer the International Program inDevelopment Evaluation Training in collabora-tion with Carleton University for the sixth timein mid-2006. The four-week course draws broadinterest from evaluation professionals and policymakers worldwide. In addition, IEG is providingcondensed versions of the training in Trinidadand Tobago, India, and Africa. The training willbe provided in China in October 2006.

IEG is supporting conferences on evaluationcapacity to broaden the reach of capacitybuilding through opportunities such as theAfrican Evaluation Association Conference andthe International Development Evaluation

Association Conference.ECD products on M&Emethodology and influ-

ential evaluations consistently draw stronginterest among practitioners.6

Harmonizing development evaluationAs the largest and oldest of the evaluation units inthe MDBs, IEG has taken a leadership role inharmonization efforts within the internationalevaluation community. IEG has actively promotedharmonization of development evaluationmethods through the MDBs’ Evaluation Cooper-ation Group (ECG), the DAC Evaluation Network,and the UN Evaluation Group.

ECG has developed good practice standards forevaluation of both public and private sectordevelopment work. Member institutions havebeen benchmarked against private sectorstandards and now are being benchmarkedagainst the public sector standards. Beyond that,IEG has proposed—and the ECG has agreed to—a joint paper synthesizing findings and lessons onthe linkages between infrastructure and environ-ment operations. If this pilot cooperation issuccessful, ECG may undertake a program of jointevaluation syntheses in the future.

As part of the DAC Evaluation Network, IEG hasengaged actively in developing a system forevaluating progress on the 2004 Paris Declara-tion. It participates in the multidonor evaluationof general budget support and is exploring thefoundation of a “network of networks” onimpact evaluation.

Strengthening IEG’s Results FrameworkTo further strengthen IEG’s contribution to theworkings of the results agenda in the WorldBank Group, IEG’s own results framework, asmonitored by the AROE, should continue, andeven enhance, its focus on the use of evaluationfindings:

• By the Board for oversight • By management through the incorporation

of recommendations into Bank policies andstrategies

• By Bank staff through the use of evaluationfindings for policy advice to country partnersand in project design

3 0


IEG has refocused its

high-intensity support for

capacity development on

a few countries.

IEG has been a leader in

harmonization.

• By external partners through the use of eval-uation findings to improve programs andpolicies.

The results framework can be used as a manage-ment tool to identify first how IEG’s activitiescontribute to achieving results and secondwhere to focus efforts. For example, as IEGincreases its emphasis on quick-turnaroundproducts, it can measure changes in the use andperceived usefulness of its products byoperational staff. It can also measure changes inclients’ perceptions of the timeliness of IEGproducts. To this end, this year’s client surveyhas further refined client feedback on influenceand use of evaluations.

Quality and Relevance of IEG OutputsIEG measures the quality and relevance of itsoutputs as part of its annual internal andexternal client surveys. This year’s surveysqueried target audiences for IEG productsprepared during 2005, including 4 sector andthematic studies, 3 corporate reports, 7 CAEs,37 PPARs, a synthesis report, and an impactevaluation. IEG surveyed 4,285 internal clients,consisting of Bank staff and executive directorsand their advisors. The response rate was 22percent, compared with 31 percent last year.

The survey of external clients regarding evalua-tions that had been disseminated externallywas sent to a targeted sample of 2,759 individ-uals and had a 12 percent response rate. Giventhese response rates, it has to be noted thatthe survey results are indicative for respon-dents, but cannot be generalized to thesurveyed population. Detailed survey resultsand graphs are presented in appendix E.

Readership and awarenessFifty-six percent of Bank staff who responded tothe annual client survey were aware of theevaluation for which they were surveyed. This iscomparable to an awareness rating of 58 percentin 2005 and 39 percent in 2004. Among respon-dents to the external survey, 76 percent wereaware of the evaluation for which they weresurveyed.

Every year the surveycontains questions onthe quality and influenceof IEG evaluations. Thisyear, a section on theuse of evaluations wasadded and the section on influence refined, tobetter understand the use of evaluations byBank staff and to measure the achievement ofIEG’s second intermediate outcome.7

Quality of IEG evaluationsAs shown in figure 3.3, Bank staff were asked torate their satisfaction with IEG’s evaluation for10 attributes of quality on a 6-point scale. Bankstaff respondents reported the highest satisfac-tion with the relevance of IEG’s evaluations totheir work, with 76 percent of respondentsrating it 4 or higher, the highest rating in the pastthree years. Bank staff were least satisfied withthe incorporation of all available informationand the depth of analysis, with 58 percent and59 percent of respondents expressing theirsatisfaction, respectively.

Executive directors and their advisors value therelevance of IEG evaluations to their work most,with 88 percent of respondents rating it 4 orhigher. They assign the lowest rating (69percent) to timeliness.

External respondents generally were highlysatisfied with the quality of IEG evaluations.Eighty-five percent ratedtheir satisfaction with therelevance of evaluationsto their work at 4 orhigher. External respon-dents ranked depth ofanalysis and incorpora-tion of all availableinformation last, with 80percent rating both aspects 4 or higher.

In response to the question of whether evalua-tions influenced their understanding of thesubject area, 79 percent of Bank staff respon-dents rated this 3 or higher. Ninety percent ofexecutive directors and their advisors rated this


3 1

IEG’s results framework

should continue and even

enhance its focus on the

use of evaluation findings.

Satisfaction was highest

for the relevance of IEG

reports and lowest for

depth of analysis and

incorporation of all

available information.

aspect at 4 or higher. Among external clients, 81percent rated the influence of evaluations ontheir understanding of the subject area at 4 orhigher. Overall, IEG does better at compilinglessons and good practices than at providingstaff with specific knowledge about resultschains and how outcomes are linked to outputs.

Executive directors and their advisors’ use of

evaluations for assessing the Bank’s policiesand procedures is high among those whoresponded: 90 percent assigned ratings of 4 orhigher. Bank staff use evaluations mostly forcommenting on the work of others, making acase for a particular course of action, and provid-ing advice to clients; they use evaluations lessfor modifying strategies or operations or design-ing new projects or programs.

About one in four (27 percent) respondents ack-nowledged using evaluations to modify ongoingoperations, assigning a rating of 4 or higher tothis factor. External clients use IEG evaluationsmostly for research (69 percent), and 62 percentuse them to refocus ongoing strategies orprograms. In open-ended questions, respon-

dents pointed to two obstacles to the usefulnessof evaluations: lack of specificity of recommen-dations suggesting how to put them to practice,and lack of reflection on the country contextwithin which the Bank operates, which wouldmake findings relevant to Bank operations in thefield.8

When asked about how the evaluation for whichthey had been surveyed could be improved, 57percent of staff respondents recommended thatIEG make its findings more operational. Last year,72 percent of Bank staff respondents made thisrecommendation. This year, more respondentsrecommended that evaluation teams obtain moreevidence (49 percent), broaden consultation withBank staff (48 percent), and improve depth ofanalysis (45 percent), compared with last year (36percent, 43 percent, and 33 percent, respec-tively). Executive directors and their advisorsmade similar suggestions.

Communicating Knowledge from IEGEvaluationsEffective communication of IEG’s knowledge toBank staff, governments, other donors, and the

3 2


4.88 4.84 4.74

4.02

4.76 4.784.29

4.71 4.47 4.244.28 4.21 4.13 4.00 3.87 3.76 3.73 3.69 3.58 3.55

0

1

2

3

4

5

6

Relevanceto yourwork

Ease ofunderstanding

Concisepresentation

of conclusions

Timeliness Usefulnessof

recommen-dations

Unbiasedand

objectiveanalysis

Transparencyand clarity

ofmethodology

Strong linkbetween

conclusionsand

evidence

Depthof analysis

Incorporationof all

availableinformation

Executive director Staff

Ratin

g

Source: IEG data

Note: Mean on a six-point scale, N = 292 for staff and N = 50 for executive directors.

Figure 3.3: Bank Staff, Executive Directors, and Advisors Generally Satisfied with Quality of IEG Evaluations

international community is a crucial linkbetween evaluations and outcomes. The 2005AROE recommended that IEG “review the

results of pilot initiatives to promote more

effective outreach and communication of IEG

products and consider lessons learned in the

full implementation of the communications

strategy.”

To strengthen communication and outreach,IEG has undertaken pilot initiatives in e-mailmarketing, Web promotions, media outreach,Web site enhancements, multilingual outreach,and IEG Help Desk enhancements.

IEG has begun to track the effectiveness of itscommunication efforts. Awareness of evalua-tions, as reported by Bank staff who werepolled this year, has increased from 39 percentin 2004 to 56 percent in 2006. The number offollow-up inquiries to IEG’s Help Desk hasincreased tenfold during the past year, reachingbetween 2,000 and 2,400 each month. HelpDesk inquiries are increasingly concerned withevaluation methodology, advice on M&Esystems, and IEG’s product portfolio, suggest-ing that IEG’s outreach campaigns are trigger-ing follow-up questions among key audiencesegments. For a comprehensive list of improve-ments in IEG’s outreach methods, seeappendix F.

In spring 2006, IEG organized two mediaoutreach campaigns to coincide with the releaseof its reports on assistance for trade reform andon natural disaster assistance. Both campaignsgenerated extensive media coverage.9 Thegrowing number of references in the media toIEG evaluations indicates strong interest in theBank’s performance on key developmentinitiatives.

In addition to electronic and media dissemina-tion, IEG reaches key audiences withcustomized dissemination events. Twice-yearlyconferences coincided with Bank–InternationalMonetary Fund meetings and featured paneldiscussions on the community-based and–driven development (IEG 2005e) and HIV/AIDS

(IEG 2005c) studies inOctober 2005 and on thenatural disaster evalua-tion (IEG 2006d) in April2006. IEG also has beenconducting in-countryevents, most recently forthe trade evaluation and studies on community-based and -driven development. The latter eventwas a videoconference with practitioners andpolicy makers in six West African countries.

IEG’s communication and outreach can bestrengthened if IEG provides more summaries ofits findings. In the clientsurveys, about two-thirds of all client groupsmade this recommenda-tion. About half of allclient groups would liketo see better notificationregarding events and the availability of newdocuments, and 52 percent of executivedirectors and their advisors would like IEGproducts to be more accessible online.

IEG Intermediate Outcomes: EvaluationRecommendations Incorporated intoBank Operations and PolicyIEG influences the Bank’s Managing for Develop-ment Results agenda through recommendationsto management as part of sector, thematic, andcorporate evaluations, as well as CAEs. Manage-ment is accountable to the Board for follow-up.One of the intermediate outcomes for IEG is theextent to which management incorporates IEGrecommendations and findings in policy adviceand project and program design.

IEG’s 2004 evaluation of lines of credit (LOCs)offers a good example of corrective action bymanagement followingIEG recommendations.IEG found poor results ofBank-funded LOCs andwidespread lack ofcompliance with opera-tional policies governingLOCs. It recommended


3 3

More than half of staff

respondents suggested

that the findings of IEG

evaluations needed to be

more operational.

Further strengthening can

be achieved by providing

more summaries of IEG

findings.

Incorporation of

evaluation

recommendations in

operations and policy is a

major intermediate

outcome for IEG.

that Bank management update OperationalPolicy (OP) 8.30 to cover all LOCs across allsectors and Regions. The Consultative Group toAssist the Poorest’s (CGAP’s) self-evaluation ofLOCs for compliance with OP 8.30 confirmedIEG’s findings. To address IEG’s recommenda-tions, management subjected all identified LOCsto a quality assurance system to foster compliancewith OP 8.30. Comments from a staff surveyindicate that the evaluation has affected projects;for example, one comment was: “evaluation of

Bank LOCs has provided guidance on our recent

design of an LOC in the area of micro and small

lending. Lessons learned in that report have

proved very enlightening.”

The MAR allows IEG to track its recommenda-tions from sector, thematic, and corporateevaluations and to monitor the degree of their

adoption by manage-ment. The MAR trackstwo indicators: the levelof adoption 10 and thestatus of individualrecommendations.11 Itpresents management’s

ratings for these two indicators and IEG’s valida-tion of the same.

The MAR for 2006 tracks management’sprogress on 79 recommendations. Theseinclude 21 new recommendations from the 6

IEG studies (excluding CAEs) presented to theBoard in calendar year 2005 and 58 recommen-dations carried forward from 2002 to 2004.

For the 2006 MAR, IEG rated 66 percent of itsrecommendations as having been adopted bymanagement at high or substantial levels (figure3.4), compared with 75 percent in 2005 and 62percent in 2004.12 The trend of high or substan-tial adoption of about two-thirds of IEG’srecommendations by management has contin-ued in 2006.

IEG and management agreed on the rating forlevel of adoption for 65 percent of recommen-dations (50 out of 77) .13 This figure represents adecline from 70 percent in 2005 and 78 percentin 2004. Where there was disagreement in levelof adoption, IEG’s ratings were lower than thoseof management for all except one of therecommendations.

The 2005 AROE included recommendations toimprove the MAR. A pilot three-year expirationcriterion has been introduced for MARrecommendations from fiscal 2006 onward.Recommendations issued in the previous threecalendar years (2002–04) are being retained inthe MAR. Recommendations from earlier years(calendar years 1998–2001) are being archived,subject to review by the Committee on Develop-ment Effectiveness (CODE) and the Board. Itshould be noted that the three-year retirementrule reduces the comparability of the 2006 MARto those of prior years because the 2006 MARhas proportionally more recent evaluations thanearlier MARs. A summary of the 2006 MAR isincluded in appendix H and posted on theBank’s external Web site.

IEG Intermediate Outcomes: Use ofEvaluation Findings by Bank Staff at theOperational LevelIEG’s performance on this intermediateoutcome can be measured by whether Bank staffuse IEG findings to improve Bank policy adviceand program and project designs. For the 2006AROE, IEG sought to understand better howBank staff use IEG findings. The internal client

3 4


Two-thirds of IEG

recommendations were

adopted at a high or

substantial level in 2005.

Figure 3.4: Level of Adoption of IEG RecommendationsSimilar to That in Previous Years

Source: IEG data.

survey on the use and influence of IEG evalua-tions indicates that IEG provides good servicesto its principal client, the Board. More than 80percent of this respondent category rated use ofevaluations to assess sector and country strate-gies and Bank policies and procedures at 4 orhigher.

Bank staff make less use of IEG products in theirday-to-day work, which consists of designingand implementing projects and programs.Although IEG is seen as enhancing theunderstanding of subject areas and lessonslearned, it is rated lower at helping to identifyresults chains and providing knowledge that canbe readily applied in ongoing operations. Thus,IEG provides high-level knowledge that is usefulfor assessing programs, giving advice to clients,and making comments, but it needs to focus oninfluencing ongoing and future operations.

Improving the timeliness of its products andgiving more attention to presenting the contextof evaluated operations is likely to improveIEG’s impact on Bank operations. Moreover, IEGcould do more in disseminating PPARs. Thoughthey contain valuable project lessons, PPARs arefully disclosed on IEG’s Web site but are notbeing disseminated to a wider audience. Consis-tently high ratings for PPARs for influence anduse suggest that they represent an untappedopportunity for applied learning at theoperational level.

In addition to the client survey, findings from adesk review of selected operational documentsfor explicit references to IEG reports suggestthat IEG evaluations are used as a referencesource more frequently at the sector-strategylevel than at the project level. A review of 234Project Appraisal Documents (PADs) and 60ESW reports from calendar year 2005 and 15sector strategies from 2000 to 2005 found fewexplicit references to IEG evaluations in projectdesign documents and ESW reports. Only about10 percent of these documents refer to thecontent of IEG reports. In contrast, sectorstrategy papers cite IEG documents much morefrequently. Thirteen of 15 sector strategy

documents containmultiple references toIEG. It should be noted,however, that thiscursory analysis doesnot capture the full extent of IEG’s influence.

Focus groups conducted for the 2006 AROEgenerated information on how Bank staff andmanagement view the use and usefulness ofIEG’s evaluation results, products, and informa-tion. Appendix D presents a summary of thefocus group findings.

TTLs are least likely to report that they incorpo-rate evaluation findings into planning anddesign, a finding consistent with the results ofthe desk review of PADs. Their operational con-text requires information on how to monitor,establish indicators, and prepare projects forevaluation. Respondentsin management posi-tions reported greateruse and usefulness ofIEG products. CODEmembers rely heavily onIEG’s reports, advice,and recommendations. Nearly all levels of Bankstaff respondents acknowledged the challengeof bridging the gap between knowledge andapplication. In addition to IEG conductingretrospective evaluations of previous projectsand programs, these findings suggest the valueof IEG pilot testing short summaries of findingsthat are directly targeted at operational staff andmonitoring their use and perceived usefulnessby TTLs.

IEG’s role in a results-focused context has beenevolving to strengthen the focus on M&E in IEGevaluations and to enhance evaluation capacityamong its internal andexternal clients. To im-prove Bank perform-ance, IEG will need tocontinue giving greateremphasis in its productsto applied learning and real-time use of evalua-tion findings. IEG’s success in increasing the


3 5

The Management Action

Record is being simplified

and improved.

Bank staff use evaluation

findings less than

executive directors and

their advisors.

IEG findings are more

often used at the sector-

strategy level than at the

project level.

focus on results will require strategic partner-ships, both internally and externally. Continuedemphasis is needed on increasing the awarenessof IEG products and on developing quick-turnaround products to provide just-in-timeknowledge to decision makers at the strategicand operational level.

The use of IEG evaluation findings is evident withstrategic decision makers. The MAR is a useful toolto assess the implementation of IEG recommenda-tions. Increased attention is needed to make IEG’sfindings more relevant to ongoing operations.IEG’s work program for fiscal 2007 indicates that itis working to meet these emerging needs.

3 6


3 7

Conclusions andRecommendations

The World Bank has been strengthening the results focus of its opera-tions, starting with the emphasis on improving project development out-comes in the early 1990s, the move to a stronger country focus in the

late 1990s, and the heightened attention given to managing for results, in con-cert with other development partners, in the early 2000s.

In 2003, the Bank put in place a Managing forDevelopment Results action plan, whichincluded actions to do the following:

• Strengthen the countries’ capacity and de-mand to manage for results (Country Pillar).

• Enhance the Bank’s relevance and effectivenessin contributing to results (Bank Pillar).

• Harmonize results-based approaches acrossdevelopment agencies (Global Pillar).

The Bank is making progress in carrying out theplan, particularly at the project and countrylevels. OPCS reported (World Bank 2006b) thatPRSs are focusing government, donor, and Bankprograms around desired results in low-incomecountries. Within the Bank, the RBCAS, CASProgress Report, CASCR, and IEG validationprovide the basis for managing for results at thecountry level. In addition, results frameworkshave been introduced for Investment Loans andDevelopment Policy Loans. The newly adoptedISR gives more prominence to monitoring. TheBank also continues to work with other MDBsand other donors to harmonize approaches,

policies, and procedures to reduce the costs tocountries dealing with multiple partners inmanaging for results.

On the sector, thematic, and global levels, effortsto focus on results are at an early stage. Sectorresults frameworks are being developed and theBank is reporting on progress in selected sectorsthrough Sector Strategy Implementation Updatereports. Following up on an IEG review thatidentified the need to strengthen resultsframeworks for better monitoring and evalua-tion in global programs, some changes havebeen made in the management of globalprograms (appendix A). Further work is neededto report achievements at the sector and globalprogram levels and to assess more systematicallyand strategically what has worked well and whyin different sectors and themes across countries.

In its 2006 progress report, OPCS identified a listof possible actions to accelerate the implementa-tion of the results agenda over the coming yearsand laid out next steps in its results framework(annex B of the report). The list of proposed

44

action items is summarized in appendix G. At thecountry level, OPCS proposes enhancing govern-ments’ capacity to use performance informationfor policy implementation at the sector andcross-sector levels.

Internally, the updated action plan calls forRegions and networks to provide leadership foradvancing the results agenda in collaborationwith OPCS under the umbrella of a newly consti-tuted Results Steering Group. New traininginitiatives and stepped-up support for Bankteams are proposed to encourage increased useof M&E information at the operational level.While the report provides a credible list of actionitems, OPCS does not identify priority actionitems or funding sources of this next phase.

The emphasis on results is hardly new. Monitor-ing and evaluation have long been considered toneed improvement.1 The most recent Monitor-ing and Evaluation Action Plan (fiscal 2001–03)introduced results-based M&E in pilot countriesand provided training and technical support toincrease the Bank’s own capacity to manage forresults (World Bank 2000). Although the countrypilots encouraged the adoption of results-basedcountry portfolio reviews and influenced thedevelopment of RBCASs, the efforts tostrengthen the Bank’s internal M&E capacitywere below expectations because of competingpriorities, fragmented efforts, and lack ofadequate funding.

The 2004 AROE identified culture, incentives, andlack of a funded action plan as constraints toimproving the results orientation of the Bank’sM&E system. It recommended that Bank manage-ment (a) develop, maintain, and communicate asustained vision of a results-oriented Bank anddefine how organizational culture, incentives,and structure need to change; (b) develop aphased and costed plan to implement resultsmanagement; and (c) review and strengthenincentives to promote results-oriented M&E.

This year’s AROE finds that the Bank has madesignificant progress in putting in place revisedpolicies and procedures to better manage for

results. However, it notes that these have not yettranslated into improved incentives and practicesat the operational level. Poorly developed resultschains and indicators without adequately definedbaselines and/or targets call into question thelikely effectiveness of these results frameworksfor M&E at the project and country levels.

The analysis identified the following constraintsto outcome-focused M&E:

• Conceptual difficulties in establishing mean-ingful results chains

• Possible crowding out by other priorities (safe-guards, fiduciary, and the like)

• The continuing perception that “delivery ofnew projects” is what matters most

• A lack of appreciation of the possible uses ofperformance information and evaluations formanagement decision making.

The analysis reconfirmed the importance offocusing on country outcomes and aligning Bankoperations around them, as well as the need topromote more cross-sectoral collaboration andmultisector teams. The Bank needs to continueto strengthen results frameworks, at both thecountry and project levels. While it is critical tocome up with a manageable number ofoutcomes and performance indicators, the focusshould not be exclusively on what is measurableand achievable, and staff should be encouragedto take on difficult but important challenges.

With retrofitting of internal policies andprocedures largely completed at the countryand project levels, and under way at the sector,thematic, and global levels, the Bank now faces akey challenge: to change the culture and percep-tions within the organization that act as informalincentives and disincentives. These are difficultchallenges that will take time to resolve. Thebasic principles of the recommendations fromthe 2004 AROE remain valid: the Bank needs tomaintain and communicate its vision for aresults-oriented Bank; address incentives; anddevelop a cohesive, funded action plan. Manage-ment could send a strong signal to staff that itremains committed to the results agenda by

3 8


proposing, communicating, and implementingsuch a funded action plan.

IEG’s role in advancing the Bank’s resultsagenda is evolving to meet the need for timelyand operationally useful information. Toincrease IEG’s effectiveness, greater use andinfluence of IEG evaluations can be achieved byemphasizing learning aspects, embeddingevaluation functions earlier and more systemati-cally into operations, developing quick-turnaround products to support decisionmaking based on evaluative findings, and contin-ued deepening of communications to raiseawareness of evaluation findings and lessons.

Recommendations for Management� Building on progress achieved so far in ad-

vancing the results agenda, agree on a three-year action program supported by appropriatefunding for the implementation of the nextstage, differentiating new allocations and re-deployed resources. Critical action items wouldbe the following:• Support country directors and country teams

in their efforts to refine and use resultsframeworks at country and sectoral levels tomanage country programs. Similar effortsare needed to strengthen the results frame-works of thematic and global programs.


• Strengthen incentives and accelerate aresults-oriented training and communica-

tions program for management and staff toencourage use of M&E information.

• Identify and support in-depth learning op-portunities to develop and use results-basedapproaches with task teams, particularly inchallenging country cases and in complexmultisectoral settings.

� Provide support to task teams and technical ad-vice to countries that intend to institutionalizeM&E systems to strengthen the collection anduse of performance information in order to en-hance their capacity and demand to manage forresults.

Recommendations for IEG-WB� To further strengthen IEG’s contribution to

the workings of the results agenda in the WorldBank Group, IEG should continue to follow itsown results framework and monitor it throughthe AROE. Its focus on the usefulness of eval-uation findings for its core audiences shouldbe enhanced: for the Board for oversight, formanagement through the incorporation ofrecommendations into Bank policies and strate-gies, for Bank staff through the use of evalu-ation findings for policy advice to countrypartners and in project design, for external

partners through the use of evaluation find-ings to improve their programs and policies,and for the countries more broadly. In play-ing this role, IEG should: • Improve the timeliness of its evaluations.• Strengthen the operational relevance of the

findings.• Increase access to and exchange of the

lessons.

C O N C L U S I O N S A N D R E C O M M E N DAT I O N S

3 9

APPENDIXES

4 3

Monitoring Monitoring is a joint responsibility of the countryand the Bank at the project and country levels. Itrequires: (a) defining the expected outcomes;(b) identifying monitoring indicators for final andintermediate outcomes, as well as outputs thatcontribute to achieving the outcomes; (c)ensuring that baseline data are available andtargets are set to assess progress; (d) makingsure a system is in place to collect, analyze, andreport the data; and (e) monitoring progress.Monitoring information should be used to assessprogress toward achieving objectives and toinform adjustments or other actions needed toensure that the objectives are met.

At the activity levelBank task teams work with the client to set upthe monitoring and evaluation (M&E)arrange-ments for Bank activities before implementationbegins.1 The Bank’s current requirements fordifferent kinds of activities are as follows:

• Investment Loans and Credits: All projectsare required to have a results framework thatincludes the Project Development Outcomes(PDOs) for which the project can be held rea-sonably accountable at completion, and inter-mediate outcomes that can be used to assessprogress toward the PDOs during implemen-tation. Each outcome should include indicators,along with baselines and targets to measure thedegree of success in achieving the PDOs andintermediate outcomes. The results frame-work is included as an annex to the Project Ap-praisal Document (PAD). It is meant to specifyhow the indictors will be used to assess per-formance and how the data will be collected.

• Development Policy Loans (DPLs): A focus

on results is one of the key features of thenew policy on DPLs.2 The program documentshould specify the expected outcomes of theoperation and include measurable indicatorsfor monitoring progress during implementationand evaluating outcomes on completion. Aswith investments, the program document hasa results framework that includes a definitionof program objectives and outcomes that aredirectly influenced by the operation, actionsthat are expected to cause the planned out-comes, and outcome indicators with baselinesand targets to measure progress and assessthe program at completion. The Bank is tak-ing a more programmatic approach to DPLs,requiring clear definition of expected medium-term outcomes and the initiation of monitor-ing systems. The program document for thefirst operation incorporates measurable indi-cators for monitoring progress and evaluatingresults upon completion of the program. Theprogram documents of each subsequent op-eration would need to report progress onthese indicators under the previous opera-tion(s). Even single-tranche operations whereall conditions are fully met prior to presenta-tion to the Board are expected to include end-of-program outcomes and a medium-termprogram with (intermediate) outcomes (evenif these were to occur outside the time frameof the single-tranche operation) that could bemonitored separately.

• Analytical and Advisory Activities (AAA):Task teams are required to outline the devel-opment objectives and establish results indi-cators for each objective when startingeconomic and sector work (ESW), nonlendingtechnical assistance, or other AAA. The objec-tives and indicators should be outlined in the

APPENDIX A: OVERVIEW OF MONITORING AND EVALUATION IN THE WORLD BANK

concept paper and recorded in the ActivityInitiation Summary (AIS). The task teamsshould select their objectives and indicatorsfrom a list available in Systems Applicationsand Products (SAP) or supply their own. Theseindicators should be used to evaluate successat the completion of the activity.

Task teams are responsible for periodicallyrecording progress using a standard reportingformat in SAP. For loans and credits, theImplementation Status and Results (ISR) Reportwas introduced in 2005 to replace the ProjectSupervision Report as part of the Bank’sbusiness simplification process. One of theobjectives in introducing the ISR was tostrengthen the focus on likely outcomes.Monitoring is given more prominence bymaking the project performance indicators partof the main report.3 For AAA, process milestonesare recorded in SAP in the Activity UpdateSummary as they are achieved. The system,however, is seen by some as “user unfriendly”and may not be uniformly used.4

At the country levelThe Country Assistance Strategy (CAS) providesthe basis for monitoring implementation ofoperations at the country level. The Results-Based CAS (RBCAS) approach, which wasstarted on a pilot basis in 2003, has beenadopted as Bank policy and is incorporated inthe revised Bank Procedure (BP) 2.11, CountryAssistance Strategies, issued in June 2005. Therevised BP now includes a separate section onCAS implementation that points out theimportance of the results framework, the use ofnational systems, and links with project-levelsystems for M&E.

A key element of the results-based approach isthe CAS results framework, which lays out thelogical relationships between Bank operationsand outcomes and includes measurableperformance indicators to track progress. Itlinks country development goals, CASoutcomes, milestones, and Bank operations,and it clarifies the Bank’s contribution to achiev-ing country development goals. A CAS Progress

Report is prepared mid-way through CASimplementation to present key developmentsand introduce corrections as needed.

The Bank is attempting to assess the impact ofits operations at the country level whenconducting Country Portfolio PerformanceReviews (CPPRs). The 2005 Annual Report on

Operations Evaluation (AROE) identifiedcountries where the Bank was using the CPPR toraise awareness of and carry out a dialogue withthe government on the results of Bankoperations. The Operations Policy and CountryServices (OPCS) review of CPPRs (World Bank2006c) found that country teams have beenproactively incorporating a results approach intothe CPPR process.5 The review concluded thatan RBCAS and a government strategy withspecific country objectives and (ideally) targetswere needed to properly link Bank operationswith the overall country program.

At the sector levelAt the sector level, the annual Sector StrategyImplementation Update (SSIU) presents aconsolidated view of progress in implementingthe sector and thematic strategies adapted bythe Bank. The report is in two parts: the firstprovides a Bank-wide overview of strategyimplementation progress and a reflection of thestrategies in CASs, as well as the Bank’s sectorand thematic inputs (resources), outputs, anddevelopment outcomes. It also discussesprogress on managing for results at the sectorlevel.

Part two of the SSIU provides a more in-depthanalysis of four strategies that are selected inrotation. Each sector presents its Bank-wideresults framework, which establishes final andintermediate country outcomes for the sector;identifies performance indicators; outlines howthe Bank could contribute to these results; andpresents process indicators to determinewhether implementation is on track. One of thechallenges in tracking implementation of thesector strategies is the quality and availability ofdata and the variability across countries. Forexample, the SSIU reports that two of the three


4 4

country outcome indicators on labor marketrelevance in the education sector resultsframework are not yet available. The Bank willwork with the International Labor Organizationand the United Nations Educational, Scientific,and Cultural Organization (UNESCO) Institutefor Statistics to develop these data.

At the corporate (Bank-wide) levelMonitoring at the corporate level is carried outthrough different instruments, each with aspecific purpose, although there are overlaps.Corporate-level performance monitoring issummarized in an attachment to this appendix.Performance reporting and performance indica-tors are divided into two tiers: country-leveloutcomes and Bank performance outcomes.

• Country-level outcomes: A higher tier ofoutcomes to which the Bank contributes, butfor which attribution is difficult to assess—forexample, the Millennium Development Goals(MDGs) in the Global Monitoring Report (GMR)and the 14 country outcome indicators in theIDA 14 Results Measurement System (RMS).6

Data on these country-level outcomes are com-piled by Development Economics (DEC).

• Bank performance outcomes: A second tierof performance outcomes and indicators wherethe Bank’s responsibility and accountabilitycan be more clearly defined. For example, theQuality Assurance Group (QAG) reports onthe effectiveness of the Bank’s current lendingportfolio7 and AAA programs through its An-nual Report on Portfolio Performance (ARPP).The IDA 14 RMS contains a second tier of nineindicators to monitor IDA’s contribution tocountry outcomes in three categories: (a) at thecountry level, where the cumulative intro-duction of RBCASs in International Develop-ment Association (IDA) countries will bemonitored; (b) at the project level, with fourindicators relevant to project quality8 that willbe monitored; and (c) outputs for completedprojects in four sectors (health, education,water supply, and rural transport).

As part of its efforts to advance the resultsagenda at the operational level, the Bank is

proposing a two-tiered Results Monitoring andLearning System (RMLS) with a first tier ofcountry indicators, which is expected to includethe IDA 14 RMS plus indicators for growth,macroeconomic management, social protec-tion, and trade to accommodate IBRD countries;and a second tier of Bank indicators consistingof QAG and Independent Evaluation Group(IEG) scores on project quality and outcomes,plus country performance indicators based onCountry Assistance Strategy Completion Report(CASCR) ratings. An Operational Performanceand Results Report would build on the existingwork carried out in the ARPP, SSIU, and CASretrospectives and include an analysis of theRMLS information. A key objective of the reportwill be to extract lessons learned that can beused in operations.

To allow benchmarking of the multilateraldevelopment banks (MDBs), the CommonPerformance Assessment System (COMPAS) isbeing developed to provide a common sourceof information on the results orientation of theMDBs under the GMR. COMPAS focuses onprocesses and specific actions to enhance afocus on results that are within the control ofeach MDB; it includes a set of process andresults indicators to compare progress.9 Theresults of the first COMPAS exercise will beavailable late in 2006.

As part of its efforts to better align resourceswith strategic directions, the Bank introducedStrategic Performance Contracts (SPCs) withthe Corporate Budget Reform of 2005. The SPChas four parts: (a) Strategic Direction (Are wefocusing on the right things?); (b) StrategicChoices (Are we deploying resourceseffectively to meet our objectives?); (c) RiskManagement (How does risk impact our workprograms?); and (d) a Performance Contract(How well are we performing?). The vicepresidential units (VPUs) outline their strategicobjectives and present outcome indicatorsunder Strategic Choices and share a commonset of key performance indicators for strategicalignment, unit outputs, resources, andstructure. Most of these indicators are taken

A P P E N D I X A : OV E R V I E W O F M O N I TO R I N G A N D E VA L U AT I O N I N T H E WO R L D B A N K

4 5

from Bank systems or are included in the two-tier corporate monitoring system describedabove. The SPCs allow the VPUs to lay out andassess their strategy, link it with their overallresource allocation, identify risk, and providekey performance indicators to monitorperformance.

Monitoring of global programs andpartnershipsIn addition to International Bank for Reconstruc-tion and Development (IBRD) loans, IDAcredits, and AAA, the Bank engages in GlobalPrograms and Partnerships (GPPs), which havegrown from about 20 programs in 1990 to about110 global and 40 Regional programs thatdisbursed around $3 billion in fiscal 2005. GPPsare partnerships, typically with a governancestructure and secretariat located in one of thepartner organizations or as an independent legalentity in its own right. Partners contributeresources, which are pooled in support of activi-ties that are global, Regional, or multicountry inscope.

Within the Bank, the Global Programs andPartnership Council oversees the strategicframework and policies for GPPs. Bank taskmanagers are assigned to implement individualGPPs within the Bank. GPP activities in the Bankare funded through the Bank’s budget (for staffcosts), Development Grant Facility (DGF)grants, and trust funds. Three large programs—the Global Fund to Fight AIDS, Tuberculosis, andMalaria; the Global Environment Facility; andthe Consultative Group for International Agricul-ture Research—accounted for about three-quarters of global program expenditures in fiscal2005.

IEG evaluated the Bank’s involvement in GPPs intwo phases: Phase 1 (IEG 2002) carried out aportfolio review of the 70 programs that existedat the time, and Phase 2 (IEG 2004a) presentedan in-depth analysis of 26 programs. Bothreports identified the need for improvement inthe management of GPPs within the Bank. Theevaluations also found that many programs hadweak M&E systems, that is, lack of clarity in

program objectives; weak results frameworks,which lacked a structured set of indicators; andinadequate arrangements for data collection andanalysis.

The Bank has taken steps to strengthen itsoversight of GPPs. The Global Programs andPartnership Council was established in responseto a recommendation made in the Phase 1evaluation. The Bank issued a strategicframework paper in 2005, which took steps tostrengthen the accountability of the VPUs foroverseeing GPPs, including more effectivelylinking GPPs to the Bank’s country and Regionalstrategies and revising the choice of issues forengagement.

To improve quality at entry, QAG is reviewing asample of DGF-supported GPPs, typically withinsix months of Board approval of the DGF alloca-tion. Monitoring and reporting is expected to bestrengthened through the Grant Reporting andMonitoring (GRM) system for trust funds, whichwas introduced in 2006 to replace the previousTrust Fund Status Report. The GRM is morefocused on outcomes and aligned with the ISR.As with the ISR, however, the GRM needs to beunderpinned by programs with robust resultsframeworks to function effectively as a monitor-ing instrument.

Evaluation The World Bank evaluates its operations on theiroutcome (including relevance, efficacy, andefficiency), sustainability (resilience to risks),and the performance of the Bank and theborrower. Evaluations are done both by Bankoperational staff (self-evaluations) and by IEG,which reviews and validates the Bank’s self-evaluations and conducts its own independentevaluations.

At the activity level

Investment and development policy loans/credits.The evaluation framework at the project level iswell established. The task team prepares anImplementation Completion Report (ICR) foreach project (both investment and DPLs), which


4 6

is supposed to be submitted to the Board withinsix months of project closing. The ICR assessesthe degree to which the project achieved itsdevelopment objectives and comments on Bankand borrower performance. It discusses lessonslearned from the success or failure of theproject. The borrower, in turn, providescomments on the ICR. IEG then conducts deskreviews of all ICRs to independently validatetheir ratings.

During the past 18 months, Bank managementand IEG have introduced several adjustments toICRs to improve their effectiveness:

• Rating outcomes of projects with for-

mally revised objectives: In the past, op-erational staff and IEG often disagreed onhow to evaluate projects where the Bank hadmade mid-course adjustments to accommo-date changing country circumstances. Guide-lines were issued in January 2005 thatprovided for projects with formally revised ob-jectives10 to be assessed against both the orig-inal and revised project objectives, usingdisbursements to determine the weight ofeach.

• Harmonizing evaluation criteria: In Oc-tober 2005, OPCS and IEG jointly announceda revised and harmonized set of ratings forICRs and IEG’s ICR reviews. With the newrating scheme: (a) a common six-point ratingscale was introduced for outcome, Bank per-formance, and borrower performance to allowfor greater nuance in judgment; (b) the as-sessment of the project’s institutional devel-opment impact was made part of the ratingfor outcome, precluding the need for a sep-arate rating (and reducing the total numberof ratings from five to four); and (c) the sus-tainability rating was replaced by a new ratingof risk to development outcome (on a four-point scale), which specifically assesses the un-certainties faced by the project’s developmentoutcome at the time of evaluation. Additionalclarification was issued on outcome, Bankperformance, and borrower performance rat-ings. The system will be implemented in fis-cal 2007.

• ICRs for programmatic DPLs: OPCS andIEG jointly announced a simplified procedurefor programmatic DPLs in March 2006. Underthis new procedure, a single ICR is preparedafter the closing of the last operation in a pre-specified series of programmatic DPLs. The ICRwould include and justify individual ratingsfor each operation in the series; simplifiedICRs for each operation are no longer re-quired. The new procedures took effect inApril 2006.

In addition to the ICR reviews, which systemati-cally review all lending operations, IEG conductsex post evaluations of one-quarter of completedprojects through its Project Performance Assess-ment Reviews (PPARs). Projects subject to PPARsare selected using a number of criteria, includ-ing the potential to learn important lessons(especially in new areas of Bank work); lack ofadequate data in project documentation toensure reliable validation of ratings; and supportfor planned IEG country, sector, or corporateevaluations. A PPAR usually involves a field visitand is generally carried out several years afterproject completion to get more information onactual outcomes than is available shortly afterproject closing.

AAA. An Activity Completion Summary (ACS) isprepared for all AAA activities, irrespective ofcost.11 The ACS consists of two parts: a rating ofeach results indicator selected in the AIS and asummary of major conclusions and recommen-dations resulting from the task. The ACS issupposed to be filed in SAP within six months ofthe delivery to client date.

IEG does not carry out regular evaluations ofAAA. However, AAA is reviewed as part ofsector and thematic studies, CountryAssistance Evaluations (CAEs), and other IEGstudies. IEG has recognized the lack of asystematic review of AAA as an evaluativeknowledge gap and is currently conducting anevaluation of the impact of ESW and nonlend-ing technical assistance on client policies,programs, and institutions. The evaluation isscheduled for delivery in fiscal 2007.


4 7

At the country levelAll CASs are now required to include a self-evaluation of the Bank’s previous countryprogram, the CASCR. The country team reviewsthe effectiveness of the previous program inachieving stated objectives and outcomes basedon the results framework (where available) setout in the previous CAS; derives lessons learned,which are incorporated in the new CAS; andsummarizes the findings in the CASCR. A draftCASCR is ready for review at the beginning of CASpreparation (upstream review), and the finalversion is made available when the draft strategyis near completion (downstream review).

IEG reviews the CASCR to independentlyvalidate the country team self-evaluation. IEG’sassessment of the final CASCR is made availableto the Board before presentation of the CAS. IEGreviews the relevance of the CAS to the country’sdevelopment priorities, implementation of thecountry program, achievement of CASobjectives, and the CASCR.

In addition, IEG carries out an independentassessment of the Bank’s country program over alonger period of time—typically 10 years, whichcovers multiple CAS periods—through its CAEs.

At the sector levelThe Bank provides a report on the status ofsome strategies, which are selected on a rotatingbasis, in the SSIU. For each strategic or thematicarea, the SSIU provides information on the keyachievements in the fiscal year, progress made inintermediate and final outcomes, improvementsmade in output quality, and progress towardimproving M&E within the sector or thematicarea. No self-evaluations are being carried out at

the sector/thematic strategy level. IEG conductssector and thematic reviews that examine Bankperformance and experience over 5–10 yearsand evaluate the relevance and effectiveness ofBank policies and operations.

Evaluation of GPPsSelf-evaluations of GPPs are being carried out aspart of the Bank’s evaluation of its DGF grantsand trust-funded activities. For activities fundedthrough the DGF, Grant Completion Reports areto be prepared within six months of the closingdate of the grant. For trust-funded activities, anICR is to be prepared for all trust funds exceed-ing $1 million within six months of activitycompletion. The current ICR format is beingrevised to provide a more complete assessmentof the achievement of trust fund objectives, anda more systematic review process of the contentof the ICRs is being put in place.

Following the Phase 2 Global Program Evalua-tion (IEG 2004a), IEG plans to carry out GlobalProgram Reviews (GPRs) for four to sixprograms over each two- to three-year period.The GPRs are similar to PPARs for projects andCAEs for CASs. Three pilot GPRs werecompleted in fiscal 2006.12 Unlike investmentprojects and Bank country programs, GPPs arepartnerships with their own governancestructures and the Bank is only one partner inthese programs. In response, IEG is workingwith the Development Assistance Committee ofthe Organisation for Economic Co-ordinationand Development (OECD/DAC) Network onDevelopment Evaluation, the United NationsEvaluation Group, and the Evaluation Coopera-tion Group of the MDBs to develop a commonevaluation framework for GPRs.


4 8


4 9

System/performance Reporting Bank’s intended Responsibility for indicators frequency user/decision maker Data collection data collectionGlobal Monitoring Report MDGs Annual—joint with Report to development From national statistical DEC

partners committee (little direct systems/WDIBank attribution)

COMPAS (process Annual—joint with Report to development Compiled from sub- MDB chairing the MDB indicators and results partners committee on the missionsfrom MDBs Working Group on indicators on MDBs’ performance of IFIs Some results indicators Managing for Results;results orientation) (direct Bank attribution available, other need currently ADB, next World

for its own performance) to be developed Bank Initially focus on Within World Bank—process indicators OPCRX handling

IDA 14 RMSTier 1—Country outcome Complied annually Reporting out to IDA From national statis- DEC + OPCRXindicators (income/poverty, Reported every 3 donors (collective action, tical systems/WDIhealth, gender, education, years in line with IDA little direct attribution infrastructure, private replenishments to Bank)sector development, governance, growth)

Tier 2—IDA’s contribution Complied annually Reported every 3 years Reporting out to IDA Some available from (RBCASs, project quality/ in line with IDA donors, external stake- SAP/Bank database, some outcomes, specific replenishments. holders (direct indicators to be developed,outputs) attribution to Bank) systems to be put in place

to capture outputs OPCRXRMLS (proposed)Tier 1—Country indicators Annual—(proposed) Reporting progress to From national statis- DEC + OPCRX (IDA 14 plus growth, macro- Operational Perfor- Board, senior manage- tical systems/WDIeconomic management, mance and Results ment. Board, manage-social protection, Report ment, and staff expected and trade) to draw lessons learned

and implications for operational work (collective action, little direct attribution to Bank)

Tier 2—Bank indicators Annual—(proposed) Reporting progress to Some available from OPCRX(CASCR assessment by Operational Perfor- Board, senior manage- SAP/Bank database, sector/theme + IDA Tier 2 mance and Results ment, staff, and drawing some indicators to be but Bank-wide) Report lessons learned and developed, systems to

implications for opera- be put in place to tional work capture outputs (Direct attribution to the Bank)

Attachment: Corporate-Level Performance Monitoring Summary



5 0

System/performance Reporting Bank’s intended Responsibility for indicators frequency user/decision maker Data collection data collectionSector Strategy Implementation Update Sector performance indica- SSIU produced Reporting progress Bank database, human Networkstors (outputs, quality, annually, but specific to Board, senior resources data, staff inputs, costs, client sector would most management (Direct estimatesresponsiveness, results likely only be attribution to the Bank)management, and selected reported on every country outcomes) 4–5 yearsAnnual Report on Portfolio Performance Overall lending Annual Reporting to the Board SAP/Bank database, QAG(commitments) and senior management QAG assessmentsPortfolio performance on the effectiveness of (percent DO satisfactory, the Bank’s lending projects/commitments at portfolio and AAArisk, etc.)

AAA (deliverables)Implementing the Results Agenda(direct attribution to the Bank)Strategic Performance Contracts (Regions)Part 1—Strategic directions Based on frequency Used as basis of annual Not clear—various Regions(outcome indicators—varies of strategy review dialogue between VPs sources depending by Region ) and senior management on indicator

during budget preparation

Part 4—Performance Fiscal 2006 QBR2 Used as basis of dialogue Depends on indicator Regions contract (key performance (most recent available) between vice presidents Most available from indicators: [1] results on discusses traditional and Bank senior manage- SAP/Bank database, the ground, [2] strategic deliverables (number ment as part of QBR but some are new and alignment, [3] unit outputs, of projects approved, indicated as “to be and [4] resources and commitment amounts, developed” structure) number of ESW com-

pleted, number of CASs completed, expenditure utilization, and so forth) or items that are meas-urable and worth meas-uring on quarterly basis

Note: AAA = analytical and advisory activities; ADB = Asian Development Bank; CAS = Country Assistance Strategy; CASCR = CAS Completion Report; DEC = Development Economics VicePresidency; ESW = economic and sector work; IDA = International Development Association; IFIs = international financial institutions; MDGs = Millennium Development Goals; OPCRX =World Bank Results Secretariat; QAG = Quality Assurance Group; QBR = Quarterly Business Review; RBCAS = Results-Based Country Assistance Strategy; RMLS = Results Monitoring andLearning System; RMS = Reseults Measurement System; SAP = Systems Applications and Products; WDI = World Development Indicators.

Attachment: Corporate-Level Performance Monitoring Summary (continued)

5 1

Monitoring and evaluation traditionally havebeen considered separate activities. This view isevident in World Bank Operational Directive(OD) 10.70, Project Monitoring and Evaluation(1989), which defines M&E at the project levelas follows:

• Monitoring is the continuous assessment ofproject implementation in relation to agreedschedules, and of the use of inputs, infra-structure, and services by project beneficiaries.Its main objectives are to provide continuousfeedback on implementation and to identify ac-tual or potential success and problems as earlyas possible to facilitate timely adjustments toproject operation.

• Evaluation is the periodic assessment of therelevance, performance, efficiency, and impact(both expected and unexpected) of the proj-ect in relation to stated objectives.

Monitoring and evaluation were also consideredthe responsibility of the borrower’s projectmanagement team. The Bank team wasexpected to assist the borrower in setting up andusing the M&E system. The OD acknowledgesthat an interim evaluation can be carried outduring implementation, but this is not necessar-ily encouraged. The OD states that “supplemen-tary data collection and special studies requiredfor interim evaluations should be kept as simpleas possible, and planned to minimize interfer-

ence with regular project operations”

(emphasis added). Evaluation was expected tobe carried out at the completion of the project.

Focus groups conducted for the 2006 AROEfound that this view of M&E is prevalent amongBank operational staff—that monitoring is about

tracking inputs (disbursements), and possiblyproject outputs, while evaluation is done at theend when the “results” are clear. At the projectlevel, evaluation has meant the ICR, which isthen validated by IEG.

The Bank has been increasing its focus on theachievement of results, specifically the impactits activities have on beneficiaries. This hasseveral implications for Bank operations.

First, the Bank will need to be clearer about whatit expects to achieve through its operations. Toaccomplish this, project development objectivesmust be clearly defined and the changesexpected at the beneficiary level clearly articu-lated. Indicators need to be found to measureand assess progress toward achieving not justthe objective, but the expected changes as well.

Second, strengthening the focus on resultsunderscores the importance of strategy. The linkbetween outputs, outcomes, and higher countrydevelopment goals will need to be strengthenedand the results chain articulated. Intermediateoutcomes that bridge higher-level outcomes andoutputs will need to be identified, to address thegap often found between broad overallobjectives and specific Bank operations (the“missing middle” in box B.1). New Bankprocedures now require project resultsframeworks to specify intermediate outcomeindicators and describe how they will bemeasured and used to track progress toward theachievement of project objectives.

Third, individual Bank operations need to bealigned with a country’s development goals.Many actors and numerous factors contribute to

APPENDIX B: IMPLICATIONS OF THE WORLD BANK’S RESULTS AGENDA FOR MONITORING AND EVALUATION

achievement of these higher-level countrydevelopment goals. This increases the need toreach out to partners, clarify roles and contribu-tions to the outcomes, and jointly monitorprogress.

The move toward a stronger results focus hasimplications for how M&E is done in the Bank.These will be discussed in more detail in thefollowing sections.

MonitoringThe increasing focus on results will requirechanges in the way the Bank approachesmonitoring of its operations. At the activity level(projects and AAA), this will require an increasein the amount of information collected andmonitored. Inputs (contracts, disbursements)and outputs will continue to be critical to projectimplementation, and the bulk of a team leader’stime is expected to be taken up by traditionalproject and AAA administration. However, teamleaders are now also being asked to monitor andtrack intermediate and final outcomes and touse that information in the ISRs.

The stronger focus on outcomes will likelychange the way monitoring data are collected.As monitoring increasingly focuses on the

changes activities (projects and AAA) have onbeneficiary behavior, implementing agenciesmay need to turn to third parties or to differentinstruments for information. For example, aroad agency may no longer just report on thelength of roads being built but may need tomonitor the effect the roads have on potentialbeneficiaries (such as increases in the use ofclinics and other social services in urban areas).This would require the road agency to commis-sion a survey or work with other agencies tocollect the necessary information.

Managers may need to revisit the design oftheir programs or projects during implementa-tion. By asking what is working and notworking with regard to outcomes, managerswill be looking at which elements of theprogram or project are contributing to produc-ing the expected impacts on the beneficiaryand which are not. For example, schools couldbe constructed and textbooks delivered, butthe children may not be learning because ofpoorly trained teachers, which may or may nothave factored into the original design. Duringimplementation, managers will be looking notonly at the efficiency of the program or project(the rate at which inputs gets translated intooutputs), but also at its effectiveness (the rate


5 2

Box B.1: Implications of Results for Bank Operations

Source: IEG data.

Program/project administration: Overseeing contracting and

compliance with agreements; monitoring disbursements and

outputs will continue

Project development objectives:expressed as outcomes with

monitoring indicator(s) and link with Country Development Goal

strengthened

Country developmentgoal

The “missing middle”:logical link between higher- level project outcomes and

outputs made clear

Inputs

Activities

Outputs


Outcomes

at which it is producing the desired outcomefor beneficiaries).

At the country level, the focus on results wouldstrengthen Bank management of its overallcountry program. The Bank has now fullyadopted RBCASs, which require resultsframeworks at the country level that link Bankactivities (projects, studies, technical assistance)with the country’s development goals. RBCASsalso include performance indicators (for CASoutcomes and intermediate outcome indicatorsor milestones to measure progress toward theseoutcomes) to track progress. An activity cancontain more than one objective; for example, ahealth project may contribute to reducing childmortality and to improving public sectorgovernance in the health sector. At the sametime, more than one activity can contribute to acommon CAS objective. Figure B.1 shows atypical RBCAS results framework, where severalBank activities with multiple objectives alignagainst a common outcome.

A results approach would require greater collab-oration and coordination among task teams

within the Bank and development partners atthe country level. Task teams have beenworking—and will continue to work—towardmeeting their project objectives, ensuring thatinputs are provided and outputs delivered toachieve the desired outcome. That is, they willlook horizontally across the matrix in figure B.1.

However, the overall effectiveness of the Bank’scountry program will depend on the extent towhich these activities work together towardmeeting a common country-level (CAS)objective. That is, the vertical links between theactivities become operationally critical for theBank’s effectiveness and would be the focus ofmonitoring at the country level.

Coordination at the country level around resultsis already happening between the Bank and otherdonors through the Poverty Reduction Strategies(PRSs). For IDA countries, the PRSs provide aframework to link country development priori-ties with donor assistance around specificcountry development outcomes. The Bank andthe other donors have committed to harmoniz-ing their monitoring and reporting requirements


5 3

Figure B.1: Implications of Managing for Results for Bank Country Operations

XStudy?Y

XTechnicalassistanceactivity

XXXProject?C

XStudy?X

XXProject?B

XXProject?A

Country?outcome

4

Country?outcome?

3

Country?outcome??

2

Country?outcome

1

Efficiency (inputs and outputs) will continue to be a key concern. But more periodic assessment of outcomes is needed.

Focus will be on effectiveness (progress toward meeting CAS objectives and outcomes). More collaboration is needed across taskteams and with borrowers and other development partners to develop M&E systems and monitor progress toward country goals.

Source: IEG data.

and to working through national systems tolessen the burden on the developing countries.

Rather than have individual project units collectoutcome information, it may be possible to workwith other interested parties to systematicallycollect the information at the country level.Some implementing agencies are reluctant topay the additional costs (for example, tocommission surveys of road users) required tocollect outcome information, preferring to usetheir budgets to produce outputs instead (suchas building more roads).

However, other parts of the government may beinterested in obtaining outcome information.For example, the planning and budgeting unit ofa ministry, local government, or governmentagency may want to compare the effectivenessof programs for budget allocation purposes.Rather than relying on the implementingagencies, they could organize surveys and otherdata collection efforts to produce a uniform setof performance information that could be usedacross all projects (both local and foreignassisted) under their control. This would reducethe burden of data collection and provide unifor-mity and transparency to program implementa-

tion. Several Latin American countries aredeveloping governmentwide M&E systems inline with their own circumstances and needs(Zaltman 2006).

Monitoring systems should be designed withboth the use and user in mind. Performanceinformation can be used for communications,management, and learning (box B.2). Theuses—and hence information needs—differ byfunction for both the borrower and the Bank.

Operational staff (Bank team leaders andgovernment project and program managers)may consider outcome information useful formanagement, to ensure that an activity will havethe expected impact on the beneficiaries, buttheir day-to-day focus will be on the input andoutput information required for administration.Operational staff also need to feed input andoutput information up the chain of command toaddress demands for accountability.

Those responsible for overseeing the overalldirection of an agency or assistance program(Bank staff at the country level; staff in govern-ment agencies; and staff within agenciesconcerned with planning, budgeting, and


5 4

To Communicate• Upward within an organization—With superiors and exter-

nal stakeholders, communicate priorities, clarify expectationsand accountabilities, and report on results that are or are notbeing accomplished to address their demand for accountability.

• Downward within an organization—Clarify priorities and setexpectations and accountabilities; motivate team/staff by set-ting clear, challenging, but realistic goals; and providestaff/team with periodic sense of accomplishment to examinelack of progress to focus future efforts or to celebrate success.

• Sideways within an organization and with clients and other de-velopment partners—Communicate priorities, clarify expecta-tions and accountabilities, promote coordination by identifyinghigher-level outcomes that multiple units may have in common.

To Manage• Monitor progress toward achieving expected objectives and

outcomes.• Identify problem areas that need attention/improvement.• Determine staffing and financial resource needs.

To Learn• Compare results (benchmark) across units/organization and

identify good practices.• Identify approaches that are working well (and could be

replicated).• Raise questions about why outcomes are not meeting ex-

pectations and trigger in-depth examination of performanceproblems.

Box B.2: Possible Uses of Performance Information

Sources: Behn 2003; Harvard University Kennedy School of Government 2001; Hatry 2003.

human resource management) would be moreinterested in outcomes (what is being achieved)than in specifics of implementation (how it isbeing done). They would use performanceinformation for resource allocation and programmanagement. They too need to feed perform-ance information up the chain of command tomeet accountability requirements.

Finally, senior managers (in both the governmentand the Bank) would be interested in perform-ance information, but largely to address account-ability concerns of their enabling environment(the legislature, the Bank Board, and so on).Those higher up in the Bank or governmenthierarchy will be less concerned with operationalinformation and more interested in strategicperformance information (information that willtell them the results of the policies and programsthey authorized); see figure B.2.

Learning, a critical use of performance informa-tion, often gets lost, especially when accounta-

bility is emphasized. There is an inherent trade-off between “managing for results,” whichemphasizes learning what results are beingachieved and why and feeding that informationback into decision making, and “accountabilityfor results,” where performance information isused for reporting and fulfilling accountability(Binnendijk 2001). Learning takes place at alllevels. The operational staff are interested ininformation (such as good practices) that wouldbe relevant to their situation, while seniormanagers would be more interested inbenchmarking (comparing performance acrossprojects or program, agencies, and the like) andmore strategic issues.

In sum, the increased focus on the outcomes ofBank operations will affect what information iscollected, how it is collected, and how it isused. It has implications for the way the Bankand the borrower approach monitoring. Themain changes this will bring about are shown intable B.1.


5 5

Figure B.2: Audiences for Performance Information

Executive, legislative

branch

Bank board, senior management,

RVPs

Government core agencies

(planning finance)

Bank country directors, CMU staff

Ministry/local government

planning, finance, human resources staff

Government project/program

managers

Bank TTLs

Policy-makingstrategic guidance

Operationalinformation

Note: CMU = Country management unit; RVPs = Regional vice presidents; TTLs = task team leaders.

EvaluationWith the focus on results, managers and staff areexpected to track not only inputs and outputsbut also outcomes. They are expected to useprogress or lack of progress toward achievingproject outcomes to focus their efforts duringimplementation. Project and program outcomeswill no longer be something to be determinedand validated at the end through an evaluation.Increasingly, evaluative approaches are expectedto be used not just at the end, but also duringimplementation of a project or program.

While both monitoring and evaluation will beconcerned with achieving objectives andexpected outcomes, evaluation would beneeded to analyze why certain outcomes wereachieved or not achieved. That is, monitoringwould inform managers and staff about what isbeing achieved or not achieved, but evaluationwould provide information on why.

A greater focus on results in Bank operations willchange the timing and type of evaluation carriedout. When managers and staff need evaluativeinformation would change. To date, evaluations(ICRs, Activity Completion Reports, and so on)have been carried out after the completion of aproject or activity (study, technical assistance).They have provided ex post information aboutwhat worked and what did not work, and why.

With a stronger results focus, managers and staff

may want information on what is working, whatis not working, and why in achieving outcomesduring implementation and use that knowledgeto make adjustments to achieve the expectedresults.

Evaluations would be carried out morefrequently, not only during implementation butalso before the start of a project or program. Forexample, program logic chain assessments couldbe done before a project or program is started todetermine strength and logic of the causal modelbehind the design. Focus groups, communityinterviews, and other rapid assessments could beused during implementation to get a betterunderstanding of the changes that the project orprogram is causing among target beneficiaries.These methods are summarized in table B.2.

Bank activities may be only one of many factors—including the role of other donor agencies—thatcontribute to an outcome. Questions of contri-bution and attribution would need to beaddressed through evaluation. This wouldrequire more sophisticated methods, such asimpact evaluations (table B.2). A greater focus onoutcomes would require closer collaborationwith the government and other developmentpartners, including conducting joint evaluations.

In sum, the increasing focus on projectoutcomes during implementation will blur the


5 6

Traditional approach Greater results focus

Monitor progress on inputs and outputs. ➡ Monitor progress on inputs, outputs, and outcomes.

Carry out data collection primarily in house. ➡ Come up with alternative ways to collect information and/or

work with other agencies.

Focus on activities, mainly investment projects. ➡ Focus on combinations of activities that share common out-

comes at the sector, country, and/or global levels.

Use monitoring systems specific to individual activities. ➡ Monitor systems common across activities and development

partners.

Use institutional approach to M&E, working with client

that may need similar information.

Have limited use of monitoring information beyond ➡ Use monitoring as a learning opportunity.

routine reporting.

Table B.1: Moving from a Traditional to a Results-Focused Approach to Monitoring

traditional distinction between monitoring andevaluation—that evaluation is carried out at theend to assess the results of a project or program.A results focus will influence when and howevaluations are done. Evaluations can take placeduring the preparation of a program and during

implementation, as well as at the end. Evalua-tion would function more as a management tool,so that more self-evaluations are expected to becarried out by the Bank. Table B.3 summarizesthe differences between traditional and results-focused approaches to evaluation.


5 7

Question Issue Evaluation approach

Do we have the right The results chains in the projects are weak. Program Logic Chain Assessments determine the strength

strategy and project/ Task teams struggle to link the project with and logic of the causal model behind a policy, program, or

program design? higher-level country development goals and project. The model addresses the deployment and se-

CAS outcomes, especially to determine key quencing of the activities, resources, or policy initiatives

intermediate outcomes that could be used that can cause the desired change in an existing condition.

for M&E. The assessment would address the plausibility of achiev-

ing the desired change, based on the record of similar ef-

forts and on the research literature. The intention is to

avoid failure from a weak design that would have little or

no chance of success in achieving the intended outcomes.

Do we fully understand Project/program managers may not have a full Rapid appraisal methods provide quick assessment data

what is happening at understanding of the changes that the project/ from beneficiaries and stakeholders on the progress of a

the beneficiary level? program may be causing to the beneficiaries. given project, program, or policy. It is a multi-method eval-

This could be caused by a lack of appropriate uation approach that would involve (a) key informant inter-

beneficiary data or difficulties in comprehending views, (b) focus group interviews, (c) community inter-

quantitative monitoring data. views, (d) structured direct observation, and (e) surveys.

Rapid appraisals are quicker and less costly than formal

surveys, but are also less valid and reliable.

To what extent were the Many factors affect the target beneficiary groups, A rigorous impact evaluation identifies a counterfactual to

outcomes a result of the so it is difficult to distinguish the impact of the analyze the situation, with and without the project/

project/ program? project from that of other factors and to under- program. It establishes the impact of the project/program

stand how the project/program affected the and address questions of causality and attribution.

beneficiaries (positively/negatively, intended/

unintended).

Note: CAS = Country Assistance Strategy; M&E = monitoring and evaluation.

Table B.2: Some Uses of Evaluation to Better Manage for Results


5 8

Traditional approach Greater results focus

Monitoring seen as an internal management activity, while ➡ Greater use of self-evaluation by management

evaluation carried out by external, independent entity. Independent validation needed to verify self-evaluations.

Evaluations usually carried out (ex post) at the completion ➡ More evaluations (self- and independent) carried out before

of the activity to determine what worked, what didn’t work, start of project/program to examine what would work and

and why. what would not work and during implementation to deter-

mine what is working, what is not working, and why.

Evaluations focused primarily on single activities (such as ➡ While the Bank would continue to conduct evaluations of

a project) and carried out separately or in parallel with other its own operations, evaluations increasingly carried out

government and/or development partner activities. jointly with government and other development partners.

Table B.3: Moving from a Traditional to a Results-Focused Approach to Evaluation

5 9

Background The Bank is strengthening the results focus of itsCASs. The results-based CAS approach, whichstarted on a pilot basis in 2003, has beenadopted as Bank policy and is incorporated inthe revised Bank Procedure (BP) 2.11, CountryAssistance Strategies, issued in June 2005. Twokey elements of the new approach are thepreparation of a CAS Completion Report(CASCR) to review the effectiveness of theprevious program and a CAS results framework(results matrix), which outlines the outcomesexpected from Bank support during the CASperiod.1 The revised BP has a separate sectionon implementation, which discusses M&E, useof country systems, and the links of Bankoperations with M&E.2

The change was instituted following the findingof a review of CASs that the treatment of resultswas the weakest element (World Bank 2003a).The review found that CASs described in detailtheir objectives and the Bank instruments to beused, but explicit links between the two wereoften missing. The CASs contained self-evaluation frameworks (program matrices) thatcontained large sets of performance targets andindicators, but only about half were specific,reasonably quantitative, and time bound. Manydid not differentiate between overall countrygoals and outcomes influenced by the Bankprogram or articulate the outcomes that theBank or the country wanted to achieve. Theresults-based approach is intended to overcomethese issues.

The CAS Results FrameworkThe CAS results framework provides the basisfor monitoring and evaluating the Bank’s

country program. The framework is expected tolay out the logical relationships between Bankoperations and outcomes and the measurableindicators to track key links. It contains thefollowing elements:

• Country development goals are the longer-term or higher-order development objectivesidentified by the country, which may not beachieved during the CAS period or are notsolely addressed by the CAS.

• CAS outcomes are highest-order resultsachievable during the CAS period; the Bank ex-pects to influence these directly through itsoperations.

• Milestones are the outputs, actions, or out-comes that are expected to be realized duringCAS implementation and can serve as progressmarkers toward CAS outcomes.

• Bank (and partner) programs representthe ongoing and planned lending and non-lending activities of the Bank (and key partners)that are expected to be influenced by CASoutcomes.

The results chains linking the country develop-ment goals, CAS outcomes, milestones, andBank operations should be visible in the resultsframework, clarifying the Bank’s contribution toachieving country development goals.

The CAS results frameworks are intended to beused as a program monitoring and managementtool during CAS implementation and to providethe basis for evaluating the effectiveness of theBank’s program. As a monitoring and manage-ment tool, the CAS results framework will needto focus on a manageable number of perform-ance indicators to allow for regular monitoring

APPENDIX C: REVIEW OF CAS RESULTS FRAMEWORKS

of progress toward CAS objectives. As an evalua-tion tool, the CAS results framework will need tocontain sufficient performance information toallow for a thorough assessment of the program.In both cases, the performance indicators mustbe specific, measurable, baselined, and targeted.CASs could include a comprehensive resultsframework but identify a smaller set of coreoutcomes and indicators that can be used tomanage the program during implementation.

In addition to the CAS results frameworks, theBank is also developing sector resultsframeworks that link the expected finaloutcomes, intermediate outcomes, andexpected Bank contributions as part of Bank-wide sector strategies. The intent is to establishthe links between final outcomes, intermediateoutcomes, and the Bank’s contribution toachieving these results within a sector. This, inturn, would allow the Bank to track progress onthese outcomes and its contributions to theseoutcomes. Sector strategy and performanceinformation will need to fit with the countrypriorities, plans, and needs. This raises thequestion of the extent to which a globalframework can be adapted across multiplecountries with different priorities and levels ofBank engagement.

Objectives, Methodology, and Scope

Objectives of the reviewThe objective of the review is to assess thequality and usefulness of the results frameworkin managing the Bank’s country program. TheCAS results frameworks were examined to assess(a) the strength of the links between countrydevelopment goals, CAS outcomes, milestones,and the Bank program; (b) the usefulness of theperformance measures for M&E; and (c) theextent of integration between CAS and sectorresults frameworks.

Methodology and scopeA maturity model3 was used to evaluate the resultsframeworks and performance measures. Assess-ment criteria were established, and four levels ofmaturity, from level 0 (lowest) to level 3 (highest),

were defined. The evaluation framework ispresented as an attachment to this appendix.

First, the overall results frameworks wereexamined to establish the extent to which CASresults frameworks specify the expectedlinkages between Bank interventions and thecountry’s long-term goals. Results frameworkswith the strongest links were rated level 3. Theassessment criteria for each level are outlined inthe attachment.

Second, the performance measures for countrydevelopment goals, CAS outcomes, andmilestones in the results frameworks wereassessed to establish their usefulness for M&Ebased on the following criteria:

• Measure: The measure is clearly identifiableas a performance measure, distinct from a goalor program activity.

• Baseline: The measure includes the currentlevel of performance as a baseline.

• Target: The measure includes a specified,time-bound performance target.

In addition, milestones were assessed for thefollowing criteria:

• Relevance: Relevance to the objective theprogram intends to achieve; can serve as goodproxy for program outcomes

• Precision: Describes the specific achievementand when it will be achieved and providesenough information to assess programimplementation.

Results frameworks with more robust measures(those that showed more of the above qualities)were rated level 3. The assessment criteria foreach level are outlined in the attachment.

Twenty-five CASs were included in the review forthe 2006 AROE.4 These included CASs from fiscal2005 that were not included in last year’s AROEand those prepared from that time to March 2006.An initial review of CAS results frameworks(results matrices) was done for the 2005 AROE.However, the results are not strictly comparable


6 0

because the analysis was carried out during atransition period when CASs were increasinglyfollowing the results-based format, but beforeRBCASs were formally adopted by the Bank

Third, a provisional assessment examined thelinkages between country and sectoral resultsframeworks. The review was limited to thesector results frameworks for water and sanita-tion from the 2005 SSIU paper.

Findings

Results chain strengthThe examination of the strength of the linksamong country development goals, CASoutcomes, milestones, and the Bank program(results chain strength) revealed scope forimprovement. All of the CASs followed theRBCAS format. A few countries made somemodifications—for example, the Uganda CASdid not distinguish between country develop-ment goals and CAS outcomes.

Of the 25 CAS results frameworks reviewed, six(24 percent) were rated at level 3, as havingrelatively stronger results chains (results chainsare generally visible and most country goals arelogically supported by Bank activities); 11 at level2, as having modest results chains (some countrygoals are logically supported by Bank activities,but many goals are less effectively portrayed);and 8 at level 1, with poor results chains (the linkbetween country goals and Bank activities wasnot clear for most country goals). No CASs wererated at level 0 (figure C.1).

Given the complexity of the results frameworkswith multiple objectives, multiple sectors, andmultiple indicators, an assessment of overall resultschain strength for the entire framework is a matterof judgment. There was a general convergence ofquality around the middle. In the least well-articulated results frameworks, it was very difficultto follow the logic among country developmentgoals, CAS outcomes, and milestones because theyoften consisted of broad goals and bullet points. Inthe strongest results frameworks, the countryteams had made an effort to link Bank operations

to CAS outcomes and attempted to identify theintermediate outcomes and milestones necessaryto achieve the final (CAS) outcomes and definedbaselines and set targets.

No clear differences were discernable by fiscalyear (figure C.2). Of the eight fiscal 2006 CASs,

A P P E N D I X C : R E V I E W O F C A S R E S U LT S F R A M E WO R K S

6 1

Figure C.1: Results Chain Strength

0

2

4

6

8

10

12

0 1 2 3

Weakest

Num

ber o

f CA

Ss

Strongest

Source: IEG data.


Figure C.2: Results Chain Strength by Fiscal Year

0

1

2

3

4

5

6

7

0 1 2 3

Fiscal 2006 Fiscal 2005

Num

ber o

f CA

Ss

Weakest Strongest

Source: IEG data.


five (63 percent) were rated at level 2. The distri-bution of 17 fiscal 2005 CASs is quite even: 5 (30percent) were rated level 3, 6 (35 percent) wererated level 2, and 6 (35 percent) were rated level1. The analysis was conducted using CASs in theBank’s publications database that were

presented to the Board from the fourth quarterof fiscal 2005 to the third quarter of fiscal 2006. Itis very likely, given the time it takes to prepare aCAS and the factors that determine when it ispresented, that most of these CASs wereprepared around the time the RBCASs werebeing adopted. So it is probably too early to drawany conclusions based on fiscal year differences.

Significant differences are evident in thecomparison by country type—IBRD, IDA, andIBRD-IDA blend (figure C.3). The linkagesbetween country development goals, CASoutcomes, milestones, and the Bank programwere rated more highly in IDA countries than inIBRD or IBRD-IDA blend countries. Of the 25CAS reviews, 10 are IBRD, 13 IDA, and 2 IBRD-IDA blend. Among IDA countries, 11 (85percent) were considered to have moderate tostrong overall results chains (levels 2 and 3).Among the IBRD countries, nine (90 percent)were considered to have weak or moderateresults frameworks (levels 1 and 2). This result isnot surprising, because IDA countries haveexperience working with PRSs, which set andtrack country-level performance measures.

Performance measuresThere is scope for strengthening the perform-ance measures in the CAS results frameworks. Inmore than half of the frameworks reviewed, themajority of the country development goals werenot clearly identifiable as performance measures;did not include baselines; and did not set aspecific, time-bound target (figure C.4). That is:

• Measure: Seven CASs (28 percent) had few(level 0) and nine (36 percent) had less than50 percent (level 1) of their country develop-ment goals clearly identified as performancemeasures, distinct from a goal or programactivity.

• Baseline: Fifteen CASs (60 percent) had few(level 0) and six (24 percent) had less than 50percent (level 1) of their country developmentgoals include the current level of performanceas a baseline.

• Target: Ten CASs (40 percent) had few (level0) and four (16 percent) had less than 50 per-


6 2

Figure C.3: Results Chain Strength by Country Type

Source: IEG data.


Figure C.4: Country Development Goals

Source: IEG data.

cent (level 1) of their country developmentgoals include a specified, time-bound per-formance target.

Overall, the review demonstrated that baselinesare the area most in need of improvement.

Country development goals reflect the country’sdevelopment program. Often these goals aregiven as strategic objectives. The availability orlack of specificity, targets, and baselines mostlikely reflect the level of detail available in thegovernment plans linked to the Bank CAS.

The assessment of CAS outcomes exhibits asimilar pattern. In about half of the CASs reviewed,CAS outcomes could be identified as a perform-ance measure. But in more than half of theframeworks, the majority of the CAS outcomes didnot include baselines and did not set a specific,time-bound target (figure C.5). That is:

• Measure: Three CASs (13 percent) had few(level 0) and nine (38 percent) had less than50 percent (level 1) of their CAS outcomesclearly identifiable as performance measures,distinct from a goal or program activity.

• Baseline: Nine CASs (38 percent) had few(level 0) and 13 (54 percent) had less than 50percent (level 1) of their CAS outcomes in-clude the current level of performance as abaseline.

• Target: Six CASs (25 percent) had few (level 0)and seven (29 percent) had less than 50 percent(level 1) of their CAS outcomes include a spec-ified, time-bound performance target.

Again, baselines were the weakest area:baselines were provided for fewer than half theCAS outcomes in 92 percent of the CASsreviewed.

The results frameworks include milestones thatare outputs, actions, or outcomes expected tobe realized during the CAS implementationperiod, and they can serve as progress markerstoward CAS outcomes. Most of the milestonesincluded in the results frameworks were actions(such as regulation adopted) or outputs.

Therefore, the review focused on whether theywere well targeted, relevant to the CASoutcome, and precisely defined.

Overall, actions that contributed to CASoutcomes were well articulated. Most of themilestones were judged to be relevant, butprecision of definition could be improved andtargeting was weak (figure C.6). That is:

• Target: Three CASs (12 percent) had few (level0) and 13 (52 percent) had fewer than 50 per-cent (level 1) of their milestones include aspecified, time-bound target.

• Relevance: Ten CASs (40 percent) had 50 to75 percent (level 2) and 13 (52 percent) hadmore than 75 percent (level 3) of their mile-stones considered relevant to the objectivethat the program intended to achieve.

• Precision: Eleven CASs (44 percent) had fewerthan 50 percent (level 1) and another nine had50 to 75 percent (level 2) of their milestones ad-equately describe the nature and timing of thespecific achievement and provide enough in-formation to assess program implementation.

Qualitative differences in performancemeasures were observed between the sectors.


6 3

Figure C.5: Country Assistance Strategy Outcomes

Source: IEG data.


Measures tended to be strongest in macroeco-nomics, education, and health, where the Bankhas been more active in using numerical indica-tors, or where, as in the social sectors, the MDGshave provided specific numeric targets.

Although there is no “correct” number of indica-tors for a country program, the numbers appearto be high. There were, on average, 24 countrydevelopment goals for each country, and 6countries had more the 30 goals (table C.1). Theaverage for CAS outcomes was 41 goals for eachcountry, and 5 CASs had more than 60outcomes. The average for milestones was 60 for

each country; 10 CASs listed more than 60milestones. Large numbers of indicators presenta challenge for collecting information andmonitoring progress. Moreover, withoutbaselines and targets, the indicators would notbe useful for carrying out an evaluation of acompleted CAS.

Larger country programs can be expected tohave a larger number of indicators. For simplic-ity, the Bank’s fiscal 2006 country budget wastaken as a proxy for Bank country program size.Figure C.7 presents the distribution of thenumber of CAS outcomes by country budget andfigure C.8 the distribution of the number ofmilestones by country budget. Overall, thenumber of both CAS outcomes and milestonesincreases with country budget size, indicating aneed for additional measurement for largerprograms. Some outliers can be observed in themiddle range ($4 million to $6 million), indicat-ing scope for examining and possibly reducingthe number of indicators in individual countries.

Links between Bank-wide sector and countryresults frameworksIEG conducted a cursory comparison of thewater and sanitation sector results frameworkand CAS results frameworks to review the extentto which a Bank-wide sectoral framework couldbe incorporated into the CASs.

The water and sanitation results framework isstructured around MDG Target 10, “Halve thenumber of people without sustainable access to


6 4

Figure C.6: Milestones

Source: IEG data.

Country development goals CAS outcomesa Milestones

Fewer than 9 3

10–29 16 8 4

30–59 6 11 11

More than 60 5 10

Total countries 25 24 25

Average number per country 24.4 41.3 59.5

Maximum number 59 90 125a. Excludes Uganda, where CAS outcomes were combined with country development goals.

Table C.1: Number of Countries by Total Number of Indicators

safe drinking water and basic sanitation” and theIDA 14 RMS, which includes “Proportion ofpopulation with access to a secure water source”as one of the country outcome indicators andwater supply as one of the areas for reportingBank outputs. The results framework sets “thenumber of people with sustainable access to safedrinking water” as the final outcome and the“number of people with access to improveddrinking water sources” as the intermediate

outcome. Performance measures specify thenumber of people and the proportion of thepopulation that have met the outcomes.

Of the 25 CASs in the review, 15 contained directreferences to water or sanitation. Othersreferred to improvement in municipal serviceswhere water and sanitation could be keyelements. Of these 15, 8 included the number orpercent of people serviced or with access to


6 5

Figure C.7: Country Assistance Strategy Outcomes and Country Program Size

0102030405060708090

100

0 2,000 4,000 6,000 8,000 10,000 12,000

Fiscal 2006 Bank country budget (in US$ thousands)

Num

ber o

f CA

S ou

tcom

es

Source: IEG data.


Figure C.8: Milestones and Country Program Size

0

20

40

60

80

100

120

140

0 2,000 4,000 6,000 8,000 10,000 12,000

Fiscal 2006 Bank country budget (in US$ thousands)

Num

ber o

f mile

ston

es

Source: IEG data.

improved water as a performance indicator foreither for country development goal or CASoutcome. While the sectoral results frameworkfocused on increasing access to water and sanita-tion, many CASs also emphasized the institu-tional aspects, such as increasing cost recoveryand financial viability of a water utility, orreducing system losses and improvingoperational efficiency.

This analysis shows the difficulties of balancingsimplicity and clarity that a results frameworkdemands with the complexities of Bank strategy,which is adapted to different country situations.The water and sanitation results framework isbased on clear international and corporatepriorities, MDGs, and IDA 14. It provides asingle performance indicator that reflects thestrategic objectives to be achieved. But devisingcomparable data that can be aggregated acrosscountries may not be straightforward. The SSIUreports that the Bank is working through theUNICEF-World Health Organization JointMonitoring Program to harmonize householdsurveys to increase the comparability of dataacross countries. In addition, the resultsframework (focused on increasing access orsupply) does not fully reflect the institutionalaspects of the Bank’s work, which is critical tosustain development. In many CASs, improvingcost recovery, reducing operational losses, andstrengthening management capacity were seenas key sectoral CAS outcomes.

Overall ConclusionsComparing the CAS results frameworks reviewedthis year with those reviewed last year, it appearsthat the creation of a single, Bank-wide structurefor the presentation of results matrices has had abeneficial effect on the strength of the matrices.

The least well-developed frameworks this yearappeared to be better structured than the leastwell developed last year. Overall, the consistentcategories of measures and definitions appear tohave formed a foundation for the logicalstructure of relationships between outcomes andactivities.

Although the structure has improved, there isstill wide variation between the quality ofmeasures and the consistency of the contents ofeach one of the columns of the resultsframework. This appears to reflect a tendency totry to include too much at each level of theresults framework, especially at the levels of CASoutcomes and milestones. Because of this,carefully defined and otherwise salient indica-tors of country progress and Bank success canbe swamped by indicators that are less wellconsidered.

In addition, many of the performance measuresmay not be useful for M&E because they lackprecision regarding the nature of the expectedchange, the baselines, and the targets. Dataalone are not useful unless they are put in theproper context—that is, do we know where wewant to go and how far have we come? Althoughthis was not a subject of this review, the lack ofbaseline data raises questions about whether theinformation is actually available.

Based on the evidence, country teams need torevisit their country results frameworks,establish priorities, and reduce the number ofindicators, especially Bank CAS outcomes andmilestones, and ensure that the remainder areproperly defined, baselined, and targeted. Thelarge number of CAS outcomes and milestonesmay reflect a need for greater selectivity.


6 6

Attachment

Country Assistance Strategy: Results Matrix Evaluation

Country

Date of CAS

Type of country [IBRD/IDA/IBRD-IDA blend]

Reviewer

Framework level

Criterion Evaluated matrix maturity level Criteria for evaluation

Results chain strength Extent to which CAS results framework specifies the expected linkages

between Bank interventions and the country’s long-term development goals (includes how far along CAS results lie in the causal chain toward country development goals). The results chain should be visible in the results matrix such that the links among country goals (country development goals), Bank goals (CAS outcomes), and Bank activities (milestones) are clear.

Maturity levels:

Level 0: Absence of evidence in the criterion areaLevel 1: Most Bank goals and activities seem misaligned with the country goals, or a profusion of goals and activities buries the logical, causalrelationships.Level 2: Some pillars or high-level goals are well organized and supported by succinctly stated Bank activities and goals, but other areas are lesseffectively portrayed.Level 3: Most pillars or country goals are logically supported by succinctly stated Bank activities and goals.

Results / Measures Level

Evaluated result/Matrix element measure maturity level Criteria for evaluation

Country development CAS goals outcomes Milestones

Measure The measure is clearly identifiable as a performance measure, distinct from a goal or program activity.

Baseline The measure includes the current level of performance as a baseline.Target The measure includes a specified, time-bound performance target.Milestone Milestones are relevant to the objective that the program intends to relevance achieve; can serve as good proxies for program outcomes.Milestone Milestones describe the specific achievement and when it will be achieved precision and provide enough information to assess program implementation.

Maturity levels:

Level 0: Very few results/measures in the matrix demonstrate the defined qualities.Level 1: Fewer than half of the results/measures demonstrate the defined qualities.Level 2: More than half, but fewer than three-quarters, of the results/measures in the matrix show the defined qualities.Level 3: More than three-quarters of the results/measures in the matrix exhibit the defined qualities.


6 7

6 9

This appendix summarizes the findings from aseries of focus group discussions and structuredinterviews of staff and managers regarding theirperceptions of the Bank’s M&E system and theapplication of M&E information.

The study was conducted during March 2006 byGroup Dimensions International to provideinformation for the 2006 AROE. A total of 66people attended 13 sessions that wereorganized separately for task team leaders(TTLs), sector managers (SMs), and countrycoordinators (CCs). Individual interviews wereconducted with 11 members of the Board’sCommittee on Development Effectiveness(CODE) and country directors (CDs). Theresearch focused on four areas:

• The use and usefulness of M&E information• Application of M&E information to implement

operations and manage for results• Challenges, incentives, and disincentives to

using M&E information • The use and usefulness of IEG evaluation find-

ings, products, and information.

The nature and purpose of focus groups isdescribed in more detail in the attachment tothis appendix. Unless otherwise specified, thefindings summarized here reflect themes thatarose across all respondent types.

Use and Usefulness of M&E Information

Respondents were clear regarding the distinctionsbetween monitoring and evaluation and the relation-ship between them. Monitoring provides the shorter-term “processand progress” picture. TTLs use monitoring for

corrections in design, allocation of funds,resetting priorities, and guiding staff. Evaluationreviews monitoring and affords reflection acrossprograms, sectors, and countries, and for alonger term. From the point of view of TTLs andSMs, monitoring means working with the ISR,and evaluation signifies working with the ICR.

• The challenge is more on evaluation than on

monitoring. Monitoring is a more natural

exercise. It’s part of supervision; you monitor

input, processes, you count outputs; some-

thing you relate easily with. It is done in a for-

mal way or not, but done in a much better

way. The evaluation is a more tricky exercise

if you don’t design that well at the very be-

ginning and you have means to collect all the

information, prepare for the analysis, and

all of that. This is something very, very tricky

sometimes. (TTL)

• People should be aware that those evalua-

tions also look backwards, but of course you

derive from it recommendations for the future.

It is an exercise based on past experiences, and

in that sense the outcomes may be linked to

the political priorities of that time. Sometimes

people have expectations which are not clearly

justified from an evaluation exercise view-

point, but you should use this type of retreat

for reflecting the issues the Bank has been en-

gaged in and that were top priorities in our

strategic framework. (SM)

TTLs practice two kinds of monitoring.The first is a less formal, ongoing collection ofinformation as part of oversight and the second,systematic, continuous, and regular collectionof information based on a predefined set ofindicators.

APPENDIX D: FOCUS GROUP AND INDIVIDUAL INTERVIEWS WITH CODE MEMBERS, COUNTRY DIRECTORS, SECTOR MANAGERS, AND TASK TEAM LEADERS

• During implementation and supervision, you

are essentially in this two-dimensional

world—the more formal and structured M&E

that you are doing, which has been designed

during project formulation, and then there is

the informal stuff, supervision, etc. Right at the

end of a project, you have the more bureau-

cratic form of M&E, which is the ICR, etc. That

may be the only time you actually return to

the benchmarks and indicators you set up in

the preparation. (TTL)

• The Bank does a lot of monitoring. When

you’re out in the field, you’re focused on prob-

lem solving. There are a myriad of issues and

problems in a project after you thought every-

thing was nicely set up at the PAD stage and

it went to the Board. You have to deal with

these—and the client has to deal with these.

The biggest problem is dealing with the ob-

stacles that hinder a project that might not

have anything to do with M&E, because oth-

erwise there will be no project. (TTL)

Respondents saw evaluation results (and monitoringinformation) being used for both accountability andlearning.They saw IEG’s role as providing both accounta-bility and learning, but thought IEG may haveshifted more toward accountability. They askedwhy IEG should not stress more learning.

• You don’t want to repeat the mistakes of the

past. You do need to make sure you have

some good information from monitoring of

past activities and broader evaluations.

(CD/CC)

• We would be more interested in themes that

push the sector forward, not things that we’ve

been working on for 30 years. (TTL)

Managers and staff learn about evaluation results inboth formal and informal ways. A great deal of wisdom about lessons, bestpractices, and problem solving occurs informallythrough discussions with colleagues, supervi-sors, and clients about what one can reasonablyexpect to achieve. In addition, participants sawdiscussions among operational staff andbetween the evaluator and staff during an

evaluation as potential opportunities for identi-fying and conveying lessons.

• The CD knows that next year he/she will pro-

duce a new CAS and will start with the team

to reflect seriously on what is going on now.

If there is a good internal debate and partners

were consultants, and there were lessons

learned that were clear, I don’t care if the re-

port was not out...it is impossible to require

this. Most important are the learning processes.

(CODE)

• It is much more useful during implementation

to have an annual program review [Ministry

together with Bank TTL] and you look at the

data together and determine whether or not

you are moving in the right direction. This

process has much more impact on the oper-

ation and implementation than some report

[does]. I have seen that we have to shift, or to

focus on a certain issue that was revealed by

the study. (TTL)

Respondents said they learn about evaluation resultsand obtain knowledge from formal reports related totheir work. They also mentioned that knowledge from self-evaluations and IEG reviews of the CASCR andICR often entered the discourse too late in the

process to have a definable, systematic in-

fluence on the future of Bank work.

• By the time the auditor report comes out, we

have moved on to the next phase or the port-

folio is finished. (SM)

Evaluation results do not flow into future worksmoothly or systematically through mechanisms setup in the project cycle. The project cycle was used in the focus groupsto discuss where and how M&E results enteredinto the process. Respondents challenged theusefulness of the notion (expressed in the cycle)that the ICR and the IEG review of the ICR leadto “lessons learned,” which are then codified asa mandated “lessons learned” section of thePAD. Many saw learning as a continuous processrather than as something to be done at certainintervals.


7 0

• The cycle bears no relationship to reality—it

might in a more structured environment but

not here. The project cycle is a business model

designed by engineers and is very logical and

linear; the actual operating environment in the

World Bank and in the Regions is chaotic. (SM)

• The project cycle is like an Apollo moon

launch, a very calculated progression from

one step to the next; the actual work is not. The

real project cycle looks more like an annual

learning cycle, than the current project cycle.

TTLs need M&E information on an annual

basis. (TTL)

Application of M&E Information toImplement the Bank’s Managing for Results AgendaRespondents at all levels were clear that M&Einformation and knowledge allow Bank staff,clients, and others to obtain a clearer picture ofhow well goals are being achieved.

Respondents agreed that managing for resultsentails serious efforts to establish and measureindicators in the field to ensure that agreed-upon objectives are being met.

• Be explicit about what you want to do, collect

good information about how well you are

doing, and use that information to impact the

way you move forward. (CD/CC)

• One must think of a project in terms of pay-

ing someone to deliver an outcome (not in-

puts). Instead of buying cement for building

a school, we pay for the increase in enrollment

of girls in school. If you are really walking the

talk, you pay for outcomes. (SM)

Respondents saw “managing for results” slightlydifferently, depending on their vantage points. For SMs, managing for results meant “makingsure a program fits outcomes and objectives forindividual projects” within the context of broadcountry/sector goal. For CDs and CCs, it meantcountry-level outcomes rather than outputs. ForCODE members, it meant a way to manage aninstitution, program, or plan to achieve a goalwith a (usually) quantitative target, but some-times also a qualitative target.

• If people can be aligned around a common

set of indicators, they will be aligned around

the agenda. Once you have that, you can

speak to that agenda through the indicators,

which is nonconfrontational and is more

constructive. (SM)

• It is all about gathering information and

making changes. You have different levels of

managing tools. You keep an eye on where you

want to go and use information about how

well you are doing to influence how you use

leverage to get results. We are a step back

from the front line. (CD/CC)

• We are interested in results on the ground, in

the country. How are our projects working to

decrease poverty? Are the projects working? We

are not interested at looking at Bank out-

puts; we look beyond that to what is happen-

ing in the country. (CD/CC)

• Managing for results means that we are not

only spending money, not only helping coun-

tries, but making sure that we have some re-

sults on the ground. Maybe the main goal is

not achieved, but at least that there is some

progress. The result for me is something that

you have on the ground and you are pro-

gressing in the right direction. This is impor-

tant for donor countries but also for recipient

countries. Sometimes people are aware of the

aid we receive, but we also have to show that

this produces outcomes. (CODE)

• Ultimately, it is the reduction of the rate of

poverty, the reduction or increase in educa-

tion for all, decrease in gender equality—all

these MDGs or other development aspects,

cleaner energy, etc. are measurable. These

are ultimate goals and there are some sub-

goals that are more important. Even though

the numbers are not good in the short term,

the Bank assistance may be doing a better

thing for now, which ultimately [will] be more

helpful to the countries than the numbers.

(CODE)

Feedback about the usefulness of the resultsframework is mixed. For some respondents, the results frameworktriggers thinking about the realism of indica-tors and brings a more strategic intentionality

A P P E N D I X D : F O C U S G R O U P A N D I N D I V I D U A L I N T E R V I E W S

7 1

in working with clients. Others saw the resultsframework as rigid in that it was difficult tointegrate new indicators during implementa-tion and to take into account the effects ofexternal factors or risks that can delay or derailresults. The results framework does not alwaysreflect current country circumstances, whichcan mean that unintended and unexpectedachievements remain undetected by formalM&E.

• It is a useful tool, because it focuses on what

we want to achieve and then to align the ac-

tivities to achieve these results. It also brings

more coherence, a more strategic view to the

program. (CD/CC)

• The results framework and using results as the

indicators is actually the right way to go. It

makes more sense. It does make it more dif-

ficult to monitor, more difficult to identify

outcomes and link them to certain actions.

Evaluation becomes more difficult, because

you are no longer saying that our success is

there if we build so many schools. Now we are

successful if the number of girls in schools in-

creases. There are so many factors that it be-

comes impossible to determine whether the

project was successful or not. I still think it is

the right way to go. It also will make evalua-

tion more difficult—by nature. (CD/CC)

• I don’t think it is useful at all if the TTL has

drawn a results matrix sitting behind his desk.

The use of the results matrix is in building

the ownership of the different stakeholders in

the program, whether they are from the sector,

internal stakeholders, or external stakehold-

ers, including the government. They are usu-

ally done by looking at reading materials,

talking to people.... There is a bit of a discon-

nect with what is actually happening in the

country. There is a missing link. They are not

specific enough. They don’t go enough in depth

in the country analysis. (CD/CC)

Managing for results has implications for relationswith clients and team building.CCs and CDs, in particular, see that the resultsorientation of the Bank, in combination with thenew tools developed for managing for results

(for example, the results framework), haschanged the way the Bank interacts with theclient around the CAS.

• The results-based CAS is a novelty and stimu-

lates dialogue; it was the beginning of being

more explicit about what we are trying to do.

It feels like the blind leading the blind, but it’s

getting better with each year. (CD/CC)

• Results management means structuring the

work in a way that it is focused on achieving

results. For the CAS, we first decide what the

results are that we want to achieve, and then

we structure the programs to achieve these re-

sults, and then build the right teams to work

on these programs, to achieve the results.

(CD/CC)

Challenges, Incentives, andDisincentives in Using M&E Informationfor Results

All respondents repeatedly underscored theimportance of conducting responsible M&E todocument Bank work and to manage for results. Professionalism, an “evidence-based” workethic, Bank system requirements for datageneration, and motivation from managementdrive the use of M&E information internally.

• To use M&E data and information generally

is good. Professionals see the benefit for that,

and you don’t need particular incentives.

(CD/CC)

• In the public health field we value it very

much. “It’s indigenous, in our blood.” We can-

not work without knowing what’s happening

and what’s going on. (TTL)

• We have many groups looking over us to see

if we are doing M&E; it’s a prominent part of

the ISR. (TTL)

But respondents described organizational pressuresthat make M&E of results problematic. Factors such as TTL overload, time pressures,and lack of resources for monitoring tend toproduce monitoring plans and indicators thatare put together quickly, just before the PADgoes to the Board.


7 2

• Time is especially important in monitoring,

because it takes a long time to reach agree-

ment with the client on what is worth both-

ering to monitor and how on earth they are

going to do that. You can’t do that within six

months to prepare a project. (TTL)

• You need to do it quickly, you need to do it be-

fore, and then you are running out of time,

because there are all the pressures to deliver.

(CD/CC)

• Real action starts during implementation.

That is where the challenge comes in, re-

garding the amounts of resources that actu-

ally have been allocated. The rationing of

resources begins to happen here. M&E is an ex-

pensive thing, and so the M&E guy gets bumped

off the train sometimes and gets pushed back.

(TTL)

• In some respects basing your program on

M&E is harder. It’s easier to just implement

something rather than to think about whether

it is making a difference or having an impact.

The disincentive is extra effort, extra time,

and extra money. (TTL)

Respondents experienced conceptual difficulties indeveloping results chains and identifying indicators.They pinpointed difficulties positioning aproject within the chain. That is, they come upwith overly ambitious project objectives orposition the objectives too high up the resultschain and then struggle to develop resultsframeworks around them. They also had difficul-ties identifying appropriate performance indica-tors and were looking for support.

• I prepared a project and I wanted to stop at

a level of improving accessibility. The inter-

vention was to rehabilitate some hospitals

and improve the quality of services. The peer

reviewers said that this was not an impact....

What is the impact on people? It would take

more than the project life to show impacts. One

of them said, “What about maternal mortal-

ity?” (TTL)

• One of the main causes of this is how ambi-

tious development projects have become. That

is where the whole thing begins. Everybody

wants to have this best project that will change

the entire face of the world—and not tone it

down, make it more humble, more realistic.

Once you are unrealistic in project concep-

tion [overambitious], everything that follows

from it is also to be overstated—indicators, etc.

That is where we should bring in more hum-

bleness and modesty—because that is where

the whole problem begins. (TTL)

Staff (and clients) feel burdened by the heavyemphasis on quantitative data. Respondents in all groups agreed that not allresults that matter for assessing the perform-ance of the project can be measured. TTLs tendto be action-oriented people; some expressedfrustration with the bookkeeping and shepherd-ing of data required for monitoring, often at theexpense, they believe, of operations.

• There is a tyranny of the measurable here

that we have to be very careful of. (SM)

• Some things are not easy to measure, and we

try to force things into a measurement box that

isn’t always appropriate, so it becomes the

“feeding the beast” syndrome—you are doing

it just because the institution demands it.

(SM)

• We try to use specific numerical targets for

projects, but that doesn’t always happen. In

many projects, it tends to be more subjective,

even though they should be more precise; not

all things are quantifiable. (SM)

Client buy-in to monitoring varies greatly betweencountries with reasonable to high M&E capacity andcountries with low M&E capacity. Governments often do not recognize the valueof developing monitoring (and evaluation)capacity and are therefore unwilling to useborrowed funds to develop that capacity.Respondents pointed to three main factors thatcan reduce the usefulness of monitoring data forclients: (a) divergent data requirements of theBank and the client; (b) a less-than-optimalprocess of designing the M&E system; and (c)uncertainty on the part of the clients regardingpossible negative consequences of less-than-positive monitoring results. Lack of client buy-into gathering data threatens effective monitor-


7 3

ing, which makes building client capacity,especially for monitoring, crucial.

• We are asking them to collect something that

will be used by us rather than by them. A lot

of indicators I used for them as well. But they

may not see it. (TTL)

• All clients want is to get the job done. They

should do M&E but it is not their priority be-

cause they know that their job ends with the

successful completion of the product, and

then they want to go on to the next project.

They do it because you ask them to do it. (SM)

Creating demand for M&E and building clientcapacity are critical. The most important thing to leave in the countryis an effective monitoring system, network, ormethod of doing M&E.

• When you have a sensible set of indicators, it

is very useful for the client. Also, in terms of

culture, we are forced to look into the results

and make sure we don’t get caught in imple-

mentation and forget about the end. In many

client countries, you get bogged down in im-

plementation and to force clients to be part-

ners in tracking the results is useful. (TTL)

• I have managed to convince clients to mon-

itor, but it is only because I represent the Bank,

and because I am tenacious about it, that

they finally give in and start monitoring these

things. Some of them then find it useful. I have

one client who became creative about com-

ing up with new indicators and using them to

convince the prime minister to back her on

certain things. (TTL)

• There are other kinds of data that are simply

difficult to do. If you want to do user satis-

faction, you have to have a survey. That has

to be programmed into the project, which

means that it is not sustainable beyond the end

of the project. (TTL)

Use and Usefulness of IEG ReportsAll respondents understood IEG’s methods andappreciated its evaluation results, but theirposition in the organization influences theextent to which they find IEG evaluations useful

(the higher the level, the more useful thereports).

CODE members found evaluations and reviews tobe important and used them in commenting onsector strategies and specific projects and in givingfeedback to their country authorities regardingdevelopment policies. CDs and CCs appreciatedthe longer view represented in the CAE and sawIEG reviews as important in managing Bankcountry operations: “How well are our projectsdoing in achieving what they set out to achieve?”SMs, for the most part, found IEG evaluationproducts useful, particularly cross-sectoralstudies. TTLs reported having little time to readIEG reports other than ICR Reviews and reportsthat appear to be directly related to their work.

• IEG is struggling to make sure that their reports

are more than reports, that is, not just a piece

of paper. This is not an easy undertaking.

Most evaluations far exceed our expectations.

Every time, we learn a lot. All the evaluations

are giving very good food for thought for man-

agement. (CODE)

• It is a problem for someone to produce a good

evaluation. The team needs to spend lots of

time to understand the circumstances—all

sides, not just one side. When these reports

are done in a superficial way, they don’t re-

ally help. When they are done well and thor-

oughly, they are very helpful. (CD/CC)

• I cite IEG all the time because it is the best we have

got. I do not like the attacks on the Bank from

outsiders who do not have a nuanced view. (SM)

• By nature of their work, IEG ends up coming

up with either irrelevant findings [for TTL] or

misinterpretations—mainly because their au-

dience of their evaluation is not TTLs. They’re

producing evaluation results for the Bank,

to talk about what’s happening in this sector,

what’s happening in this country, etc. (TTL)

All respondent types expressed fairly high levels ofconfidence in evaluation results and analysis andthought that the vast majority of IEG reports are ofgood quality. They perceive IEG’s independence as central togenerating confidence in evaluation results.


7 4

Involving evaluators with (sector/country)expertise was seen as a key determining factorfor quality. Across groups, respondents defineda useful evaluation as one that does thefollowing:

1. Provides systematic evidence on impacts ratherthan simply reporting on outcome data

2. Adequately reflects all the factors that pre-sented roadblocks to achieving results

3. Highlights what works and what does not workand why

4. Gives practical recommendations for how toimprove in the future.

• As I am representing recipient countries, [IEG

reports are] probably more important for me

than to my colleagues. If I have to discuss

with one of my governments, I know that the

evaluation was made by an independent

body and so the information I have is good in-

formation. I can say to my authorities on this

project, an independent body has found this.

(CODE)

• Often you find an evaluation report that does-

n’t tell you why a certain result was achieved.

It’s one thing to evaluate at one level and it’s

another thing to tie it back down to the busi-

ness we are in. We need to have those reports

that focus on our sector and then tell us why

results were or were not achieved. Those are

good reports for us to adjust our business. (SM)

• An evaluation of the open water fisheries com-

ponent looked fantastic on fish output, but

when IEG did the analysis of who was bene-

fiting from the project, it was a case of elite

capture. For inputs and outputs, it was suc-

cessful, but not in outcomes; the project had

to be redesigned. (SM)

Respondents found some IEG evaluations tooacademic or retrospective to be useful. CODE members, Bank managers, and staff aretackling current development issues. Poorevaluations lacked many of the qualities ofuseful evaluations cited above.

• Recommendations may be very idealistically

written. In those cases, management might

want to fully take into account all this—and

in normal cases, they do. But in terms of im-

plementation, it may be very costly to apply

all these recommendations. One has to strike

a balance. In development you have to take

into account the ability for implementation.

(CODE)

• It is very easy to say, okay, here are the results

for outcomes that we would like, and three

years later do an assessment where we real-

ize we have not met any of them and the eval-

uation report essentially says “unsuccessful”

and often does not put this into the broader

context of what was achieved or the under-

lying context. (CD/CC)

Respondents prefer shorter, timelier, more focused,and more accessible reports.All respondent types would like shorter reports,briefings, and e-mails announcing reports thatapply specifically to their project type, sector,and/or country (or on “hot” topics). CODEmembers would appreciate more focused, morecurrent, and shorter reports that have sharperand timelier recommendations. Respondentsneed lessons that culminate in forward-lookingrecommendations. Although IEG has movedtoward publishing more of its products in otherlanguages, CODE and country directors/coordi-nators suggested translating even more

documents into more languages and routinelydisseminating them to counterparts in clientcountries. Results that do not reach clients andbeneficiaries probably do not have as broad animpact, they noted.

• The major problem for the Board is infor-

mation overload, being flooded by e-mails

and paper. We have to condense and crys-

tallize how we present information. (CODE)

• Maybe IEG can make some choices: maybe

[focus] just on the relationship with private

structure... for other issues, deepening may not

be necessary. There is some room for im-

provement here. I would like to see some eval-

uation reports that are less broad in coverage,

but that go into depth on selected issues only—

with a rationale behind how IEG chose the

strategic issues. (CODE)


7 5

• IEG should go away from the approach that we

need to evaluate everything. IEG needs to have

the courage to evaluate areas where we have

a clear indication that our performance is

not so good... forget the areas where perform-

ance is satisfactory but not necessarily spec-

tacular. This is not easy. The DG has to answer

this in the end, as he is responsible. (CODE)


7 6

Focus groups constitute a form of scientific social,policy, and public opinion research. As structuredgroup interviews that proceed according to acareful research design and the principles ofgroup dynamics, focus groups should be distin-guished from discussion groups, problem-solvinggroups, buzz groups, or brainstorming groups.They are not designed to help a group reachconsensus or to make decisions, but rather toelicit the full range of ideas, attitudes, experi-ences, and opinions held by a selected sample ofrespondents on a defined topic.

Through focused interaction on questions ofinterest to the client, respondents from targetgroups can provide a wealth of qualitative datathat cannot be gained from surveys alone. Partici-pants are chosen because of background charac-teristics of special interest to the client (in thiscase, Bank staff and management) and are giventhe opportunity in a guided interaction to discussand debate issues surrounding a program, policy,service, plan, or product. Focus groups normallyrange from one to two hours in duration. Themoderator can pursue ideas that are generatedduring the discussion. Motivations, feelings, andvalues behind reactions to products can beelicited through probing, restating questions, andeliciting opinion from others in the group.

Advantages of the focus group are that the clientis brought closer to the target groups throughobservation of the session, listening to tapes, orreading transcripts and/or final reports (in thiscase, because of the in-house nature of theresearch and guarantees of respondentanonymity, clients did not observe, listen totapes, or read transcripts). Participants stimulateeach other in an exchange of ideas that may notemerge in individual interviews or surveys; ideascan be linked to areas of particular interest tothe client for in-depth exploration.

Through focus groups, it can be learned whatcharacteristics are most salient to participants,the level and nature of emotional value associ-ated with those characteristics, and how partici-pants differ on key issues. Focus groupinterviews are useful for identifying how targetgroups think and feel about the topic underdiscussion. Focus groups also work well in elicit-ing solutions to problems that participantsdescribe; how systems do or do not work; andtheir own levels of satisfaction with organiza-tional requirements, policies, and culture.

Especially for in-house focus groups, there is apossibility of bias when potential respondentswho have what they perceive as negativeorganizational experience with the client mightbe over-represented among those who chooseto respond to the invitation. Moderators aretrained to ensure balanced participation,regardless of a respondent’s point of view.Because focus groups are especially helpful indrawing out the full range of opinions andideas, the presence of participants with morenegative views is useful. A more serious biaswould occur if respondents with only positiveviews (or only negative views) of the clientwere to attend.

The complexity of insights generated by focusgroups extends far beyond the numbersinvolved or the cost of conducting them.Because of the small numbers involved,however, focus groups should typically not beused to generate quantitative data, nor areparticipants statistically representative of thetarget populations from which they are drawn.However, if the research design is well craftedand recruitment is carried out in a neutrallymanner from a stratified set of respondentcategories, then respondents should be concep-tually reflective of their populations.


7 7

Attachment: The Nature and Purpose of Focus Groups

7 9

Since 1999, IEG has surveyed its clients annuallyto gauge their perceptions of the quality andimpact of IEG evaluations as part of the continu-ous process of self-evaluation. The surveyassesses IEG evaluations in four areas: reader-ship and awareness; quality; influence of thereport on Bank staff understanding, processes,and policies; and recommendations made toIEG for improving the quality and relevance ofits reports. This year’s survey also includedquestions on the use of IEG evaluations in Bankoperations and sources of knowledge in theBank.

This year, IEG conducted three separatesurveys: internal Bank staff; executive directors(EDs) and their staff; and selected externalaudiences. This appendix reports on thefindings from these three surveys.

Results of Survey of Internal Clients andExecutive DirectorsThis year’s internal survey of 4,285 Bank staffand EDs and their advisors was the largest thathas been done in this series (figure E.1). Itsought to elicit the perceptions of respondentsregarding IEG evaluations produced during2005.

The following IEG products were the subject ofthe survey:

• CAEs for Albania, Bolivia, Mauritania, Moldova,Pacific Islands, Romania, and Turkey

• The 2004 ARDE• The 2005 AROE • PPARs (37) • Sector, thematic, and corporate evaluation

studies:

– Capacity Building in Africa: An OED Eval-

uation of World Bank Support

– Committing to Results: Improving the

Effectiveness of HIV/AIDS Assistance

– Pension Reforms and the Development of

Pension Systems: An Evaluation of World

Bank Assistance

– The Effectiveness of World Bank Support for

Community-Based and -Driven Development

– IEG Review of World Bank Assistance for

Financial Sector Reform

– Improving the World Bank’s Development

Effectiveness: What Does Evaluation Show?

– Maintaining Momentum to 2015: An Im-

pact Evaluation of Interventions to Im-

prove Maternal and Child Health and

Nutrition in Bangladesh

Two primary audiences were targeted for thisyear’s survey: Bank staff who could be definedas intended audience for a particular evaluation

APPENDIX E: CLIENT SURVEY RESULTS

Figure E.1: Number of Staff Surveyed 2003–06

4,285

2,120

3,159

1,172

0

1,000

2,000

3,000

4,000

5,000

2006200520042003

Year

Num

ber

Source: IEG data.

based on their role or affiliation (4,095surveyed) and EDs and their advisors (190surveyed).

This appendix provides a summary of keyfindings. Where possible, this year’s results arecompared with results from previous years andsimilar types of respondents.

Response rates The surveys were sent electronically to 4,285 staffover a four-week period in April 2006. Theresponse rate averaged 22 percent over allevaluations (933 responses in total; figure E.2).For individual reports response rates rangedfrom 13 percent (for Improving the World

Bank’s Development Effectiveness—What Does

Evaluation Show?) to 40 percent (for theBangladesh impact evaluation) and averaged 22percent across all survey products. As in lastyear’s survey, reminder notifications were sent tononrespondents. While survey response ratesare directly comparable only to 2005,1 responserates and sample sizes over the past four yearsshow a correlation between larger samples andsmaller response rates.

Given the response rate of 22 percent it has tobe noted that the survey results are indicative

for respondents but are not generalizable for theoverall surveyed population.

Readership and awarenessOverall awareness of reports was 56 percentamong non-ED respondents (figure E.3). Thisincludes respondents who had read, partici-pated in a discussion, or attended an event onthe evaluation for which they were surveyed(direct awareness) and respondents who wereaware of the evaluation’s findings even if theyhad not read the report or participated in anevent (indirect awareness). Forty-two percent ofrespondents had read, participated in a discus-sion, or attended an event on the evaluation forwhich they were surveyed. This continues lastyear’s trend of 45 percent and 13 percent,respectively, and marks an increase from 27percent and 9 percent in 2004.

For non-ED respondents who were aware of theevaluation, about two-thirds (64 percent) heardabout it through an e-mail announcement, 12percent saw the evaluation Web site, and 20percent received a hard copy. Seventy percentsaid they had shared information or findingsfrom the evaluation with colleagues, 26 percentwith colleagues and clients.

Quality of IEG evaluationsSurvey recipients were asked to rate theirsatisfaction with the specific IEG evaluation theywere queried about on a six-point scale for 10attributes of quality: relevance to their work,ease of understanding, concise presentation ofconclusions, timeliness, usefulness of re-commendations, unbiased and objective analy-sis, transparency and clarity of the methodology,strength of the link between conclusions andevidence, depth of analysis, and whether allavailable information was incorporated. FigureE.4 shows the proportion of respondents thatrated each aspect of the evaluations 4 or better.

This reflects a new approach to obtaining userviews and differs from the method used inprevious years, when timeliness, transparency ofmethodology, and incorporation of all availableinformation were yes/no questions and other


8 0

Figure E.2: Response Rates

38

21

3122

0

10

20

30

40

50

60

70

80

90

100

2003 2004 2005 2006

Year

Perc

ent

Source: IEG data.

attributes were to be rated on a four-point scale.2

Due to this change in methodology, the surveydata better reflect audience satisfaction andenable more meaningful comparisons of IEG’sclient audiences’ satisfaction with attributes ofevaluation quality. However, the change in therating scale introduces some discontinuity intrends over time.

This year Bank staff survey respondents (292 Bankstaff responded to this question) gave the highestratings to the relevance of IEG evaluations to theirwork (average of 4.28, with 76 percent rating it 4or better), closely followed by ease of understand-ing (4.21, or 76 percent) and concise presentationof conclusions (4.13, or 72 percent). Staff satisfac-tion was lowest for incorporation of all availableinformation (3.55, or 57 percent) and depth ofanalysis (3.58, or 56 percent).

Overall, the EDs and advisors who responded(50) expressed higher satisfaction with evalua-tions than Bank staff. EDs and advisors ratedrelevance of IEG evaluations to their workhighest (mean of 4.9, or 88 percent of respon-dents rating it 4 or higher), followed by ease ofunderstanding (4.8, or 90 percent) and

unbiased and objective analysis (4.8, or 90percent). They were least satisfied withincorporation of all available information (4.2,or 82 percent) and timeliness (4.0, or 69percent).

A P P E N D I X E : C L I E N T S U R V E Y R E S U LT S

8 1

Figure E.3: Readership and Awareness of IEGEvaluations

4245

2737

1413

1210

0

10

20

30

40

50

60

70

80

90

100

2003 2004 2005 2006

Indirect awareness Direct awareness

Year

Perc

ent

Source: IEG data.

Figure E.4: Bank Staff and Executive Director Satisfaction with Quality of IEG Evaluations

4.88 4.84 4.74

4.02

4.76 4.784.29

4.71 4.47 4.244.28 4.21 4.13 4.00 3.87 3.76 3.73 3.69 3.58 3.55

0

1

2

3

4

5

6


Ease ofunderstanding

Concisepresentation

of conclusions

Timeliness Usefulnessof

recommen-dations

Unbiasedand

objectiveanalysis


ofmethodology

Strong linkbetween

conclusionsand

evidence

Depthof analysis

Incorporationof all


Executive director Staff

Ratin

g

Source: IEG data.

Note: Mean on a six-point scale, N = 292 for staff and N = 50 for executive directors.

As noted, although these results cannot bedirectly compared with previous years, the 2006survey shows timeliness ratings of IEG evalua-tions that are comparable with previous years(figure E.5). 89% of Bank staff rated their satisfac-tion 3 or higher, and 67 percent rated it 4 orhigher, compared with 89 percent who consid-ered evaluations to be timely last year. Theresponses are similar for EDs and advisors, ofwhom 89 percent rated their satisfaction 3 orhigher and 69 percent rated it 4 or higher,compared with 80 percent who consideredIEG’s evaluations to be timely last year.

Comparing those attributes that used a four-point scale in previous years shows thatrelevance of IEG evaluations to Bank staff ’s workwas rated highest this year, with 76 percentrating it 4 or higher, compared with 67 percentrating it excellent or good last year and 73percent in 2004. Link between conclusions andevidence (2006, 61 percent; 2005, 58 percent;2004, 65 percent) and concise presentation ofconclusions (2006, 72 percent; 2005, 73 percent;2004, 75 percent) fare similarly well. Satisfactionwith depth of analysis, however, was rated 56percent this year, compared with 71 percent in2005 and 80 percent in 2004 (figure E.6).

Influence of IEG evaluationsThis year, Bank staff were asked to rate on a six-point scale—from 1 = not at all to 6 = a greatdeal—the influence of an IEG evaluation ontheir understanding of the subject area, theBank’s country assistance and sector strategy,lending and nonlending services, and howoutcomes are linked to outputs (resultsframework) at the project, sector, and countrylevels. Moreover, they were asked to rate howmuch the evaluation helped them gain a view ofessential lessons and best practices. This year268 Bank staff responded to this question.

EDs and advisors were asked to use the same six-point scale to rate how much IEG evaluationsinfluenced their understanding of the subjectarea; helped them better understand Bankactivities at country, sector, and project levels;and helped them gain a view of good practicesand essential lessons. Forty-nine EDs and theiradvisors responded to this question.

This year more than three-quarters (79 percent)of Bank staff respondents rated the influence ofevaluations on their understanding of thesubject area 3 or higher, and 28 percent rated it5 or 6 (figure E.7). This result is similar toprevious years, when a four-point scale of not atall, little, somewhat, and a great deal was used.In 2005, 78 percent rated IEG evaluations ashaving either somewhat or a great deal ofinfluence on their view of the subject area. Therating in 2004 was 73 percent.

Overall, as in previous years, EDs and advisorsperceive themselves to be strongly influenced byIEG products. Among EDs and advisors, 90percent of respondents rated the influence ofIEG evaluations on their understanding of asubject area 4 or higher, with 69 percent rating it5 or 6. Last year, 95 percent of respondents ratedIEG influence 3 or 4 out of 4. IEG was consideredhelpful to gaining a clear view of lessons by 94percent (mean of 4.7) and to gaining a clear viewof good practices by 86 percent (mean of 4.4).

Among Bank staff respondents, IEG’s rec-ognized strengths at compiling lessons and good


8 2

Figure E.5: Bank Staff’s Satisfaction with Timeliness

Source: IEG data.

practices do not translate into an equally strongrating for informing staff about the make-up ofresults chains (figure E.8). Staff respondentsranked IEG evaluations highest on helping themto gain a clear view of lessons (mean of 3.8, 65percent ranking it 4 or higher) and of goodpractices (3.51, or 52 percent). Influence onhow outcomes are linked to outputs is rankedlowest (3.22, or 44 percent average over project,sector, and country level). Forty-four percentand 46 percent of staff respondents rated IEG’sinfluence on the Bank’s sector and countrystrategies, respectively, at 4 or higher, comparedwith 47 percent rating it somewhat or a greatdeal in 2005.

Use of EvaluationsThis year, the client survey also probed the useof evaluations (figure E.9). This section wasadded to obtain a better understanding ofevaluation use. EDs and advisors were askedhow they use evaluations to assess the Bank’swork. Staff were asked about different modali-ties for use, such as giving advice, commenting,designing new products and strategies, andmodifying ongoing operations and strategies.Two hundred forty-one Bank staff and 53 EDsresponded to this question. Overall, EDs andadvisors use evaluations primarily to fulfill their

oversight function. Bank staff report usingevaluations mostly for commenting, making acase, and providing advice, less for modifyingongoing operations or designing new Bankproducts.

Bank staff use evaluations mostly for comment-ing on the work of others, making a case for aparticular course of action, and providing adviceto clients, and less for modifying strategies or


8 3

Source: IEG data.

Figure E.6: Quality of Studies over Time

Figure E.7: IEG’s Influence on Bank Staff’s Under-standing of Subject Area

Source: IEG data.

operations, or designing new projects orprograms. About one in four (27 percent)respondents acknowledged that they use evalua-tions to modify ongoing operations, assigning arating of 4 or higher. EDs and advisors use IEG

evaluations heavily to assess Bank policies andprocedures; 90 percent of respondents rate it 4or higher (mean of 4.7), followed by use to assescountry strategies (mean of 4.8, 88 percentrating it 4 or higher) and sector strategies (4.6,


8 4

Figure E.8: Bank Staff’s Rating of IEG’s Influence

3.80 3.62

Ratin

g 3.51 3.30 3.29 3.27 3.23 3.22 3.21

1

2

3

4

5

6

Helped yougain a clear

view ofessentiallessonslearned

Influencedyour

understandingof the

subject area

Helped yougain a

clear viewof best

practices

Influencedthe Bank’s

sectorstrategy


countrystrategy

Influencedhow outcomes

linked tooutputs(results

framework)in projects



framework)in thesector



framework)at the

country level


lending andnonlending

services

Source: IEG data.

Note: Mean on a six-point scale, N = 268.

Figure E.9: Bank Staff’s Use of IEG Evaluations

3.60 3.57 3.573.22 3.21 3.05 3.02

2.68

1

2

3

4

Ratin

g

5

6

Commenting onor making inputs

into the workof others

Making thecase for aparticular

course of action

Providingadvice to

your clients

Modifyingpoliciesand/or

strategies

Designing/modifying

your resultsframework

Designingnew lendingoperations

Designingnew nonlending

operations

Modifyingongoing

operations

Source: IEG data.

Note: Mean on a six-point scale. N = 241.

or 81 percent). EDs and advisors use evaluationsto a lesser degree when assessing projects (3.7,or 57 percent).

Recommendations from respondentsWhen asked how to improve the quality ofevaluations, Bank staff respondents selected“make findings more operational” as their toprecommendation. However, this year just 57percent of respondents chose this recommen-dation, compared with 72 percent last year(figure E.10). Nearly half of non-ED respondentsalso suggested that obtaining more evidenceand broadening consultation with Bank staffcould help improve evaluation quality.

EDs and advisors were asked to select from thesame set of recommendations for IEG evalua-tions. Sixty-four percent of respondents saidthey would make the findings more operational,down from 87 percent last year. However, morerespondents than last year suggested increasingexternal consultation (2006, 56 percent; 2005,46 percent) to obtain more evidence (2006, 38percent; 2005, 29 percent) and to broadenconsultation with Bank staff (2006, 38 percent;

2005, 21 percent). Thus, EDs and Bank staff havea similar view on where and how IEG needs toimprove.

Both EDs and Bank staff were asked for theirviews on various recommendations for improv-ing IEG’s outreach (figure E.11). Respondentsgenerally called for more summaries of IEGfindings and suggested better notification aboutnew documents and events. In addition, Bankstaff would like to have more access to evalua-tion data, while EDs and their staff want IEGcontent to be more accessible online.

Results of Survey of External ClientsThis year’s external survey queried 2,759external clients from governments, internationalorganizations, bilateral donor organizations,nongovernmental organizations (NGOs),academia, and the general public. They wereasked about the following:

• Two corporate evaluations– The 2004 ARDE– Capacity Building in Africa: An OED Eval-

uation of World Bank Support


8 5

Figure E.10: Bank Staff Recommendations to Improve IEG’s Evaluation Quality

72

3643

3337 38

5749 48 45

35 35

0

20

40

60

80

100

Make findingsmore operational

Obtain moreevidence

Broadenconsultation with

Bank staff

Improve depth ofanalysis

Broaden externalconsultation

Link conclusions toevidence and

analysis

2005 2006

Perc

ent

Source: IEG data.

• Three sector and thematic evaluations:– Committing to Results: Improving the Ef-

fectiveness of HIV/AIDS Assistance

– Pension Reforms and the Development of

Pension Systems: An Evaluation of World

Bank Assistance

– The Effectiveness of World Bank Support

for Community-Based and -Driven Devel-

opment (CBD-CDD)• One synthesis paper: Improving the World

Bank’s Development Effectiveness: What Does

Evaluation Show?

Surveys were sent to individuals who had eitherreceived an IEG report through an e-mailannouncement or as a hard copy, had partici-pated in a seminar, had been consulted duringthe evaluation, or had placed a request concern-ing a report with the IEG Help Desk. For theCBD-CDD study, Pension Reforms, and thesynthesis paper, surveys were only sent to thosee-mail recipients who had acknowledged receiptof the report by opening the report’s Web page.Thus, as for the internal survey, the sample wastargeted, not random. The following is asummary of survey responses.

Response ratesThe surveys were sent electronically to thereported e-mail addresses of 2,759 externalclients during a two-week period in April 2006.Valid responses were received from 322 people,a response rate of 12 percent. Response rateswere highest where surveys were sentexclusively to those who had acknowledgedreceipt of a report. Multiple reminder notifica-tions that directly targeted nonrespondentswere used. As for the Bank staff survey, resultsfrom the external survey are not generalizable tothe overall surveyed population, but are indica-tive for the sizeable number of respondents.

Asked to identify the type of their organization,about one quarter (27 percent) of respondentswere from international organizations, 15percent from governments, and 9 percent fromgovernmental donor organizations. Fifteenpercent were working with NGOs and 20percent with academia/research organizations.

Readership and awareness More than half (55 percent) of all respondentsread, participated in a discussion, or attended an


8 6

Figure E.11: Bank Staff Recommendations to Improve IEG’s Outreach

67

4940

30 29

16 14

66

4541 40

28 2723

0

10

20

30

40

50

60

70

80

90

100

Provide moresummaries ofIEG findings

Provide betternotification of

new documentsand events

Make IEGcontent more

accessibleonline

Provide moreaccess toevaluation

data

Provide morecustomizedinformation

Provide moretraining/educationmaterials

Provide morematerial inlanguagesother than

English

2005 2006

Perc

ent

Source: IEG data.

event on the evaluation for which they weresurveyed (figure E.12). Overall awareness(including those who were aware of the findingsbut had not read the evaluation or participatedin an event) reached 76 percent on average.

The majority of respondents (69 percent) hadheard about the evaluation via e-mail, 28 percenthad visited the evaluation Web site, and 15percent had received a hard copy. About sevenpercent of respondents stated that they hadbeen consulted during the evaluation.

The majority of respondents had sharedinformation or the findings from the evaluationwith colleagues. Fifty percent shared it with twoto four colleagues and 18 percent shared it withmore than 10 colleagues, expanding the indirectreach of evaluations.

Quality of IEG evaluationsExternal survey audiences were asked to ratetheir satisfaction with different attributes ofquality using a six-point scale. Respondentsrated their satisfaction with the quality of IEGevaluations very favorably (figure E.13). Ease ofunderstanding received the highest rating, witha mean of 4.75 and 89 percent of respondentsrating it 4 or higher, followed by concise presen-tation of conclusions (4.73, or 89 percent).Eighty-five percent of respondents rated theirsatisfaction with the evaluation’s relevance totheir work at 4 or higher (mean = 4.66).Although still high, satisfaction was lowest withdepth of analysis (4.32, or 80 percent) andincorporation of all available information (4.15,or 75 percent).

Comparing these ratings with last year’s results,this year’s respondents rate their satisfactionwith IEG’s quality more favorably overall.

A strong majority of all respondents rated theirsatisfaction with each of the aspects positively:

• Ease of understanding (mean of 4.75, 89 per-cent rated 4 or higher)

• Concise presentation of conclusions (4.73, or89 percent)

• Relevance to their work (4.66, or 85 percent)• Usefulness of recommendations (4.51, or 84

percent)• Transparency and clarity of the methodology

(4.49, or 86 percent)• Timeliness (4.45, or 78 percent)• Unbiased and objective analysis (4.34, or 78 per-

cent)• Strong link between conclusions and evidence

(4.34, or 79 percent)• Depth of analysis (4.32, or 80 percent)• Incorporation of all available information (4.15,

or 75 percent).

Influence of IEG Evaluations External respondents were asked to rate on asix-point scale the extent to which the evalua-tion influenced their understanding of thesubject area, helped them better understand theBank’s activities within a sector, and conveyedessential lessons learned, best practices, andother relevant information to help them makedecision on strategies and programs (figureE.15). Respondents rated all dimensions highly.Influence on their understanding of the subjectarea was rated 4 or higher by 81 percent,compared with 68 percent who rated it as goodor excellent last year.


8 7

Figure E.12: Awareness among External Audiences

4364

54 55

29

1624 21

0

20

40Perc

ent

Audience

60

80

100

Corporate Sector andthematic

Synthesis Overall

Direct awareness Indirect awareness

Source: IEG data.

UseRespondents were asked to rate—using a six-point scale—their use of the evaluation forresearch, making the case for a particular courseof action, refocusing ongoing strategies orprograms and education. A majority of respon-dents assigned the highest “use” ratings fromevaluations to research (mean of 4.0, 69 percent

rating it 4 or higher) and advocacy, that is,making the case for a particular course of action(3.95, 70 percent). Sixty-two percent of respon-dents relied on evaluations to refocus ongoingstrategies. Use of evaluations for educationalpurposes was ranked fourth, with a mean of 3.65and 57 percent of respondents rating it 4 orhigher (figure E.16).


8 8

Figure E.13: External Client Satisfaction with Quality of IEG Evaluations

4.75 4.73 4.66 4.51 4.49 4.45 4.34 4.34 4.32 4.15

1

2

3

4

Ratin

g

5

6

Ease ofunder-

standing

Concisepresentation

ofconclusions


Usefulnessof

recommen-dations


of themethodology

Timeliness Unbiasedand

objectiveanalysis

Strong linkbetween

conclusionsand

evidence

Depth ofanalysis

Incorporationof all


Source: IEG data.


Figure E.14: External Client Satisfaction with Evaluation Quality 2005–06

74 6981

7179808589

0

20

40

60

80

100

Perc

ent

Concisepresentation ofconclusions

Relevance to yourwork

Depth of analysis Strong linkbetween

conclusions andevidence

2005 2006Source: IEG data.

Recommendations from respondentsWhen asked about improvements, the mostoften-selected recommendation was tobroaden external consultation (58 percent), upfrom 31 percent the previous year, followed bymaking findings more operational (52 percent;2005, 61 percent) and improving depth ofanalysis (44 percent; 2005, 50 percent). Thirty-nine percent of respondents recommendedthat IEG obtain more evidence, up from 32percent last year (figure E.17).

When asked what recommendations they wouldmake to improve IEG’s outreach, 58 percent ofexternal survey respondents suggested providingmore summaries of IEG findings. About half of therespondents would like to have more access toevaluation data, more training and educationmaterials, better notification of new documents,and more online accessibility (figure E.18).Overall, there is little difference from last year’srecommendations.

Implications for IEGThe survey results indicate that IEG providesgood services to its main client, the Board.

Board members are largely satisfied with thequality of IEG evaluations and use themextensively to assess the Bank’s activities at thesector and country level.


8 9

Figure E.15: Influence of IEG Evaluations on External Clients

4.51 4.45 4.31 4.26 4.23 4.12

1

2

3

4

5

6

Helped you betterunderstand theBank’s activities

within asector

Helped you gaina clear view of

essentiallessonslearned

Provided you withpractical information

to help you makedecisions on

strategies andprograms

Influenced yourunderstanding

of thesubject area

Helped you gaina clear view ofbest practices

Helped you betterunderstand theBank’s activities

within acountry

Ratin

g

Source: IEG data.


Figure E.16: External Client Use of IEG Evaluations

4.00 3.95 3.81 3.65

69% 70% 62% 57%

1

2

3

4

5

6

Ratin

g

Research Making thecase for aparticularcourse of

action

Refocusingongoing

strategiesor

programs

Education

Source: IEG data.

Note: Percentages show how many respondents rated 4 or higher.


9 0

Figure E.17: External Clients’ Recommendations to Improve IEG Evaluations

31

61

50

3238

11

58

12

28

3944

52

0

20

40

60

80

100

Perc

ent

Broaden externalconsultation

Make findingsmore operational

Improve depth ofanalysis

Obtain moreevidence

Link conclusionsto evidence and

analysis

Broadenconsultation with

Bank staff

2005 2006

Source: IEG data.

Figure E.18: External Clients’ Recommendations to Improve IEG’s Outreach

53 5146 42

3731

21

49 4847

4858

1823

0

20

40

60

80

100

Perc

ent

Provide moresummaries ofIEG findings

Provide moreaccess toevaluation

data

Provide moretraining/

educationmaterials

Provide betternotification

of newdocumentsand events

Make IEGcontent more

accessibleonline

Provide morematerial inlanguagesother than

English

Provide morecustomizedinformation

20062005

Source: IEG data.

External audiences, which consist mainly ofother multilaterals, governments and bilateraldonors, NGOs, and academia, were alsohighly satisfied with the quality of IEG evalua-tions and considered them influential. Thus,IEG is successful in turning its evaluationknowledge into a valuable global public good.However, external audiences would like IEGto broaden its external consultation duringevaluations.

IEG products help deepen staff understandingof a given subject area and lessons, but staff useevaluations less frequently in their day-to-daywork of designing and implementing projectsand programs. Thus, IEG provides high-levelknowledge that is useful for assessing programs,giving advice to clients, and making comments,

but it needs to focus on influencing ongoing andfuture operations.

Across all audiences, the survey results suggestthat improving its timeliness to provide informa-tion when decisions need to be made; deepen-ing its analyses through more consultation andattention to the context of evaluated operations;and focusing on the operational value of itsfindings will enhance IEG’s influence onoperations and subsequently on developmentoutcomes.

As IEG looks for novel ways to contribute to theBank’s results agenda, one possible entry pointmay be to help identify results chains andoperational knowledge “nuggets” that can bereadily applied in ongoing operations.


9 1

9 3

Disclosure of evaluations on the Internet

Before Now Why?

Post electronic document Systematic disclosure clearance process requiring (1) To comply with disclosure requirements

on Web site without TTL sign-off (e-mail) copying relevant IEG group (2) To inform TTLs and managers of postings, to minimize

formal sign-off process managers error in timing or contents of posted items

Notify OPCS prior to disclosure of major evaluations To notify OPCS of major outreach campaigns that may

draw attention from external audiences

E-mail marketing

Before Now Why?

None Conduct targeted internal and external audience To ensure core audiences are reached by e-mail and/or

research and analysis hard-copy distribution, including those identified by previ-

ous AROEs as neglected, such as country office and

operational staff

None Send e-mail announcements for important sector, To significantly increase awareness within the Bank and

thematic, and corporate evaluations to selected externally of IEG’s role and its major evaluation findings

internal and external audiences (ranging from

1,000 to >10,000)

None Include in e-mail announcement a tracking code To monitor who is reached by, and has taken an interest

to identify audience response patterns in, the evaluation. Monitoring mechanisms allow IEG to

tap into virtual networks, for example, by identifying

individuals who forwarded the announcement to

colleagues.

Web promotions

Before Now Why?

None Google AdWords To increase visibility of IEG and its products on Google

None Cross links from key internal and external websites (1) To increase ranking of IEG and its products on Google

search results

(2) To provide links to IEG products from relevant pages,

such as CAEs on the Bank’s country office home pages

None Section on World Bank’s home page dedicated To increase the visibility and reach of IEG content by

to featuring IEG content placing links to new products on the Bank’s high-traffic

home page

APPENDIX F: OVERVIEW: CHANGES IN IEG’S DISSEMINATION METHODS


Media campaigns

Before Now Why?

”Today” stories and Increased coverage of IEG events (IEG Announce evaluation products or events

kiosk announcements conferences, Good Practice Awards, launch events)

None Definition of key messages (1) Identify key messages for internal and external

audiences

(2) Express messages so they are concise, engaging, rele-

vant to the current policy agenda inside the Bank and

outside, consistent across dissemination channels, and

grounded in evaluation evidence

None Media strategy To maximize impact with targeted journalists in major

media organizations covering evaluation topic area

None Press kit: press release, media advisory, fact sheet To provide effective briefing materials for accurate re-

porting and increased coverage

None Press conference To generate interest from media to increase coverage

None Media monitoring following campaign launch To monitor media coverage of evaluation, to measure

impact of campaign

Web site enhancements

Before Now Why?

Simple one- or two-page Content and feature rich Web sites for promoting To make user experience more engaging, to increase re-

Web sites for products completed and ongoing evaluations peat visits and referrals

None Careful attention to Web site messages (1) To ensure consistency across IEG’s message delivery

products, such as the press release, executive summary,

and so forth

(2) To ensure balanced messages that highlight both the

positive and negative and are constructive in tone

Basic Web statistics Enhanced Web statistics tracking software To monitor the impact of outreach campaigns and meas-

software including campaign tracking ure impacts of different outreach instruments

Contact IEG Help Desk Request page for ordering hard copies and To provide internal and external audiences with a quick

providing feedback mechanism for ordering hard copies

Multilingual outreach

Before Now Why?

Translation of print Increase in volume of content translated for (1) To increase the cost effectiveness of IEG’s translation

publications, but limited Web-only publication, or for Web site-only text budget

translation of Web content (2) To reach non-English language audiences through

into other languages electronic media, as recommended by the Bank’s transla-

tion framework

None Partnerships with EXT team to announce IEG To extend dissemination of IEG content electronically in

content in non-English languages on Web sites multiple languages to reach non-English-speaking

and newsletters in relevant languages audiences

None IEG key Web pages in multiple languages To provide visitors to the Bank’s external home page in

Spanish, French, and Arabic with IEG Web pages in these

languages


9 4

IEG help desk

Before Now Why?

Ad hoc requests on wide Significant increase in volume of requests for To integrate the Help Desk into promotional campaigns

range of topics hard copies and questions related to launched for products

products

Note: EXT = External Affairs, Communications and United Nations Affairs Vice Presidency.

A P P E N D I X F : OV E R V I E W: C H A N G E S I N I E G ’ S D I S S E M I N AT I O N M E T H O D S

9 5

9 7

In its most recent progress report on the Bank’sresults agenda (World Bank 2006b), OPCS laidout a set of program actions to be undertakenover the next two years. The salient features ofthe action program are summarized in thefollowing table. The AROE found the plan to be

comprehensive and to reflect elements criticalto furthering the Bank’s results agenda.However, the plan could benefit from morespecificity on how it would be implemented,especially its cost and funding implications. TheAROE’s assessments are noted in table G.1.

APPENDIX G: ACCELERATING THE RESULTS AGENDA

Table G.1: OPCS Action Plan and AROE Assessments

Report section III—Program of action AROE assessment

Pillar I: Strengthening country capacity to manage for results

Country Statistical Capacity Building: Continue to support countries

improve their statistical capacity through DGF in line with

Marrakech Action Plan.

Sectoral Concerns and Management Capacity: Enhance Critical element going forward. “Develop tools for country/sectoral

government capacity to interpret statistics and use them teams to assess the incentive/demand for results based planning” was

for policy implementation at the sectoral and/or identified as a Next Step in the Results Framework (annex B in the

cross-sectoral levels. report). Some Regions (for example, Latin America and the Caribbean)

are more advanced in working with countries on M&E support and eval-

uation capacity development (ECD). The Bank could go beyond analysis,

reporting, and tool development to providing funding for direct support.

This would give M&E and its use more prominence and send a signal to

staff that management considers this important.

Pillar II: Strengthening results in the Bank

Regional action program:

Describe how the Region will organize itself to manage

for results.

Provide just-in-time training for front-line staff. Critical element going forward. (See comments on OPCS’s knowledge

and learning strategy, below.)

Develop a process for identifying regional projects/ programs

that should receive support for the design and implementation

of impact evaluations in coordination with the Development

Impact Evaluation (DIME) Initiative.


Table G.1: OPCS Action Plan and AROE Assessments (continued)


Review, comment, and advise on the results orientation and Critical element going forward. Regions already strengthening review/

quality of all operational products. support functions in vice presidents’ offices.

Review and comment on the preparation of Regional strategy

and performance contracts to ensure both clarity of focus on

results and consistency with regional and country-level

internal budgets and work programs.

Facilitate sector/country discussions of work program alignment Critical element going forward. Regions (with support from OPCS)

should provide support to CDs and country teams to refine the results

framework, identify common CAS and intermediate outcomes, and align

Bank operations (across sectors where relevant) with CAS outcomes.

Engage in preparation and review of Country Assistance

Strategy Completion Reports (CASCRs) and other elements

of the Regional Monitoring and Learning System.

Organize the Region’s contribution to the proposed

Operational Performance and Results Report.

Network Action Program:

Describe how the network will organize itself to manage

for results.

Provide advice on the development of performance indicators Critical element going forward. AROE focus groups stated that country

and implications for design of institutional arrangements teams and task teams are looking to networks for guidance good indica-

and the specification of data requirements for monitoring tors to use. In addition, AROE focus groups raised difficulties teams

sectoral performance at the country level. were facing in coming up with results chains when developing results

frameworks—that is, how to conceptually link higher-level outcomes

with specific Bank operations. They were looking to networks for

support.

Encourage development of methodologies and approaches to Critical element going forward. Some networks are selectively

enable comparative assessments and benchmarking of engaging.

country-level performance in sectors/themes covered by

the network.

Identify priorities for selection of sectoral projects/programs

that should receive support for the design and implementation

of impact evaluations in coordination with the DIME initiative.

Provide just-in-time training for front-line staff. Critical element going forward (see comments on OPCS’s knowledge

and learning strategy, below)

Engage in preparation and review of CASCRs and other

elements of the Regional Monitoring and Learning System.

Organize network’s contribution to the proposed Operational

Performance and Results Report.


9 8


Results Steering Group: Create a Results Steering Group to

replace current network of volunteer results focal points.

Knowledge and Learning: OPCRX will lead the design and Critical element going forward. It includes the key areas identified in

implementation of a results knowledge and learning strategy. AROE. Focus is on traditional learning. Annex F provides details of

Strategy will aim to: learning priorities and proposes two new initiatives—a flagship Man-

– Clarify and promote the managing for results approach among aging for Results training module and an Institutionalization of Manag-

managers at all levels ing for Results self-assessment tool. Bank needs to go beyond passive

– Support staff’s efforts to adopt and apply results approaches support (it did not work in MEIP) but be more active in supporting oper-

and knowledge and to identify factors that influence the ational activities, say with funding for enhancing government capacity

demand for use of information on results in countries’ to use M&E information for policy making as per Pillar 1. The Bank

decision making should also pursue opportunities to incorporate IEG products into

– Facilitate the application of a “results lens,” including the learning strategy.

provision of training to TTLs preparing projects and

identification of the incentive environment at the

country/sectoral level for managing for results.

– Collaborate with sector and networks to heighten learning

about indicators, statistical tools for managing for results

– Improve outreach to a globally dispersed staff

– Ensure the quality and consistency of results knowledge

and learning by further dissemination of good practices.

Results Monitoring and Learning System: To be developed. OPCS should consolidate corporate-level monitoring around a common

Tier 1: Country Indicators—IDA 14 + growth, macroeconomic set of key indicators. OPCS should work with SFRM to use SPCs to link

management, social protection, and trade corporate-level reporting with resource allocation and regional report-

Tier 2: Bank indicators—CASCR ratings, QAG/IEG ratings on ing with the CAS results frameworks and country reporting.

project quality

Global indicators—global programs

Results Reporting: Operational Performance and Results

Report—apex report building on ARPP, CAS Retrospective, and SSIU.

Pillar III: Strengthening global capacity

Collaborative Activities: Continuing activities following from the

Paris Declaration, the OEDC/DAC Joint Venture for Managing for

Results, and the MDB Working Group on Managing for Results

Finalize Sourcebook on Managing for Results

COMPAS

New “Mutual Learning Initiative”

Develop communities of practice

Third Roundtable on Managing for Development Results

Note: OPCRX = Results Secretariat; MEIP = Metropolitan Environmental Improvement Program; SFRM = strategy, finance, and risk management; SPCs =strategy and performance

contracts.

A P P E N D I X G : AC C E L E R AT I N G T H E R E S U LT S AG E N DA

9 9

1 0 1

IEG influences the Bank through recommenda-tions to management as part of sector, thematic,and corporate evaluations, as well as CountryAssistance Evaluations. Management is account-able to the Board for follow-up. One of theintermediate outcomes for IEG is the extent towhich management incorporates IEGrecommendations and findings in policy advice,program design, and project design.

The Management Action Record (MAR) allowsIEG to track recommendations from sector,thematic, and corporate evaluations and tomonitor the degree to which management hasadopted them. The MAR tracks two indicators:the level of adoption and the implementationstatus of individual recommendations. The MARpresents management’s ratings on these twoindicators and IEG’s validation of the same onan annual basis.

The rating categories for the level of adoption ofrecommendations are as follows:

• High—fully adopted• Substantial—largely adopted but not yet fully

incorporated into policy, strategy, or opera-tions

• Medium—adopted in some operational andpolicy work but not to a significant degree inkey areas

• Negligible—no evidence or plan for adoption,or plans and actions for adoption are in a verypreliminary stage.

The categories for the status of recommenda-tions are the following:

• Active and remain actionable by management

• Complete and archived• Obsolete or overtaken by events and archived• Difference of Opinion between management

and IEG.

The MAR contains information on all IEGrecommendations, subsequent managementactions, and IEG’s assessments. Validation ofmanagement actions responds to CODE’sinterest in a transparent documentation of allrecommendations, their implementation status,and substantive differences between manage-ment and IEG as to their level of adoption.

The 2005 AROE included recommendations toimprove the MAR. A pilot three-year expirationcriterion has been introduced for MARrecommendations from fiscal 2006 onward.Recommendations issued in the previous threecalendar years (calendar years 2002–04) arebeing retained in the MAR. Recommendationsfrom earlier years (calendar year 1998–2001) aresubmitted to CODE/the Board and archived,subject to review by CODE/the Board.

The 2006 Management Action RecordThe 2006 MAR tracks management actions on 79recommendations. These include 21 newrecommendations from the six IEG studies(excluding CAEs) presented to the Board incalendar year 2005, and 58 recommendationscarried forward from calendar year 2002 to 2004.

For the 2006 MAR, IEG rated 66 percent of itsrecommendations as having been adopted bymanagement at high or substantial levels (figureH.1). In 2005, 75 percent of recommendationswere noted by IEG as having been adopted at ahigh or substantial level, compared with 62

APPENDIX H: WORLD BANK MANAGEMENT ACTION RECORD

percent in 2004. The ratings for 2006 reflect atrend of high or substantial adoption bymanagement of about two-thirds of IEG’srecommendations. This year, for six percent ofrecommendations, implementation was ratedby IEG as negligible, which is slightly higherthan in previous years. There is a difference ofopinion between management and IEG on tworecommendations; these have therefore notbeen rated and are not included in the percent-ages on adoption.

IEG and management agreed on the rating onlevel of adoption for 65 percent of recommenda-tions (50 of 77). This represents a decline from 70percent in 2005 and 78 percent in 2004. It shouldbe noted that the introduction of the three-yearexpiration rule for recommendations reduces thecomparability of these numbers over time,because the 2006 MAR contains proportionallymore recent evaluations than the MARs fromprior years. Where there was disagreement in thelevel of adoption, IEG’s ratings were lower thanmanagement’s ratings for all except one of therecommendations. Table H.1 shows the distribu-tion of adoption ratings by management and IEG.

Fifty-eight recommendations have been carriedover from the 2005 MAR to the 2006 MAR. IEGupgraded its rating of adoption for about one-

fourth (14) of them and downgraded it for fiverecommendations. Management downgradedits rating for one recommendation andupgraded eight, five of which IEG concurredwith. Management considered 14 recommenda-tions as completed. IEG agreed in seven cases.

IEG and management agreed on the level ofadoption for about three-quarters (43 out of 58)of these recommendations. IEG rated 79percent of the recommendations with “high” or“substantial” for the level of adoption. Examplesfor positive follow-up action to IEG recommen-dations from calendar year 2002–04 are thereassessment of the Bank’s approach toinfrastructure in response to IEG’s evaluation ofprivate sector development in the power sector;reforms of the PRSP Joint Staff Assessmentfollowing IEG’s PRSP study; and changes in theHIPC debt sustainability analysis in response toIEG’s HIPC evaluation.

Of the 21 new recommendations entered intothe MAR during calendar year 2005, IEG concurswith management’s adoption ratings on 40percent (8) of the recommendations. There isone difference of opinion. Management rates itsadoption high or substantial for 85 percent (17of 20) of the recommendations and medium for15 percent. IEG considers the adoption of 29


1 0 2

Figure H.1: Level of Adoption of IEG Recommendations Similar to Previous Years

Source: IEG data.

percent of the recommendations “high” or“substantial,” 43 percent “medium,” and 24percent “negligible.”

For MAR recommendations from calendar year1998–2001 that are being archived, among 7 IEGreports, 10 recommendations are incomplete,and there is a difference of opinion on onerecommendation.

Selected Examples of Follow-up Action The 2006 MAR contains a number of recommen-dations that have prompted responsive actionsby management and improvements of Bankoperations. MAR recommendations that havehad an impact include IEG’s review of lines ofcredit (IEG 2006e), the global programs review(IEG 2004a), and the evaluation of the WorldBank Group’s experience with extractiveindustries (IEG-WB, -IFC, -MIGA 2005).

IEG’s 2004 evaluation of lines of credit (LOCs)offers a good example of corrective action bymanagement following IEG recommendations.IEG found poor results of Bank-funded LOCsand widespread lack of compliance withoperational policies (OPs) governing LOCs. Itrecommended that Bank management updateOP 8.30 to cover all LOCs across all sectors andRegions. CGAP’s self-evaluation of LOCs forcompliance with OP 8.30 confirmed IEG’sfindings. To address IEG’s recommendations,

management subjected all identified LOCs to aquality assurance system to foster compliancewith OP 8.30. Comments from a staff surveyindicate that the evaluation has affectedprojects; for example, one comment was:“evaluation of Bank LOCs has provided

guidance on our recent design of an LOC in the

area of micro and small lending. Lessons

learned in that report have proved very enlight-

ening.”

IEG’s evaluation of the World Bank’s approachto global programs found that the Bank’sselectivity and oversight of its global programportfolio had been weak, because the existingselectivity criteria were broad and difficult toapply and because of poorly defined expecta-tions with respect to the roles, responsibilities,and accountabilities of Bank staff who overseeindividual programs.

In response to IEG’s recommendation that theBank improve its selectivity and oversight ofglobal programs, management has put in placenew business processes that recognize GPPs as aseparate Bank product line and has initiatedwork on a monitoring framework and indicatorsto assist global program task managers inimproving oversight. QAG has also initiatedquality-at-entry reviews of new DGF-supportedprograms to help strengthen the adoption ofthese new business processes. Management

A P P E N D I X H : WO R L D B A N K M A N AG E M E N T AC T I O N R E C O R D

1 0 3

IEG ratings Sum of Difference management

Management ratings High Substantial Medium Negligible of opinion ratings

High 17 8 7 2 34

Substantial 25 6 2 33

Medium 1 8 1 10

Negligible

Difference of opinion 2 2

Sum of IEG ratings 17 34 21 5 2 79

Source: IEG data.

Note: Highlighted fields show agreement between management and IEG; fields to the right of the diagonal represent higher rating by management than by IEG, fields to the left of the

diagonal show higher rating by IEG than by management. The sixth column shows the sum of ratings in each category by management; the sixth row shows the sum of ratings by IEG.

Table H.1: Management and IEG Ratings of Recommendation Adoption Levels

rated the level of adoption of all four recommen-dations from this report high; IEG rated theadoption as substantial for two recommenda-tions and high for the remaining two.

IEG’s evaluation of the World Bank Group’sexperience with extractive industries (EI) foundthat, while the Bank’s EI projects have producedpositive economic and financial results, the Bankcould do more to enhance the EI sector’s contri-bution to sustainable development and povertyreduction. Among the issues highlighted wereensuring greater compliance with environmen-tal and social safeguards, promoting improvedgovernance and management of EI revenues,and developing an integrated sector strategythat went beyond returns to investors.

Following IEG’s recommendations, manage-ment has adopted a clearer strategy for thesector and is mainstreaming new guidelines forbetter addressing EI sector issues in strategydocuments (CAS), particularly for low-incomecountries under stress. Guidelines are beingprepared for improved upstream screening ofprojects for safeguards requirements. The EITransparency Initiative is being expanded, andthe distribution of project benefits among keystakeholders is being done as standard practice.A World Bank Group community developmentfacility has been established for increasing localcommunity participation in EI projects. Thelevel of adoption for all recommendations fromthe EI evaluation was rated high by bothmanagement and IEG in the 2006 MAR.


1 0 4

1 0 5

IntroductionThe 2006 Annual Report on Operations Evalua-

tion (AROE) updates the actions taken since the2004 and 2005 AROEs to strengthen the resultsfocus in monitoring and evaluation. It exploresfirst the implications for managing for results inBank operations and second the extent to whichthe Bank’s M&E systems provide staff with theinformation they need to better manage forresults. The report also evaluates the productsand services of IEG. The AROE also makesrecommendations for further progress onimplementing the results architecture and onways IEG might encourage more timely andwidespread use of IEG findings by operationalstaff.

Emerging results architectureManagement welcomes this AROE, the third in asmany years to focus on the Bank’s progress inarticulating a results architecture and proceduresfor sharpening its focus on results. This year’sreport confirms the relevance of the emergingresults architecture. That architecture isgrounded in the country business model thatenvisions a results management system focusedon the articulation of an RBCAS as the overarch-ing framework for prioritizing the Bank’sprogram of lending and knowledge support toclient countries. Key supporting elements are theCAS mid-term progress report and self-evaluationthrough the preparation of a CASCR. Last, theCASCR ratings of program achievements andBank performance are validated by IEG.

The AROE’s analysis confirms management’sown assessment of recent progress (World Bank2006b) and contributes more in-depth analyses,particularly of RBCASs and feedback from staff

focus groups on the opportunities andchallenges of strengthening the results focus ofoperational activities. The analysis is a usefulcomplement to management’s work to deepenits understanding of the key factors that willdetermine the success of its results initiative,notably how best to encourage staff efforts andimprove the quality and utility of the key instru-ments in the results architecture.

AROE AnalysisThe AROE catalogues the measures taken tostrengthen the Bank’s results orientation inM&E at the country, sector, and product levels,updating the overview in the 2003 and 2004AROEs and focusing particularly on the currentstatus of the M&E tools. Like management’srecent update on the results agenda (WorldBank 2006b), it notes progress but also areas forimprovement.

Challenges to developing results-basedmonitoringThe AROE’s compilation of the challenges todeveloping results-based monitoring systems,derived from focus group discussions within theBank, yields a familiar listing of key issues: (a)accelerating work to help staff overcome gaps intheir understanding of the practical dimensionsof constructing useful results frameworks, (b)ensuring client capacity and buy-in, and (c)helping country and task teams integrate thedemands of the results agenda into alreadyoverloaded work programs.

Overcoming internal challengesWork to overcome many of these conceptualconstraints to more active implementation(moving from seeing M&E as primarily a tool of

APPENDIX I: MANAGEMENT RESPONSE

relevance to project completion to viewing it asa tool to facilitate responsive managementduring implementation) is under way. Notably,management is developing a Bank-wide learningprogram that is supported by the Knowledgeand Learning Board.

A critical element of creating a results culture inthe Bank is to align the incentive and rewardsstructure with the results agenda. To improveincentives, management has made portfolioquality oversight a required performance evalua-tion item for sector managers. Staff have alsobeen recognized Bank-wide in a variety of waysfor their contributions to results, notablythrough the President’s Awards for Excellence,IEG’s Good Practice Awards, and the ResultsShowcase, which brought competitivelyselected good practice cases to senior manage-ment attention.

In addition, management has moved to increasethe focus on results in its learning and accounta-bility processes around operational products(RBCASs, operational product reviews duringpreparation and implementation, and the QAGprocess). More needs to be done, and manage-ment is committed to continuing to focus onincentives and expanding programs tocommunicate to staff the importance of thisagenda. Two key elements going forward aremanagement attention to the IDA 14 resultsmeasurement system and the results monitoringand learning system under preparation (see theresults framework for the agenda on managingfor results in the recent update) (World Bank2006b, p. 33).

The global element in the focus on resultsThe AROE reviews progress internally in improv-ing the results focus and externally in support-ing countries in strengthening their focus onresults. Management notes, however, that itswork on the results agenda includes a thirdelement, at the “global” level. The Bank has putconsiderable effort into harmonizing itsapproach with that of other developmentpartners and working with them to align withcountry results efforts, a topic that this year’s

AROE analysis does not explore. Management’srecent update, discussed at the Board’s Commit-tee on Development Effectiveness on March 8,2006, documents the progress on this front(World Bank 2006b, p. 24).

Challenges at the country levelThe AROE rightly cites these challenges. As notedin the recent update, in countries preparingPRSs, the Bank has continued to encouragecountries to improve their PRS monitoringsystems through ESW, lending for public sectormanagement, training programs, and knowledgesharing. The Bank has given attention to helpingpartner countries gather national and sub-national data for monitoring progress towardtheir PRS goals. Analytic work and policy advicefocus on aligning allocations with the PRS priori-ties and on achieving operational efficiency,service delivery, and outcomes.

However, more needs to be done to stimulategreater demand for results within and across theline agencies that are most directly responsiblefor the delivery of public goods and services, aswell as from citizens. One implication is thatwhat is needed is a shift in incentives—fromdonor-driven results reporting to building thecountries’ capacity to provide credible evidenceof effectiveness to their own citizens. Increas-ingly, management seeks to signal to staff thatthey are accountable for helping their countrypartners build systems and procedures thatenhance accountability and transparency. Oneelement in this process is the DevelopmentImpact Evaluation Initiative. This strategicscaling up of impact evaluations will providecountries with additional information that theycan use in evidence-based decision making.

Self-evaluation tools and the bank’s resultsagendaManagement agrees with the AROE analysis thatthere is a need to bring a results focus to themethods and tools that are used to improveperformance during project implementation.The Rapid Results Approach cited by the AROEseems to hold promise, and management isassessing this experience. Management


1 0 6

envisions this as the first of a series of collabora-tive learning activities across Regions, networks,and central service units designed to developtools and examples that task and country teamscan use in “real time” as they struggle to meet—and to help countries meet—the challenge ofmanaging for results.

AROE RecommendationsThe AROE 2006 proposes two core recommen-dations for management consideration:

� Building on progress in advancing the resultsagenda, agree on a three-year action programwith a corresponding budget to fund imple-mentation of the next stage, differentiatingnew allocations and redeployed resources.Critical action items would be as follows:• Support country directors and country

teams in their efforts to refine and use re-sults frameworks at country and sectorallevels to manage country programs. Similarefforts need to be undertaken to strengthenthe results frameworks of thematic andglobal programs.


• Strengthen incentives and accelerate aresults-oriented training and communica-

tions program for management and staff toencourage use of M&E information.

• Identify and support in-depth learning op-portunities to develop and use results-basedapproaches with operational teams, partic-ularly in challenging country cases and incomplex multisectoral settings.

� To enhance countries’ capacity and demand tomanage for results by strengthening the col-lection and use of performance information,provide budgetary support to task teams andtechnical advice to countries that intend to in-stitutionalize M&E systems.

Management viewsManagement broadly agrees with the thrust ofthese recommendations, with one majorexception. A results focus is the responsibility ofcountry teams and task teams and needs to beincluded in the formulation of RBCASs and intask preparation and task supervision budgets.To move forward on agreed priorities, a ResultsSteering Group has been formed to facilitatecommunication and coordination among Re-gional and network results teams, which are nowcharged with responsibility for leadership inidentifying the priorities for implementing theaction plan proposed in the recent progressreport. A response to the AROE recommenda-tions is included in the Summary of ManagementResponse to IEG Recommendations (p. xv).

A P P E N D I X I : M A N AG E M E N T R E S P O N S E

1 0 7

1 0 9

Chapter 21. The figure was adapted from an actual country

example in the East Asia and Pacific Region. It is based

on a middle-income country with significant Bank

program. Smaller country programs may have fewer

projects. In such cases, a results chain may help to

identify the key gaps in the country program.

2. Performance indicators, taken from the Project

Appraisal Document, had been included in an annex

to the PSR and were often ignored and not updated.

The ISR brought these indicators into the main doc-

ument, giving them greater visibility.

3. The 2005 AROE found that only 18 percent of

the 25 CASs reviewed were considered well targeted

and 13 percent adequately baselined. However, the

RBCAS had not been mainstreamed when these CASs

were prepared, so a direct comparison between 2005

and 2006 CAS reviews is not appropriate.

4. Milestones are “outputs, actions, or outcomes

expected to be realized during CAS implementation

and which can serve as progress makers toward CAS

outcomes.”

5. QAG went on to observe that problem of defin-

ing indicators was further compounded by the absence

of practical intermediate indicators and failure to reg-

ularly update the key performance indicators and to use

them to shed light on obstacles to outcomes through

out the implementation period (World Bank 2005b).

6. The Bank’s safeguard and fiduciary policies had

led to a doubling of the Regional procurement and fi-

nancial management staff between fiscal 1998 and

2003. Between fiscal 2000 and 2003, headquarters ex-

penditures on compliance with environmental and so-

cial safeguards, including costs during project

preparation, increased by 52 percent from an esti-

mated $21 million to about $32 million. No update on

these costs is currently available.

7. A program chain logic assessment would re-

view: (a) the deployment and sequencing of the ac-

tivities, resources, or policy initiatives to achieve the

project or program outcome (the desired change in

beneficiaries) and (b) the plausibility of achieving

that desired outcome, based on similar prior efforts

and on the research literature. The purpose is to

avoid failure from a weak design that would have lit-

tle or no chance of success in achieving its intended

outcomes.

8. Impact evaluation establishes a counterfactual

to make a comparison of the situation with and with-

out the project or program.

9. OPCS announced the creation of impact evalu-

ation as a new AAA product line, effective July 14, in

an effort to respond to an increasing demand for im-

pact evaluation both within the country and the Bank.

This would also allow the Bank to track the amount

of Bank resources being used for such activities (World

Bank 2005d).

10. IEG’s evaluation of extractive industries found

that ex post cost-benefit analysis was done in 80 percent

of reviewed investment projects, but hardly at all in

technical assistance or sectoral adjustment loans. Sim-

ilarly, IEG’s recent evaluation of agricultural water man-

agement found that “project economic analysis seems

to have been allowed to slide over the past 20 years.”

Also, IEG found, in these and other evaluations, that

many cost-benefit analyses are flawed or too simplistic.

Chapter 31. IEG’s independence has been validated ac-

cording to the standards of OECD/DAC and the U.S.

Government Accountability Office (among others)

(IEG 2003).

2. The three key criteria for assessing the results

orientation of M&E are the results framework, de-

sign of M&E, and implementation of M&E (IEG 2004b,

box 4).

3. A good example for a CASCR and CASCR Review

is the Philippines CASCR from April 2005 and IEG’s re-

ENDNOTES

view from May 2005. This CASCR won IEG’s 2006

Good Practice Award for a “candid and thoughtful as-

sessment of the achievements and deficiencies in im-

plementing the Country Program” (quoted from

http://www.worldbank.org/ieg/awards).

4. A desk review showed that all 38 calendar year

2005 PPARs used the word outcome an average of 16

times, but only 53 percent (20) of these PPARs used

the word output more than once.

5. The award categories are Projects, ICRs, Coun-

try Programs, Country Assistance Strategy Completion

Reports (CASCRs), Monitoring and Evaluation, and Ini-

tiatives with Demonstrated Impact/Results.

6. During the 12-month period April 2005–March

2006, ECD materials accounted for 36 percent of total

recorded downloads of IEG documents.

7. In previous years, these were yes/no questions

and four-point scales. This year, respondents were

asked to rate all questions in the three sections on a

six-point scale. For further details on the conversion,

see appendix D.

8. For example, one respondent wrote, “It was not

really useful or relevant to our work. We would have

liked the evaluation team to think on [sic] the context

in which we operate (rather than on its Washington

stakeholders), so as to contribute ideas that can be use-

ful for our teams at the trenches.”

9. In the first four months of 2006, IEG was men-

tioned more than 20 times in leading English lan-

guage mass media, which is almost triple the number

of mentions of IEG in each year from 1996 to 2004.

10. Level of adoption ratings are high–fully adopted;

substantial–largely adopted but not fully incorporated

into policy, strategy, or operations as yet; medium–

adopted in some operational and policy work but not

to a significant degree in key areas; and negligible–no

evidence or plan for adoption, or plans and actions for

adoption are in a very preliminary stage.

11. The status of recommendations is rated as ac-

tive and remain actionable by management; complete

and archived; obsolete or overtaken by events and

archived; and difference of opinion between man-

agement and IEG.

12. Due to the new 3-year expiration criterion,

comparability between the numbers of rated recom-

mendations in the 2006 MAR to MARs from prior years

is limited.

13. Two recommendations constitute a difference

of opinion and did not receive an adoption rating.

These are excluded from the sample when compar-

ing levels of adoption.

Chapter 41. The Bank has been focusing on improving the

outcomes of its projects since the Wapenhans Task

Force Report in the early 1990s. In response to the re-

port, the Bank introduced project logframes, which

linked project components with outputs, project de-

velopment objectives and country (CAS) objectives,

and sector key performance indicators, which could

be used to monitor performance. QAG’s review of

quality of supervision for fiscal 1999 found that mon-

itoring was satisfactory in 58 percent of the projects

reviewed and that task teams do not know how to im-

prove monitoring and evaluation or do not have the

resources to do so, despite technical support from

OPCS (or its equivalent at that time). The Bank re-

sponded with an effort to strengthen M&E at the

project level. A working group, organized in 1999 to

come up with an action plan, concluded that the

weaknesses in M&E were due to (a) limited incentives

to systematically focus on results, (b) unclear roles and

accountabilities in the Bank and between the Bank and

borrowers, and (c) inadequate capacity to carry out

M&E.

Appendix A1. For investment loans/credits, Template and

Guidelines for the Project Appraisal Document, from

Operations Policy and Country Services (OPCS) is

used. For development policy loans, OPCS’s Good

Practice Note for Development Policy Lending: Results

in Development Policy Lending is used (World Bank

2005a).

2. A new OP/BP 8.60, Development Policy Lending,

was approved in August 2004 to replace the previous

OD 8.60, Adjustment Lending, and nine operational

memoranda. A series of good practice notes, includ-

ing one on results, has been issued to provide guid-

ance to staff on specific aspects when designing and

implementing a development policy loan.

3. Previously the performance indicators were in-

cluded in an annex, which was not automatically at-

tached to the report when it was viewed electronically.

The annex tended to be neglected and data often

were not updated regularly.

4. In its fiscal 2005 Annual Report on Portfolio Per-

formance, the Quality Assurance Group (QAG) noted


1 1 0

a slowdown in analytical and advisory activities (AAA)

implementation and attributed this to a lack of man-

agement attention (after the concept stage) and dif-

ficulties in using the system. QAG observed that “some

of those leading the AAA work in the Regions view the

Bank-wide task reporting system as virtually unus-

able and pay little attention to it” and that the users

“argue that creation of a separate code for each task

combined with the lack of user-friendliness in in-

putting information and updating the system leads to

inaccuracy in reporting.”

5. About 75 percent of the CPPRs reviewed were

attempting to incorporate elements of a results ap-

proach, although only 16 percent could be considered

results-based country performance reviews.

6. The IDA 14 Results Measurement System in-

cludes 14 country outcome indicators covering areas

that are consistent with the MDGs, are priorities in

Poverty Reduction Strategies, and reflect the Inter-

national Development Association’s (IDA) activities:

income/poverty, health, gender, basic education, water

supply, infrastructure, private sector development,

governance, and economic development.

7. The Annual Report on Portfolio Performance re-

ports on the percentage of project that have meet

their PDOs, assesses the percentage of ongoing proj-

ects which may be at risk of not achieving their ob-

jectives, and recommends actions the Bank needs to

take to remedy problems identified through the

analysis.

8. These are percent of IDA projects with (a) sat-

isfactory outcome ratings, (b) satisfactory quality at

entry ratings, (c) first ISRs with satisfactory baseline

data on expected outcomes, and (d) Implementation

Completion Reports with satisfactory data on project

outcomes.

9. The COMPAS master matrix of categories and in-

dicators is included as annex 4.1 in the Global Mon-

itoring Report (World Bank 2006a).

10. Revised formally through approval by the Bank

authority that approved the original loans, credits,

or grants.

11. Previously, the ACS was required for activities

costing more than $50,000. In 2004, the Bank simpli-

fied the tracking of AAA but required ACS for all ac-

tivities, irrespective of cost, because of the increase

in the number of tasks costing less than $50,000 and

the Bank’s emphasis on tracking and results and the

impacts of operational activities.

12. The three GPRs were on Partnership for Child

Development, the Trust Fund for Statistical Capacity

Building, and the Global Corporate Governance Forum.

Appendix C1. In addition, IEG reviews the CASCR.

2. The section on implementation states: “The

Bank collaborates with the government and external

partners in implementing CAS activities. The country

team monitors and evaluates progress toward achiev-

ing CAS objectives, including outcomes identified in

the results framework. The CAS M&E system uses na-

tional M&E systems, where possible, and draws on the

M&E arrangements built into Bank operations.”

3. Capability maturity models were first developed

at Carnegie-Mellon University in the mid-1980s as a way

of comparing management practices of private sector

companies providing services to the U.S. Department

of Defense. Maturity models have since been devel-

oped and employed across a wide variety of man-

agement practices. A maturity model identifies key

characteristics that an evaluator expects to find as

the process in question becomes more fully developed

and integrated into related management activities.

The maturity model was chosen for this analysis be-

cause it does not require a particular process to be em-

ployed; it requires only that the process chosen

produces the desired characteristics.

4. The CASs covered the following countries: Al-

bania, Bangladesh, Bhutan, Burkina Faso, Cambodia,

Djibouti, the Dominican Republic, Egypt, El Salvador,

Gabon, Georgia, Guatemala, Jamaica, Lao People’s

Democratic Republic, Morocco, Nigeria, Philippines,

Poland, São Tome and Principe, Sierra Leone, Tajikistan,

Timor-Leste, Uganda, Uruguay, and Uzbekistan.

Appendix E1. Before 2005, individuals did not receive a unique

survey login and could enter the survey multiple

times. They might have had multiple records within

the database. For the 2005 and 2006 surveys, indi-

viduals were limited to a single record but could re-

enter that record until they had completed the survey

to their satisfaction.

2. This new approach to obtaining IEG client

views is based on the Massachusetts Institute of

Technology Center for Technology, Policy, and In-

dustrial Development approach to managing infor-

mation quality, which compares consumers’ ratings

E N D N OT E S

1 1 1

of importance of attributes of information quality with

their satisfaction with these attributes. Assuming

that all attributes of quality listed in the survey are

important, clients rated their satisfaction with these

attributes. In previous years, clients rated evalua-

tions on the attributes themselves. The switch to a

six-point scale was made for two reasons: to achieve

coherence with the six-point scale of satisfaction

(highly unsatisfactory-highly satisfactory) that is used

in IEG evaluations and has also recently been adopted

by QAG; and to provide room for clearer distinction

between the different attributes.


1 1 2

1 1 3

Abadzi, Helen. 2006. Efficient Learning for the

Poor: Insights from the Frontier of Cognitive

Neuroscience. Directions in Development.

Washington, DC: World Bank.Behn, Robert D. 2003. “Why Measure Performance?

Different Purposes Require Different Measures.”Public Administration Review 63 (5).

Bennett, Charles, and Herb Hill. n.d. “Perfor-mance Management Systems: The Impor-tance of Defining Their Purpose.” Richmond,Virginia: Virginia Department of Planning andBudget. http://www.dpb.state.va.us/VARe-sults/Material/PolicyService.pdf.

Binnendijk, A. 2001. “Results-Based Managementin the Development of Co-operation Agen-cies.” Paris: Organisation for Economic Co-operation and Development–DevelopmentAssistance Committee.

Harvard University, Kennedy School of Govern-ment, Public Sector Performance Manage-ment, Executive Session. 2001. Get Results

Through Performance Management: An

Open Memorandum to Government Execu-

tives. Cambridge, MA.Hatry, Harry. 2003. “Results Matter: Suggestions

for a Developing Country’s Early OutcomeMeasurement Effort.” In Handbook on Pub-

lic Sector Performance Review, Volume II,Measuring Governement Performance in

the Delivery of Public Services, ed. A. Shah.Washington, DC: World Bank.

IEG (Independent Evaluation Group-WorldBank). Various years. IEG Good Practice AwardWeb site. http://www.worldbank.org/ieg/awards.

———. 2006a. Assessing World Bank Support for

Trade, 1987–2004. Washington, DC: WorldBank.

———. 2006b. Engaging with Fragile States: An

IEG Review of World Bank Support to Low-

Income Countries Under Stress. Washington,DC: World Bank.

———. 2006c. From Schooling Access to Learn-

ing Outcomes: An Unfinished Agenda. An

Evaluation of World Bank Support to Pri-

mary Education. Washington, DC: WorldBank.

———. 2006d. Hazards of Nature, Risks to De-

velopment: An IEG Evaluation of World Bank

Assistance for Natural Disasters. Washing-ton, DC: World Bank.

———. 2006e. World Bank Lending for Lines of

Credit: An IEG Evaluation. Washington, DC:World Bank.

———. 2005a. 2005 Annual Report on Opera-

tions Evaluation. Washington, DC: WorldBank.

———. 2005b. Capacity Building in Africa: An

OED Evaluation of World Bank Support.

Washington, DC: World Bank.———. 2005c. Committing to Results: Improv-

ing the Effectiveness of HIV/AIDS Assistance:

An OED Evaluation of the World Bank’s As-

sistance for HIV/AIDS Control. Washington,DC: World Bank.

———. 2005d. Country Assessment Evaluation

Retrospective: An OED Self-Evaluation. Wash-ington, DC: World Bank.

———. 2005e. The Effectiveness of World Bank

Support for Community-Based and -Driven

Development. Washington, DC: World Bank.———. 2005f. “Mandate of the Director-General,

Operations Evaluation.” http://www.world-bank.org/ieg/about.html.

———. 2004a. Addressing the Challenges of

Globalization: An Independent Evaluation

of the World Bank’s Approach to Global Pro-

grams. IEG Study Series. Washington, DC:World Bank.

BIBLIOGRAPHY

———. 2004b. 2004 Annual Report on Opera-

tions Evaluation. Washington, DC: WorldBank.

———. 2004c. Evaluation Capacity Develop-

ment: OED Self-Evaluation. Washington, DC:World Bank.

———. 2004d. The Poverty Reduction Strategy

Initiative: An Independent Evaluation of the

World Bank’s Support Through 2000. Wash-ington, DC: World Bank.

———. 2003. “Independence of OED.” OED

Reach. February 23. http://lnweb18.world-bank.org/oed/oeddoclib.nsf/b57456d58aba40e585256ad400736404/50b9b24456b788be85256ce0006ee9f8/$FILE/Independence_Reach.pdf.

———. 2002. The World Bank’s Approach to

Global Programs: An Independent Evalua-

tion. Phase 1. Washington, DC: World Bank.IEG-World Bank, -IFC, and -MIGA (Independent

Evaluation Group-World Bank, -InternationalFinance Corporation, and -Multilateral In-vestment Guarantee Agency). 2005. Extractive

Industries and Sustainable Development—

An Evaluation of World Bank Group Expe-

rience. Washington, DC: World Bank.———. 2003. Power for Development: A Review

of the World Bank Group’s Experience with

Private Participation in the Electricity Sector.

Washington, DC: World Bank.IEG/Thematic Group for Poverty Analysis, Mon-

itoring, and Impact Evaluation. 2006a. “Con-ducting Quality Impact Evaluations underBudget, Time and Data Constraints.” Wash-ington, DC: World Bank.

———. 2006b. “Institutionalization of Monitor-ing and Evaluation Systems to Improve Pub-lic Sector Management.” ECD Working PaperSeries, No. 15. Washington, DC.

Kusek, Jody, and Ray Rist. 2004. Ten Steps to a

Results-based Monitoring and Evaluation

System. Washington, DC: World Bank.Mackay, Keith. 2006. “Institutionalization of Mon-

itoring and Evaluation Systems to ImprovePublic Sector Management.” IEG EvaluationCapacity Development Working Paper Series,No. 15. Washington, DC: World Bank.

Rist, Ray, and N. Stame, eds. 2006. From Studies

to Streams: Managing Evaluative Systems.

Comparative Policy Evaluation, Vol 12. NewBrunswick, NJ: Transaction Publishers.

World Bank. n.d. Template and Guidelines for

the Project Appraisal Document. Washing-ton, DC.

———. 2006a. 2006 Global Monitoring Report.

Washington, DC.———. 2006b. Accelerating the Results Agenda:

Progress and Next Steps. Washington, DC.———. 2006c. FY05 Annual Report on Portfo-

lio Performance. Quality Assurance Group.Washington, DC.

———. 2006d. Quality at Entry in FY04–05

(QEA7). Quality Assurance Group. Washing-ton, DC.

———. 2005a. Good Practice Note for Devel-

opment Policy Lending: Results in Develop-

ment Policy Lending. Washington, DC.———. 2005b. Quality of Supervision in

FY03–04 (QSA6). Quality Assurance Group.Washington, DC.

———. 2005c. “Reporting in the ISR onProgress Toward Outcomes for InvestmentLending Operations.” Operations Policy andCountry Services–Advisory Services. Wash-ington, DC.

———. 2005d. “Updating the Bank’s Opera-tional Policy on Monitoring and Evaluation.”Consultation Paper. Washington, DC.

———. 2004. “Implementation of the Agenda onManaging for Results: Progress Report.” Op-erations Policy and Country Services. Wash-ington, DC.

———. 2003a. Country Assistance Strategies:

Retrospective and Future Directions. Oper-ations Policy and Country Services. Washing-ton, DC.

———. 2003b. Quality of Supervision in FY01/02

(QSA5). Quality Assurance Group. Washing-ton, DC.

———. 2000. Report of the Working Group on

Improving Quality of Monitoring and Eval-

uation in Bank Financed Projects. Wash-ington, DC.

The World Bank, Portfolio Management TaskForce. 1992. Effective Implementation: Key to

Development Impact: Report of the World

Bank’s Portfolio Management Task Force


1 1 4

(also known as Wapenhans Report). Wash-ington, DC.

Zaltman, Ariel. 2006. “Experience with Institu-tionalizing Monitoring and Evaluation Sys-

tems in Five Latin American Countries: Ar-gentina, Chile, Columbia, Costa Rica, andUruguay.” IEG ECD Working Paper Series No.16. Washington, DC.

B I B L I O G R A P H Y

1 1 5

Study Series2004 Annual Review of Development Effectiveness: The Bank’s Contributions to Poverty ReductionAddressing the Challenges of Globalization: An Independent Evaluation of the World Bank’s Approach to Global Programs Agricultural Extension: The Kenya ExperienceAssisting Russia’s Transition: An Unprecedented ChallengeBangladesh: Progress Through PartnershipBrazil: Forging a Strategic Partnership for Results—An OED Evaluation of World Bank AssistanceBridging Troubled Waters: Assessing the World Bank Water Resources StrategyCapacity Building in Africa: An OED Evaluation of World Bank Support The CIGAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural ResearchCountry Assistance Evaluation Retrospective: OED Self-Evaluation Debt Relief for the Poorest: An OED Review of the HIPC InitiativeDeveloping Towns and Cities: Lessons from Brazil and the PhilippinesThe Drive to Partnership: Aid Coordination and the World BankEconomies in Transition: An OED Evaluation of World Bank Assistance The Effectiveness of World Bank Support for Community-Based and –Driven Development: An OED EvaluationEvaluating a Decade of World Bank Gender Policy: 1990–99Evaluation of World Bank Assistance to Pacific Member Countries, 1992–2002 Financial Sector Reform: A Review of World Bank AssistanceFinancing the Global Benefits of Forests: The Bank’s GEF Portfolio and the 1991 Forest Strategy and Its ImplementationFiscal Management in Adjustment LendingIDA’s Partnership for Poverty ReductionImproving the Lives of the Poor Through Investment in CitiesIndia: The Dairy RevolutionInformation Infrastructure: The World Bank Group’s ExperienceInvesting in Health: Development Effectiveness in the Health, Nutrition, and Population SectorJordan: Supporting Stable Development in a Challenging RegionLesotho: Development in a Challenging EnvironmentMainstreaming Gender in World Bank Lending: An UpdateMaintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh The Next Ascent: An Evaluation of the Aga Khan Rural Support Program, PakistanNongovernmental Organizations in World Bank–Supported Projects: A ReviewPoland Country Assistance Review: Partnership in a Transition EconomyPoverty Reduction in the 1990s: An Evaluation of Strategy and PerformanceThe Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank’s Support Through 2003Power for Development: A Review of the World Bank Group’s Experience with Private Participation in the Electricity SectorPromoting Environmental Sustainability in DevelopmentPutting Social Development to Work for the Poor: An OED Review of World Bank ActivitiesReforming Agriculture: The World Bank Goes to MarketSharing Knowledge: Innovations and Remaining ChallengesSocial Funds: Assessing EffectivenessTunisia: Understanding Successful Socioeconomic Development Uganda: Policy, Participation, PeopleThe World Bank’s Experience with Post-Conflict ReconstructionThe World Bank’s Forest Strategy: Striking the Right BalanceZambia Country Assistance Review: Turning an Economy Around

Evaluation Country Case SeriesBosnia and Herzegovina: Post-Conflict ReconstructionBrazil: Forests in the Balance: Challenges of Conservation with DevelopmentCameroon: Forest Sector Development in a Difficult Political EconomyChina: From Afforestation to Poverty Alleviation and Natural Forest ManagementCosta Rica: Forest Strategy and the Evolution of Land UseEl Salvador: Post-Conflict ReconstructionIndia: Alleviating Poverty through Forest DevelopmentIndonesia: The Challenges of World Bank Involvement in ForestsThe Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF SupportUganda: Post-Conflict Reconstruction

ProceedingsGlobal Public Policies and Programs: Implications for Financing and EvaluationLessons of Fiscal AdjustmentLesson from Urban TransportEvaluating the Gender Impact of World Bank AssistanceEvaluation and Development: The Institutional Dimension (Transaction Publishers)Evaluation and Poverty ReductionMonitoring & Evaluation Capacity Development in AfricaPublic Sector Performance—The Critical Role of Evaluation

IEG PUBLICATIONS

All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual selection, please visithttp://www.worldbank.org/ieg

THE WORLD BANK

ISBN 0-8213-6845-1