“Ex post evaluation of investment projects co-financed by the European Regional Development Fund (ERDF) and/or Cohesion Fund (CF)

in the period 1994-1999”

TEN PROJECTS OBSERVEDFinal Report

Evaluation Network Meeting Brussels, 23 November 2012

THE TEN PROJECTS

2

IRELAND:•M1 Motorway•Dublin wastewater treatment

SPAIN:•Sogama solid waste treatment in Galicia;•Wastewater treatment in Ría de Vigo•Madrid Metro Line 8•Mediterranean Corridor

GREECE:•Egnatia Motorway

PORTUGAL:•Valorsul solid waste treatment in Lisbon

ITALY:•Port of Gioia Tauro•Water supply in Palermo

INVESTMENT COSTS AND EC CO-FINANCING

3

PROJECT TOTAL INVESTMENT

COSTS

EC CO-FUNDING

EC CO-FUNDING RATE

Port of Gioia Tauro 473 M€ 61.9M€ 13%

Water supply in Palermo 120 M€ 44 M€ 37%

M1 Motorway 787 M€ 301 M€ 38.2%

Dublin waste water treatment 296 M€ 157 M€ 53.1%

Egnatia Motorway 7,053 M€ 3,091 M€ 43.8%

Urban solid waste treatment in Lisbon 366 M€ 137 M€ 37.38%

Waste water treatment in Ria de Vigo 172 M€ 118 M€ 68.6%

Madrid metro line 518 M€ 393 M€ 76%

Mediterranean Corridor 759 M€ 531 M€ 70%

Solid waste treatment in Galicia 275 M€ 100 M€ 36.3%

*Note: prices are expressed in constant terms (Euro 2011)

CONCEPTUAL FRAMEWORK

4

WHAT:“What kind of long term contributions can be

identified for different types of investment in the field of environment and transport

infrastructure?“

WHEN: “What is the minimum and average time needed for

a given long term contribution to materialise and stabilise? What are these time spans for different

types of investment in the field of environment and transport infrastructure? “

HOW:“How are these long term contributions generated

for different types of investment in the field of environment and transport infrastructure, i.e., what is the causal chain between certain short term socio-

economic returns and long term returns from investment?”

•Appropriateness to the context

•Project design

•Forecasting capacity

•Project governance

•Managerial response

Economic development

• Direct welfare and economic growth

• Endogenous dynamics

Quality of life

• Social cohesion

• Environmental effects

• Territorial cohesion

• Institutional learning

• Social happiness and wellbeing

Effects stabilised

• In the short run

• In the long run

Effects not yet stabilised

DEVELOPMENT EFFECTS AT A GLANCE

5

-5-4-3-2-1012345

Environmental projects

Median Min Max

-5-4-3-2-1012345

Transport projects

Median Min Max

LESSONS LEARNT ON THE EFFECTS (1)

6

Direct welfare and economic growth

Endogenous dynamics:• Less important than direct economic growth effects• More likely to be secured by transport projects

Social cohesion: • Side effects in 8 out of 10 projects• They can reach significant magnitude if appropriately exploited

Environmental sustainability:• As expected, environmental projects produce positive environmental

effects• Their full achievement is hampered by various operational difficulties

LESSONS LEARNT ON THE EFFECTS (2)

7

Territorial cohesion:• Side effects for environmental projects• Transport projects have no far-reaching and automatic effects

Institutional quality: • Low or medium effects for all the projects observed, particularly for

environment projects• Possible lack of critical mass: major projects are big but few

Social happiness:• They can be influenced by proactive measures • Social (dis)satisfaction can be ‘captured’ by organised vested interests

LESSONS LEARNT ON THE EFFECTS (3)

8

• The majority of effects appears to have already stabilised, either in the short or long run

• In some cases effects are not stabilised, but planned investments are expected to improve performance in the future

THE TEMPORAL DYNAMICS OF PROJECT EFFECTS

9

40%

27%

33%

Environment

Stabilised in the short term

Stabilised in the long term

Not stabilised yet

21%

40%

39%

Transport

Stabilised in the short term

Stabilised in the long term

Not stabilised yet

PERFORMANCE DRIVERS AT A GLANCE

10

-5-4-3-2-1012345

Approp. to the context

Project design

Forecast. capacity

Project govern.

Behav. response

Transport projects

Median Min Max

-5-4-3-2-1012345

Approp. to the context

Project design

Forecast. capacity

Project govern.

Behav. response

Environmental projects

Median Min Max

Appropriateness to the context: • All projects provided adapted solutions to population’s needs (except the

Port of Gioia Tauro)• Context influenced project design by imposing constraints in terms of time,

space and budget

Project design: • All transport projects were characterised by efficient and effective designs • Environmental projects (particularly 2 out of 5) were adversely affected by

budget and time constraint, forecasting mistakes and social pressure, and they lacked the necessary flexibility

Financial sustainability: • Financial sustainability is completely ensured only for the Palermo water

supply and Vigo waste water treatment projects• Future financial sustainability is uncertain for seven of the projects

assessed

LESSONS LEARNT ON THE PERFORMANCE DRIVERS (1)

11

Forecasting capacity:

12

LESSONS LEARNT ON THE PERFORMANCE DRIVERS (2)

PROJECT Cost overruns

Time overruns Overcapacity Undercapacity

Madrid Metro Line

Palermo Water Supply √

Valorsul solid waste treatment √

Ría de Vigo waste water treatment √

Sogama solid waste treatment √

Port of Gioia Tauro √ √

Egnatia motorway √ √

M1 motorway √ √

Dublin waste water treatment √ √

Mediterranean Corridor √ √ √

Project governance:• Governance is one fundamental factor accounting for project

performance• It should provide for a clear division of competence and adequate

mechanisms to integrate the views of stakeholders

Behavioural response:• Appropriate behavioural response can palliate deficiencies in terms of

design, governance or forecasting but it entails costs

Role of the EC:• The EC’s involvement in the project planning has been highly variable

across countries and type of project

LESSONS LEARNT ON THE PERFORMANCE DRIVERS (3)

13

• Drivers are combined in different ways, through project-specific patterns

• Project design and governance are two legs upon which a project stands: they should both adhere to a precise and specific vision of what the project wants to achieve

• Timing of effects is more determined by the project’s structural features than by specific drivers

THE INTERPLAY BETWEEN FACTORS

14

• Ex-post CBA provided the framework of analysis to disentangle the most crucial aspects of the project’s performance and final outcomes

• This exercise allowed the team to raise different methodological issues and to draw lessons learnt of general interest, concerning the following themes:

Project identification Time horizon Counterfactual scenario Demand analysis Social Discount Rates Quantification of costs and benefits Shadow prices

LESSONS LEARNT ON EX-POST CBA (1)

15

LESSONS LEARNT ON EX-POST CBA (2)

16

THEME LESSON

Project identification

• To be assessed on a case by case basis

• Criteria to be considered: 1) self-sufficiency and 2) pertinence and timing of the investments

Time horizon • Appropriate to its economically useful life and long enough to encompass its long term impact

• A time horizon longer that 30 years seems appropriate in case of long-lived mega projects (e.g. Egnatia motorway)

Counterfactual scenario

• The BAU option should be adopted as counterfactual

• In some cases, a feasible and realistic do-minimum option would be preferable

LESSONS LEARNT ON EX-POST CBA (3)

17

THEME LESSON

Demand analysis • More complex in transport projects, since they operate in more competitive markets

• Sensitivity and scenario analysis to test the forecasts

Social Discount Rates

• Adoption of two SDRs (backward and forward)

• Risk analysis to test the realism of the SDRs

Quantification of costs and benefits

• Aggregate costs and benefits to deal with data limitations

Shadow prices • Backward and forward and/or region-specific shadow prices could be adopted

The ingredients of success for public investment projects

1. The origin of project ideas: entrepreneurialism within government

2. Professionalism as a built-in insurance mechanism

3. Social ownership to cope with civil claims

4. Forecasting and monitoring as learning devices

5. Governments and projects

6. Incentives for success and identity

CONCLUSIONS

18

RECOMMENDATIONS

19

Provide solid technical design and forecasting Develop sound and reliable cost-benefit analyses Cost-benefit analysis as key input of the selection

Promote an independent quality review Make social acceptance part of the process

Design responsive governance arrangements

Ensure financial sustainability Design solid risk management

schemes

Develop ‘intrapreneurial’ teams in public administration

Establish a close relationship between project identification and planning

Endorse a role of policy advisor for the EC

Carry out ex-post evaluation Reward good projects and teams Promote knowledge transfer and sharing of best

practice Track project life: a project data warehouse

Undertake systematic monitoring Allow restructuring and redesign

Thank you

vignetti@csilmilano.com