Evaluating Economic Impacts of Agricultural Research: Examples and Lessons

George W. NortonAgricultural and Applied Economics

Seminar at the International Center for Tropical Agriculture (CIAT), Cali, Colombia, June 30, 2015

Introduction Growing demand for impact assessment of

agricultural research Improvements in assessment methods Agriculture faces dynamic environment

Population, income, climate, energy, pests Multiple goals and non-priced benefits Institutionalized system for research data

management useful for impact assessment

Objectives

Key impact evaluation issues Assessment examples Lessons

Key Impact Evaluation Issues

1. Counterfactual (what would have happened without the research)

2. Multiple objectives3. Aggregation 4. Integrating impact assessment with

research data management

D

S0

S1

Price

Quantity0

P0

P1

d

a

b

cI0

I1

Q0 Q1

R

Bt = P0Q0K(1+.5Ken/(e+n)) =

Where: (1) K = (a-c)/a reflects yield and cost changes, technology adoption, probability of success, and (2) e and n = supply and demand elasticities

1. Identifying what would have happened without the research

Estimating K is Key

Kt=((E(Y)/ε) - (E(C)/(1+E(Y) )At(1-d)t

Kt = Per unit cost reduction

E(Y) = proportionate yield increase per ha for adopters

ɛ = the price elasticity of supply

E(C) = the proportionate variable input cost change per hectare

A = proportion of the area affected by the technology

d = the technology depreciation rate

Approaches for estimating K

• For specific technologies, can obtain K from:• Expert opinions of scientists and others• Input and yield data from biological field

experiments in budgets combined with adoption data from surveys

• Farm-level survey data in regressions (e.g., using instrumental variables, propensity score matching, double difference)

• Randomized controlled trials (RCTs);villages and farmers are randomized with treated (receives technology) and untreated groups• Usefulness narrow for research evaluation

Retrospective (Ex Post) versus Prospective (Ex Ante)

Impact assessment methods can be similar, but data sources differ

Analysis often part ex ante, part ex post Probabilities and expectations are key in

ex ante impact analysis (Probability of research success) X

(Expected cost change per unit) X (Expected adoption ratet)

Example of estimating K (part ex ante, part ex post)

Myrick et al (2014): benefits of biocontrol program for papaya mealybug in Southern India Benefits of more than $500 million on an

investment of $500,000

Ex post: CIAT-VT (DIVA) evaluation of bean varieties in Rwanda and Uganda

Larochelle et al (2015) Yield impacts estimated econometrically (IV)

with plot-level data from 1440 households in Rwanda and 1908 H.H. in Uganda

Compared counterfactual and actual income distributions -- Poverty would have been 0.4 and 0.1 percent higher in Rwanda and Uganda in absence of the improved bean varieties.

Counterfactual for the Value of CIP Genebank

Study underway to assess, for varieties that used material (genetic resources) from CIP Genebank, what it would have cost to obtain the desired traits elsewhere without using the Genebank. Provides lower bound but credible

economic estimate of GB value

2. Managing Multiple objectives

Productivity/Income Poverty Environment Health/nutrition Risk/Resilience GenderTradeoffs among objectives; effects on

some easier to measure than others

Price

Quantity0

S0

S1

D

P0

P1

a

b

cdI 0

I1

Q0 Q1

a) Productivity or Income Impacts

Δ TS = +Δ CS = + Δ PS = -

Example: Ex post impacts of improved maize varieties in rural Ethiopia

Zeng et al (2015) Plot-level yield and cost changes due to adoption were estimated in an IV econometric model

Results were included in an economic surplus model to identify the counterfactual household income that would have existed without improved maize varieties.

Poverty differences assessed -- Improved maize varieties have led to a 0.8–1.3 percentage drop in poverty headcount ratio

b) Poverty Impacts Income gains can be estimated, adoption assessed, and

change in poverty rate calculated using a poverty index (such as Foster-Greer-Thorbecke) or by calculating income distributions with and without the intervention. Assessing changes in poverty indexes or distributions are complementary with RCTs, IVs, economic surplus analyses, and other impact assessment methods.

Example

Moyo et al (2007) calculated economic surplus changes from virus resistant groundnut varieties, disaggregating income and poverty rate changes from FGT poverty index to (a) adopters who were also groundnut consumers, (b) adopters who were not, and (c) consumers who were not groundnut producers (.5% to 1.5% poverty reduction)

c) Environmental or Sustainable Intensification Impacts

Many methods for assessing bio-physical (RCT, IV) and economic values (CV, Choice Experiment, Benefit Transfer) Must document research-induced

biophysical changes first Soil loss avoided, pesticide risk reduction,

carbon sequestered, etc. Then value non-market benefits of

technology or policy change

Examples

Using contingent valuation, Cuyno et al., (2001) estimated the value of environmental benefits from IPM-induced pesticide risk reduction on onions to be $150,000 per year in six villages in the Philippines.

Using a choice experiment, Vaiknoras et al., (2015) estimated that farmers would be willing to pay $10 per hectare in eastern Uganda for a one-half reduction in soil erosion per year.

d) Nutrition/Health Impacts

More nutritious food has complex impact pathways

For micro-nutrients, can use RCT or IV analysis to establish change in nutrient consumption due to the intervention and calculate disability-adjusted life years

For macro-nutrients, combine results from analysis of production and income changes with demand system to project consumption (and nutrient) changes

Example: Biofortified Cassava

Nguema et al (2011)

Tj = total number of people in target group jMj = mortality rate associated with the deficiency in target group jLj = average remaining life expectancy for target group jIij = incidence rate of disease i in target group jDij = disability weight for disease i in target group jdij = duration of disease i in target group j (for permanent diseases

dij equals the average remaining life expectancy Lj)r = discount rate for future life years

DALYs lost to Vitamin A deficiency in Nigeria and DALYs saved by bio-fortified cassava

e) Risk/Resilence Impacts Important due to climate change effects on poor Benefits from reduction in yield variance

Kostandini et al (2011):

B/Y0 = .5R (Y0) (σ2Y0 - σ2

Y1) where B is the money value of reduction in income variation, R is coefficient of relative risk aversion Y0 is the mean of the income distribution before the technology

Y1 is the mean after the new technology

σ2Y0 is CV squared for income distribution before the new technology and σ2Y1 is CV squared for the income distribution after the new technology.

Example

Kostandini et al., (2009) found the ex ante benefits of drought-tolerance research on cereals in eight African countries to total more than $1 billion per year with almost half of the benefits due to yield variance reduction

f) Gender Impacts

Few quantitative assessments of gender impacts of agricultural R&D Change in gender empowerment index Gender-disaggregated adoption analyses

3. Addressing Aggregation

Project, program, portfolio Field, farm, market Research Spillovers

Impact Matrix to Organize Data and Methods to Aggregate up

 

Level for which impact observed/assessed

 

Minimum Data used

 

Type of analysis/

model

Indicators Measured/Modeled

Outputs

Human Welfare Outcomes

EnvironmentIncome Poverty

Nutrition/health

International              

National              

Region/sub-sector/

ecosystem

             

Farm/Household/ Enterprise

             

Plot/Field/…              

4. Integrating impact assessment with research data management

In-house research impact assessment capacity is important

Key data for impact assessment are often lost over time Need an IT system for entering and

storing data on inputs, yields, and other traits from (1) near final trials, (2) adoption surveys Used for internal and external assessments

Example Reviewing research data management

system at CIP and possibilities for improving it for impact assessment Met with program leaders to discuss major

topics related to CIP strategic plan Identified candidates for assessment,

methods and data needs Reviewed current research data collection by

RIU and suggesting possible changes to make it more useful in the future

Undertaking impact case studies

Lessons Many research evaluation methods are

complementary in addressing multiple objectives

RCTs are unfortunately less useful for assessing agricultural research impacts than for other development interventions

Tradeoff between cost and credibility of impact assessment

Need plan for collecting and managing data from scientists to facilitate assessment