Model Calibration and Sensitivity Analysis

A Framework for Model Applications 1

Calibration Criteria

1. Mean Error (ME)

( )MEn

cal obsi ii

n

= −=∑1

1

2. Mean Absolute Error (MAE)

MAEn

cal obsi ii

n

= −=∑1

1

3. Root Mean Squared (RMS)

( )RMSn

cal obsi ii

n

= −

=∑1 2

1

1 2

4. Correlation Coefficient (ϒ)

( )( )

( ) ( )γ =

− −

− −

=

= =

∑

∑ ∑

cal cal obs obs

cal cal obs obs

i ii

n

ii

n

ii

n1

2

1

2

1

A Framework for Model Applications 2

Presentation of Calibration Results

• qualitative comparison of calculated and observed contour maps

• tabulated results with summary statistics

• scatter diagrams • comparison of observed

and calculated breakthrough curves

• spatial distribution of residual errors

• comparison of observed and calculated mass distributions

A Framework for Model Applications 3

Observed concentrations superimposed on calculated contours

A Framework for Model Applications 4

A Framework for Model Applications 5

61 62 63 64 65

Observed Head (m-amsl)

61

62

63

64

65

Pre

dict

ed H

ead

(m-a

msl

)

A Framework for Model Applications 6

Comparison of calculated and observed discharge from 4 pumping wells

A Framework for Model Applications 7

Scatter Diagram

Comparison of breakthrough curves

A Framework for Model Applications 8

NATS Site, Columbus, MS

(Julian et al., 2001)

-30 -20 -10 0 10 20x (m)

-10

0

10

20

30

40

y (m

)

Source Trench

lnK (cm/s)

-9 -7 -5 -3 -1

-100 -50 0 50 100x (m)

-50

0

50

100

150

200

250

300

y (m

)

MLS Boundary

SourceTrench Groundwater

Flow

Abandoned Meander

K (cm/s)

Source Trench -5 -4 -3 -2 -110 10 10 10 10

-50 0 50 100 150 200 250 300y (m)

62

60

58

56

54

z (m

-am

sl)

A Framework for Model Applications 9

-10

0

10

20

30

40

50

y (m

)

-30 -20 -10 0 10 20x (m)

-10

0

10

20

30

40

50

y (m

)

-30 -20 -10 0 10 20x (m)

SourceTrench

MLS

(a)

(b)

Observed Predicted

Bromide(ppm)0 1 2 5 10 20

Observed and Calculated Bromide Concentrations

A Framework for Model Applications 10

Scatter diagram of observed versus calculated bromide peak concentrations. Snapshots 3 and 4 correspond to observation times of 152 and 278 days after source emplacement.

0

2

4

6

8

10

12

14

0 2 4 6 8 10 12 14

Observed Concentration (ppm)

Cal

cula

ted

Con

cent

ratio

n (p

pm)

Snapshot 3

Snapshot 4

A Framework for Model Applications 11

Methods for Model Calibration

1 by trial-and-error procedures a) select one or more calibration criteria b) adjust one input parameter at a time c) compare values of calibration criteria

2 by “automated” procedures a) parameter values b) parameter structures c) available codes

3 points to ponder a) perform transient calibration, if possible at all b) calibrate flow rates, if possible c) much can be gained from calibration against transport data

A Framework for Model Applications 12

Model Calibration as an Optimization Problem

( )Minimize objective function

subject to one or more specified constraints

S obs cali i i= −∑ω2

Example of simple objective function

A Framework for Model Applications 13

Example of more complex objective function with multiple local optima

A Framework for Model Applications 14

Illustrative Example

0 2000 4000 6000

X Axis (m)

0

2000

4000

6000

Y A

xis

(m)

No-flow boundary

No-flow

boundary

Specified-flow

boundary (Q=0.25 m

/day)

Constant head boundary (h=100 m)

Zone 1

Zone 2

Zone 3

Observation wellPumping well

P1

P2

P3

Figure 4-1. Configuration of the two-dimensional test problem. The solid dots and open circles indicate the locations of pumping and monitoring wells where hydraulic heads and/or concentrations are used to estimate hydraulic conductivity in the three zones.

A Framework for Model Applications 15

0 2000 4000 6000

X Axis (m)

0

2000

4000

6000

Y A

xis

(m)

Zone 1

Zone 2

Zone 3

0

20

40

60

80

100

0 200 400 600 800 1000

Time (days)

Con

cent

ratio

n (p

pm)

P1 P2 P3

True head distribution

True concentration data at 3 wells

A Framework for Model Applications 16

Objective Functions for Parameter Estimation

Head data only

( ) 2

1

ˆ NH

i i ii

Minimize S h hα=

= − ∑

Both head and concentration Data

( ) ( )2 2

1 1

ˆ ˆ NH NC

i i i j j ji j

Minimize S h h C Cα β= =

= − + − ∑ ∑

K1 (m/d) K2 K3 Obj. Func. # Generations

True 150 50 200

H only 150 50 200 0 28

H & C 150 50 200 0 20

A Framework for Model Applications 17

Sensitivity Analysis

• sensitivity coefficient

measure of the effect of change in one factor on another factor

X ya

yai k

i

k

i

k,

$ $= ≈∂∂

∆∆

normalized with respect to ak,

X ya a

ya ai k

i

k k

i

k k,

$ $= ≈∂

∂∆

∆

dimensionless,

X y ya a

y ya ai k

i i

k k

i i

k k,

$ $ $ $= ≈∂∂

∆∆

A Framework for Model Applications 18

Procedure for Sensitivity Analysis

• base case: calibrated model

• change a parameter by a certain percentage from the base case (∆a/a)

• run the model again

• calculate the change in model response (∆y) which could be any variable of interest, such as head, flow rate, concentration at a receptor, etc.

• calculate sensitivity coefficient and evaluate the results

A Framework for Model Applications 19

Examples of Sensitivity Analysis

A Framework for Model Applications 20

A Framework for Model Applications 21

A Framework for Model Applications 22

Observed plumes of BTEX and electron acceptors at the Hill Air Force Base (Lu et al., 1999)

Results of sensitivity analyses for total BTEX mass and BTEX plume front (Lu et al., 1999)

Degradation rate constants Sensitivity Coefficient K Recharge Long.

Disp. Retard. Factor

DO Nitrate Ferrous iron

Sulfate methane

Total BTEX mass

-0.202 -0.047 -0.036 -0.058 -0.031 -0.047 -0.084 -0.111 -0.054

Front of BTEX plumea

+0.563 -0.088 -0.419 -0.077 -0.331 -0.287 -0.188 -0.265 -0.003

a Sensitivity analysis performed on 0.05 mg/l contour line of BTEX plume

10.00

1.005.00

0.05

2.00 0.50

0.005

60.0 40.0

60.02.0

40.020.0

60.0

N

0.05

1.008.00100 m

300 ft

(a) BTEX

4.00

2.001.00

(b) Oxygen

5.001.00

0.05

(c) Nitrate (f) Methane

(e) Sulfate

(d) Ferrous iron

A Framework for Model Applications 23

Prediction and Uncertainty

• simulation of future conditions evaluation/assessment vs. prediction

• sources of uncertainty

⇒conceptual related to the mathematical model

⇒geological harder to quantify

⇒stress and other conventional parameters targets of most uncertainty analysis studies

A Framework for Model Applications 24

Methods for Uncertainty Analysis

• sensitivity analysis ⇒simple, straightforward, but cannot examine correlation between parameters

• first-order error analysis ⇒in the simplest form

[ ] [ ][ ]Var y Var x y xii

N

i==∑

1

2∂ ∂

⇒approximation

• Monte Carlo simulation ⇒most general, but computationally intensive

A Framework for Model Applications 25

A Framework for Model Applications 26

A Framework for Model Applications 27

Procedure for Monte Carlo Analysis

A Framework for Model Applications 28

Example of Monte Carlo Analysis Woldt et al. (1992)

A Framework for Model Applications 29

Only K as random variable

Both K and Initial Plume as random variables

Only Initial Plume as random variable