sequential experimental designs for sensitivity experiments nist glm conference april 18-20, 2002...

Sequential Experimental Designs For Sensitivity Experiments

NIST GLM ConferenceApril 18-20, 2002

Joseph G. VoelkelCenter for Quality and Applied StatisticsCollege of EngineeringRochester Institute of Technology

2

CQAS

Sensitivity Experiments ASTM method D 1709–91 Impact resistance of plastic film by free-falling

dart method

Film

D art Pass

Fail

3

CQAS

Objectives

Engineer Specify a probability of

failure – 0.50, 0.10, … Find dart weight x= such

that Prob(F; )=

Statistician Find a strategy for

selecting weights {xi} so that is estimated as precisely as possible

0.1

X

p

Darts are dropped one at a time. Weight of ith dart may depend on results obtained up to date

4

CQAS

Data Collection PossibilitiesNon-sequential Specify n and all the {xi} before any Pass-Fail data {Yi} are

obtained. Find dose of drug at which 5% of mice develop tumors

Group-sequential Example: two-stage. Specify n1 and the {x1i}. Obtain data {Y1i} Use this info to specify n2 and the {x2i}. Obtain data {Y2i}

Same mice example, but with more time.

(Fully) Sequential Use all prior knowledge: x1Y1 x2Y2 x3Y3 x4Y4

Dart-weight example. One machine, one run at a time.

5

CQAS

Model and Objectives

Objective: Example Estimate weight at

which 10% of the samples fail

0.1

X

p

1 ( )

1 (1 exp( / ) )

x xP Y p F x

x

x

ln( /(1 ))

logit(p )x xx p p

logit( )

ln(0.10 / 0.90)

logit(0.10)

= 2.2

ˆvar( ) So, try to set the {xi} to

minimize

6

CQAS

Our Interest

(Fully) Sequential experiments Estimating a corresponding to a given , e.g. 0.10. The real problem. = 0.50? = 0.001?

7

CQAS

A Quick Tour of Some Past Work

Up-Down Method. Dixon and Mood (1948)

Only for =0.50 Robbins-Munro (1951)

wanted {xi} to converge to .

Like Up-Down, but with decreasing increments

far from 0.50 convergence is too slow

151050

7

6

5

Run

Set

ting

151050

7

6

5

Run

Set

ting

Pass

Fail

8

CQAS

A Quick Tour of Some Past Work

Wu’s (1985) Sequential-Solving Method Similar in spirit to the R-M procedure Collect some initial data to get estimates of and Choose the next setting, xn1 , to solve

( )F xn n 1 . So xn n 1

Better than R-M, much better than Up-and-Down Performance depends somewhat heavily on initial runs Asymptotically optimal, in a certain sense

9

CQAS

Some Non-Sequential Bayesian Results

Tsutakawa (1980) How to create design for estimation of for a given . Certain priors on and Some approximations Assumed constant number of runs made at equally

spaced settings. Chaloner and Larntz (1989)

Includes how to create design for estimation of for a given

Some reasonable approximations used Not restricted to constant number of runs or equally

spaced settings.

10

CQAS

Examples of Optimal Designs for estimating when =0.50.

(Based on Chaloner and Larntz)

~ , , .Unif 1 1 14a f ~ , , .Unif 10 10 14a f

100-10x

-10 0 10x

11

CQAS

Examples of Optimal Designs for estimating when =0.05.

(Based on Chaloner and Larntz)

~ , , .Unif 1 1 14a f ~ , , .Unif 10 10 14a f

100-10x

100-10x

12

CQAS

This Talk. Bayesian Sequential Design

A way to specify priors Measures of what we are learning about , ,

and —AII and Information Specifying the next setting, with some insights Some examples and comparisons Rethinking the priors

13

CQAS

Specifying Priors Consider the related tolerance-distribution problem The r.v. Xi represents the (unobservable) speed at which

the ith sample of film would have failed. Say from a location-scale family (e.g., logistic, normal, …)

1

1 (1 exp( / ) )

xP Y P X x

x

14

CQAS

Specifying Priors

Two-parameter distribution Could specify priors on

(, ) (, ) (, )

For simplicity, want to assume independence so only need to specify marginals of each parameter

, , not ,s X s X z s

0.50

(, )

15

CQAS

Specifying Priors

Instead of (,) … Consider =0.10

example Consider

distance

from to = 2.2

1050

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

x

c5/.7

logit(0.10)= 2.2

Easier for engineer to understand

(, )

16

CQAS

Specifying Priors

Ask engineer for Best guess and 95% range for

5.0 ± 3.0 Best guess and 95% range for – distance

6.6 / 2.0 Translate –=2.2 into terms: 3.0 / 2.0 Translate into normal, independent, priors on

and ln() We used a discrete set of 1515=225 values as

prior distribution of (,) (, )

17

CQAS

Specifying Priors

More natural for engineer to think about priors on and . We let engineer do this as follows.

We created 27 combinations of prior distributions: best guess—10 uncertainty (95% limits)— ± 2, 4, 6. best guess— 1, 3, 5 uncertainty (95% limits)— / 2, 4, 6.

We graphed these in terms of (,)

(, )

18

CQAS

Example of Prior Distributions of =10± 4 /

10 4

= 1 = 3 = 5 2

15105 100-10 100-10 100-10 4

15105 100-10 100-10 100-10 6

15105 100-10 100-10 100-10

19

CQAS

Finding the next setting xn+1 to runAssume we have collected data 1,..., nY Y based on

settings 1,..., nx x (This includes the case of not having any data, for which n=0.)

B a s e d o n t h i s , u p d a t e t h e o r i g i n a l p r i o r s o n a n d ( a n d h e n c e ) t o f o r m 1D i s t , | , . . . , D i s t ,n nY Y ,

o u r c u r r e n t p r i o r o n a n d . We would like to select a new setting 1nx so that if we run

there and get the result xY, we want Varn xY , the (new)

posterior variance of , to be as small as possible. However, we don’t know xY, so can’t find Varn xY . So find 1nx to minimize result on average, or EVarnnxY, where expectation is wrt xY. The optimal setting.

20

CQAS

AII Measure

Before we make the ( )n1th run, current prior variance of

, denoted by 1Var | ,..., =Varn nY Y , is determined. 1Varn (total) information about after n runs. E q u iva le n t w a y to fin d o p tim a l x is th e x th a t m a x im ize s

1 1

A IIV arE (V ar ) nn n x

xY

AII = anticipated increase in information. Plotting AIIxaf versus x gives a good indication for how sensitive our results are to the choice of the next x

21

CQAS

Simple Example

Objective: find the corresponding to =0.10

Prob

8 0.25

9 0.50

10 0.25

Prob

1 0.25

2 0.50

3 0.25

Pr8 1 5.8 .0625

8 2 3.6 .1250

8 3 1.4 .0625

9 1 6.8 .1250

9 2 4.6 .2500

9 3 2.4 .1250

10 1 7.8 .0625

10 2 5.6 .1250

10 3 3.4 .0625

0E 5.8 0.0625 ... 3.4 0.0625 4.6

22

CQAS

Simple Example

23

CQAS

Simple Example

Finding the AII for various x settings

20100

0.02

0.01

0.00

x

AII

24

CQAS

How AII “Thinks”

20100

0.02

0.01

0.00

x

AII

25

CQAS

First Simulation (,)=(8,1.82). Makes =4.0

403020100

6

5

4

3

2

Run

Set

ting,

E(

P assFailSettingX

Setting increment = 1

26

CQAS

Example with a More Diffuse Prior

=10 ± 4, =5 / 6

Simulation againdone with =8,=4

100-10

27

CQAS

Example with a More Diffuse Prior

6050403020100

7

6

5

4

3

2

1

0

Run

Set

ting,

E(

P assFailSettingX

403020100

6

5

4

3

2

Run

Set

ting,

E(

28

CQAS

Behavior of AII after 0, 2, 10, 20, 60 runs

20100

0.010

0.005

0.000

Setting

AII

0

2

10

20

60

29

CQAS

Information on , , = 2.2

6050403020100

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

Run

Info

rmat

ion

on

A serious problem—all the information on was obtained through The simulation trusted the relative tight prior on … Another problem: more

objective methods of estimation, such as MLE, will likely not work wellAre there other ways to specify priors that might be better? Two methods…

30

CQAS

Equal-Contribution Priors For =2.2, restrict original prior so that

Var0()=Varo(2.2) Results of another simulation

0 10 20 30 40 50 60

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Run

Info

rmat

ion

on

6050403020100

0.14

0.12

0.10

0.08

0.06

0.04

0.02

0.00

Run

Info

rmat

ion

on

Problem: fails for case =: Var0()=Varo(0)?

31

CQAS

Relative Priors Consider the tolerance-distribution problem The r.v. Xi represents the (unobservable) speed at which

the ith sample of film would have failed. Say from a location-scale family (e.g., logistic, normal, …)

32

CQAS

Relative Priors

We observe only the ( x, Yx ) ’s If we could observe the X ’s, the problem would

be a simple one-sample problem of finding the 100 percentile of a distribution.

Assume the distribution of the X ’s has a finite fourth moment.

33

CQAS

Relative Priors

Using delta-method to find Var(s) and m-1m

22sd sd

4X m s

m

So, to a good approximation

1sd sds X k

2 2 4 22Var Var

1X m s

m m

After m runs, observing X1, X2, …, Xm, we have

34

CQAS

Relative Priors

So, with k1 and k2 known,

1 2ˆ ˆsd = sdk k

So, in this sense it is defensible to specify only the prior precision with which is know, and base the prior precision of upon it.

2ˆ ˆˆ ˆ= , , corr , =0 X k s

Now assume tolerance distribution is symmetric and its shape is know, e.g. logistic. Then

35

CQAS

Logistic Example

6050403020100

1.0

0.5

0.0

Run

Info

rmat

ion

on

36

CQAS

Summary

AII as a useful measure of value of making next run at x. Combination of shift in posterior mean & probability that a failure will occur at x

Informal comparison to non-Bayesian methodsBayesian x-strategy is more subtle

Danger of simply using any prior, and recommended way to set priors