sequential experimental designs for sensitivity experiments nist glm conference april 18-20, 2002...
TRANSCRIPT
Sequential Experimental Designs For Sensitivity Experiments
NIST GLM ConferenceApril 18-20, 2002
Joseph G. VoelkelCenter for Quality and Applied StatisticsCollege of EngineeringRochester Institute of Technology
2
CQAS
Sensitivity Experiments ASTM method D 1709–91 Impact resistance of plastic film by free-falling
dart method
Film
D art Pass
Fail
3
CQAS
Objectives
Engineer Specify a probability of
failure – 0.50, 0.10, … Find dart weight x= such
that Prob(F; )=
Statistician Find a strategy for
selecting weights {xi} so that is estimated as precisely as possible
0.1
X
p
Darts are dropped one at a time. Weight of ith dart may depend on results obtained up to date
4
CQAS
Data Collection PossibilitiesNon-sequential Specify n and all the {xi} before any Pass-Fail data {Yi} are
obtained. Find dose of drug at which 5% of mice develop tumors
Group-sequential Example: two-stage. Specify n1 and the {x1i}. Obtain data {Y1i} Use this info to specify n2 and the {x2i}. Obtain data {Y2i}
Same mice example, but with more time.
(Fully) Sequential Use all prior knowledge: x1Y1 x2Y2 x3Y3 x4Y4
Dart-weight example. One machine, one run at a time.
5
CQAS
Model and Objectives
Objective: Example Estimate weight at
which 10% of the samples fail
0.1
X
p
1 ( )
1 (1 exp( / ) )
x xP Y p F x
x
x
ln( /(1 ))
logit(p )x xx p p
logit( )
ln(0.10 / 0.90)
logit(0.10)
= 2.2
ˆvar( ) So, try to set the {xi} to
minimize
6
CQAS
Our Interest
(Fully) Sequential experiments Estimating a corresponding to a given , e.g. 0.10. The real problem. = 0.50? = 0.001?
7
CQAS
A Quick Tour of Some Past Work
Up-Down Method. Dixon and Mood (1948)
Only for =0.50 Robbins-Munro (1951)
wanted {xi} to converge to .
Like Up-Down, but with decreasing increments
far from 0.50 convergence is too slow
151050
7
6
5
Run
Set
ting
151050
7
6
5
Run
Set
ting
Pass
Fail
8
CQAS
A Quick Tour of Some Past Work
Wu’s (1985) Sequential-Solving Method Similar in spirit to the R-M procedure Collect some initial data to get estimates of and Choose the next setting, xn1 , to solve
( )F xn n 1 . So xn n 1
Better than R-M, much better than Up-and-Down Performance depends somewhat heavily on initial runs Asymptotically optimal, in a certain sense
9
CQAS
Some Non-Sequential Bayesian Results
Tsutakawa (1980) How to create design for estimation of for a given . Certain priors on and Some approximations Assumed constant number of runs made at equally
spaced settings. Chaloner and Larntz (1989)
Includes how to create design for estimation of for a given
Some reasonable approximations used Not restricted to constant number of runs or equally
spaced settings.
10
CQAS
Examples of Optimal Designs for estimating when =0.50.
(Based on Chaloner and Larntz)
~ , , .Unif 1 1 14a f ~ , , .Unif 10 10 14a f
100-10x
-10 0 10x
11
CQAS
Examples of Optimal Designs for estimating when =0.05.
(Based on Chaloner and Larntz)
~ , , .Unif 1 1 14a f ~ , , .Unif 10 10 14a f
100-10x
100-10x
12
CQAS
This Talk. Bayesian Sequential Design
A way to specify priors Measures of what we are learning about , ,
and —AII and Information Specifying the next setting, with some insights Some examples and comparisons Rethinking the priors
13
CQAS
Specifying Priors Consider the related tolerance-distribution problem The r.v. Xi represents the (unobservable) speed at which
the ith sample of film would have failed. Say from a location-scale family (e.g., logistic, normal, …)
1
1 (1 exp( / ) )
xP Y P X x
x
14
CQAS
Specifying Priors
Two-parameter distribution Could specify priors on
(, ) (, ) (, )
For simplicity, want to assume independence so only need to specify marginals of each parameter
, , not ,s X s X z s
0.50
(, )
15
CQAS
Specifying Priors
Instead of (,) … Consider =0.10
example Consider
distance
from to = 2.2
1050
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
x
c5/.7
logit(0.10)= 2.2
Easier for engineer to understand
(, )
16
CQAS
Specifying Priors
Ask engineer for Best guess and 95% range for
5.0 ± 3.0 Best guess and 95% range for – distance
6.6 / 2.0 Translate –=2.2 into terms: 3.0 / 2.0 Translate into normal, independent, priors on
and ln() We used a discrete set of 1515=225 values as
prior distribution of (,) (, )
17
CQAS
Specifying Priors
More natural for engineer to think about priors on and . We let engineer do this as follows.
We created 27 combinations of prior distributions: best guess—10 uncertainty (95% limits)— ± 2, 4, 6. best guess— 1, 3, 5 uncertainty (95% limits)— / 2, 4, 6.
We graphed these in terms of (,)
(, )
18
CQAS
Example of Prior Distributions of =10± 4 /
10 4
= 1 = 3 = 5 2
15105 100-10 100-10 100-10 4
15105 100-10 100-10 100-10 6
15105 100-10 100-10 100-10
19
CQAS
Finding the next setting xn+1 to runAssume we have collected data 1,..., nY Y based on
settings 1,..., nx x (This includes the case of not having any data, for which n=0.)
B a s e d o n t h i s , u p d a t e t h e o r i g i n a l p r i o r s o n a n d ( a n d h e n c e ) t o f o r m 1D i s t , | , . . . , D i s t ,n nY Y ,
o u r c u r r e n t p r i o r o n a n d . We would like to select a new setting 1nx so that if we run
there and get the result xY, we want Varn xY , the (new)
posterior variance of , to be as small as possible. However, we don’t know xY, so can’t find Varn xY . So find 1nx to minimize result on average, or EVarnnxY, where expectation is wrt xY. The optimal setting.
20
CQAS
AII Measure
Before we make the ( )n1th run, current prior variance of
, denoted by 1Var | ,..., =Varn nY Y , is determined. 1Varn (total) information about after n runs. E q u iva le n t w a y to fin d o p tim a l x is th e x th a t m a x im ize s
1 1
A IIV arE (V ar ) nn n x
xY
AII = anticipated increase in information. Plotting AIIxaf versus x gives a good indication for how sensitive our results are to the choice of the next x
21
CQAS
Simple Example
Objective: find the corresponding to =0.10
Prob
8 0.25
9 0.50
10 0.25
Prob
1 0.25
2 0.50
3 0.25
Pr8 1 5.8 .0625
8 2 3.6 .1250
8 3 1.4 .0625
9 1 6.8 .1250
9 2 4.6 .2500
9 3 2.4 .1250
10 1 7.8 .0625
10 2 5.6 .1250
10 3 3.4 .0625
0E 5.8 0.0625 ... 3.4 0.0625 4.6
22
CQAS
Simple Example
23
CQAS
Simple Example
Finding the AII for various x settings
20100
0.02
0.01
0.00
x
AII
24
CQAS
How AII “Thinks”
20100
0.02
0.01
0.00
x
AII
25
CQAS
First Simulation (,)=(8,1.82). Makes =4.0
403020100
6
5
4
3
2
Run
Set
ting,
E(
P assFailSettingX
Setting increment = 1
26
CQAS
Example with a More Diffuse Prior
=10 ± 4, =5 / 6
Simulation againdone with =8,=4
100-10
27
CQAS
Example with a More Diffuse Prior
6050403020100
7
6
5
4
3
2
1
0
Run
Set
ting,
E(
P assFailSettingX
403020100
6
5
4
3
2
Run
Set
ting,
E(
28
CQAS
Behavior of AII after 0, 2, 10, 20, 60 runs
20100
0.010
0.005
0.000
Setting
AII
0
2
10
20
60
29
CQAS
Information on , , = 2.2
6050403020100
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Run
Info
rmat
ion
on
A serious problem—all the information on was obtained through The simulation trusted the relative tight prior on … Another problem: more
objective methods of estimation, such as MLE, will likely not work wellAre there other ways to specify priors that might be better? Two methods…
30
CQAS
Equal-Contribution Priors For =2.2, restrict original prior so that
Var0()=Varo(2.2) Results of another simulation
0 10 20 30 40 50 60
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Run
Info
rmat
ion
on
6050403020100
0.14
0.12
0.10
0.08
0.06
0.04
0.02
0.00
Run
Info
rmat
ion
on
Problem: fails for case =: Var0()=Varo(0)?
31
CQAS
Relative Priors Consider the tolerance-distribution problem The r.v. Xi represents the (unobservable) speed at which
the ith sample of film would have failed. Say from a location-scale family (e.g., logistic, normal, …)
32
CQAS
Relative Priors
We observe only the ( x, Yx ) ’s If we could observe the X ’s, the problem would
be a simple one-sample problem of finding the 100 percentile of a distribution.
Assume the distribution of the X ’s has a finite fourth moment.
33
CQAS
Relative Priors
Using delta-method to find Var(s) and m-1m
22sd sd
4X m s
m
So, to a good approximation
1sd sds X k
2 2 4 22Var Var
1X m s
m m
After m runs, observing X1, X2, …, Xm, we have
34
CQAS
Relative Priors
So, with k1 and k2 known,
1 2ˆ ˆsd = sdk k
So, in this sense it is defensible to specify only the prior precision with which is know, and base the prior precision of upon it.
2ˆ ˆˆ ˆ= , , corr , =0 X k s
Now assume tolerance distribution is symmetric and its shape is know, e.g. logistic. Then
35
CQAS
Logistic Example
6050403020100
1.0
0.5
0.0
Run
Info
rmat
ion
on
36
CQAS
Summary
AII as a useful measure of value of making next run at x. Combination of shift in posterior mean & probability that a failure will occur at x
Informal comparison to non-Bayesian methodsBayesian x-strategy is more subtle
Danger of simply using any prior, and recommended way to set priors