psylab advanced course: introduction to simulation
TRANSCRIPT
PsyLab Advanced Course: Introduction tosimulation
Karl Bang ChristensenMike Horton
Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
Overview
Two examples
Item fit statistics
INFIT and OUTFIT are often reported - but evaluated using ’rulesof thumb’ (e.g. values larger than 1.3 indicating misfit) - this hasbeen critisized
Can we trust, e.g., the item χ2 in RUMM ?
Impact of LD
does local dependence impact the planned analysis
Solution
Simulation
Use simulated data sets to learn about the critical values (what toexpect if Rasch model fits)
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
Item fit statistics
Let Xiv denote the response to item i by subject v with the valueθv on the latent trait. Standard evaluation of individual item fituses
Ziv =Xiv − E (Xiv |θv = θ̂v )√
V (Xiv |θv = θ̂v )
to compute fit statistics like
OUTFITi =1
N
N∑v=1
Z 2iv
with no established null distribution.
History lesson
early Rasch literature incorrectly claims χ2 distribution1
1Wright, Stone (1979). Mesa Press, Chicago, USA.Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
Item fit statistics
Re-writingN∑
v=1
Z 2iv =
k∑r=0
∑v :Rv=r
Z 2i |r︸ ︷︷ ︸
iid
’known’ null distribution2
History lesson
This fact has been forgotten/abandoned3
2Christensen, Kreiner (2013). Rasch Models in Health (chap. 5, pp.83-104), Wiley https://doi.org/10.1002/9781118574454.ch5
3Wright, Panchapakesan (1969). Educational and PsychologicalMeasurement, 29, 23-48. http://doi.org/10.1177/001316446902900102
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
Item fit statistics
It has been suggested that the Wilson-Hilferty cube-roottransformation
ti = (OUTFIT1/3i − 1)
3
V (OUTFITi )+
V (OUTFITi )
3
has an approximate t distribution
History lesson
It seems unlikely that anyone has looked at
Wilson, Hilferty (1931). The Distribution of Chi-Square. InProceedings of the National Academy of Sciences of theUnited States of America (Vol. 17, pp. 684-688).https://doi.org/10.1073/pnas.17.12.684
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
Item fit statistics
It has been suggested that
FitResidi =f (log(N · OUTFITi )− log(f ))√
V (N · OUTFITi )
with f = (Nk − N − k + 1)/k , has a symmetrical distribution withmean zero and variance one.
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
Item fit statistics
RUMM
groups respondents in G ’class intervals’ based on the estimatedperson locations θ̂v . ANOVA item fit statistic or item χ2 fitstatistic
χ2(Xi ) =∑g
(∑v∈Vg
Xvi −∑
v∈VgE (Xvi )
)2∑v∈Vg
V (Xvi )
where Vg denotes the set of respondents in class interval g .
problem
still relies on θ̂1, θ̂2, . . .
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
AMTS data example - individual item fit
item chisq df p
ADDRESS 1.73 1 0.1889
AGE 2.74 1 0.0976
COUNTBAC 0.98 1 0.3226
DOB 0.60 1 0.4382
FIRSTWW 2.25 1 0.1337
MONARCH 3.39 1 0.0656
MONTH 12.99 1 0.0003
NAME 2.74 1 0.0976
TIME 2.21 1 0.1369
YEAR 6.25 1 0.0124
can we trust the p-values ? How can we address multiple testing ?
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
AMTS data example - evaluating item fit
The null distribution (what would we expect if the data fit themodel)
11:12 Wednesday, September 19, 2018 111:12 Wednesday, September 19, 2018 1
0 5 10
Item fit statistic
0
10
20
30
40
50
Pe
rce
nt
Theoretical distribution
unlikely that χ2MONTH is above 3.84
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
AMTS data example - simulation
The steps
1 Simulate data set based on estimated item parameters andestimated person locations
2 This yields a data set where we know the Rasch model fits
3 Estimate item parameters and estimated person locations
4 Compute χ2MONTH
are repeated, say, 1000 times
Summarizing
Now we know what to expect if the Rasch model fits - and we cananswer the question: ’how often would you see large values just bychance ?’
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
21:31 Monday, January 22, 2018 121:31 Monday, January 22, 2018 1
0 5 10
Item fit statistic
0
10
20
30
40
50
Pe
rce
nt
empiricaltheoretical
Empirical and theoretical distribution
0 5 10
Item fit statistic
0
10
20
30
40
50
Pe
rce
nt
4.5095% percentile8.3%rejection rate
empiricaltheoretical
Empirical and theoretical distribution
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
Perspectives in using simulation
Appropriate critical value for item χ2
Appropriate critical value for maxi=1,..,10(χ2i )
Tabulate critical values
Implement parametric bootstrap
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation
Appropriate critical value for maxi=1,..,10(χ2i )
11:12 Wednesday, September 19, 2018 111:12 Wednesday, September 19, 2018 1
95th: 10.52 99th: 14.61
0 5 10 15 20
Empirical distribution for maximum value of the chisq item fit statistic
0
5
10
15
20
Pe
rce
nt
Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation