causal inference for time-varying instructional treatments stephen w. raudenbush university of...
TRANSCRIPT
Causal Inference for Time-varying Instructional Treatments
Stephen W. RaudenbushUniversity of Chicago
Joint Work with Guanglei HongThe research reported here was supported by a grant from the Spencer Foundation entitled “Improving Research on Instruction: Models Designs, and Analytic Methods;” and a grant from the W.T. Grant Foundation entitled “Building Capacity for Evaluating Group-Level Interventions.” See Hong and Raudenbush, 2008, Journal of Educational and Behavioral Statistics
2
Outline1. Cumulative effects of sequences of instruction
* The changing social structure of instruction* Potential outcomes and causal effects
2. Statistical inference* Under sequential randomization* Given time-varying confounding
3. Estimating and applying IPTW
4. Illustrative results
3
1. Instructional Regimes
• Explicit
– Connor, Morrison, Fishman, Schatschneider, and Underwood, Science (2007)
– Borman, Slavin, Cheung, Chamberlain, Madden, and Chambers Educational Evaluation and Policy Analysis (2005)
– Bryk and Kerbow
• Implicit
4
2. Cumulative Effects of Sequences of Regimes
• Changing Social Structure of Instruction
5
Special Crossed and Nested StructureChild School 1 School 2 …
Teacher 1 Teacher 2 … Teacher 1 Teacher 2 …
1 X X
2 X X
.
.
.
N X x
6
Figure 3 Potential Outcomes in a 2-year Study of Binary Treatments, Z1 and Z2
z1 = 1
z1 = 0
Y1(1)
Y1(0)
z2 = 1
z2 = 0
z2 = 1
z2 = 0
Y2(1,1)
Y2(1,0)
Y2(0,1)
Y2(0,0)
7
Potential Outcomes
• Year-0 Outcome
• Year-1 Outcome
• Year-2 Outcome
00ijY
)(110 11 jjij zY
),(21210 211 jjjjij zzY
8
Causal Effects of Time-Varying Treatments
Grade-1 treatment on grade-1 outcome: E[Y1(1) – Y1(0)] = 1
Grade-1 treatment on grade-2 outcome: E [Y2(1,0) – Y2(0,0)] = 21
Grade-2 treatment on grade-2 outcome: E [Y2(0,1) – Y2(0,0)] = 22
“Amplifying Effect:” E [Y2(1,1) – Y2(0,0)] – (21 + 22)= *
*212222112212 )0,0(),(, zzzzYzzYor
9
Figure 4 Causal Effects of Z1, Z2 in a Randomized 2-year Study
U1 U2
Y1 Y2
Z1 Z2
Y0
U1, Y0 indep. of Z1; U1, U2, Y0, Y1 , Z1 indep. of Z2
10
3. Hierarchical Model for Observed Data• Observed Year-0 outcome
• Observed Year-1 outcome
growth yr1 treat yr 1 teacher
• Observed Year-2 Outcome
ikkjikkij cY 000 00
ikkjkjkkjikikkjij cczY 111101 10110
*21222211102 2121210
2 kjkjkkjkkjikikkjjij zzzzY
ikkjjjkjjkj ccc 2210100
growth yr1 treat in yr2 yr2 treat synergy
yr 2 teacher
11
Hierarchical Model (continued)
),0(~
,0
0~,,
0
0~
,,,
,
1110
0100
1
0
1110
0100
1
0
**22222121
11
1111
0000
cj
bb
bb
ik
ik
dd
dd
k
k
kkk
k
ikkik
ikkik
Nc
Nb
bN
d
d
bd
bd
t
13
Figure 4 Causal Effects of Z1, Z2 in a Randomized 2-year Study
U1 U2
Y1 Y2
Z1 Z2
Y0
U1, Y0 indep. of Z1; U1, U2, Y0, Y1 , Z1 indep. of Z2
14
Statistical Inference given time-varying confounding
15
Figure 5 Causal Effects of Z1, Z2 in a Non-Randomized 2-year Study Assuming Strongly Ignorable Treatment
AssignmentU1 U2
Y0 Y1 Y2
Z1 Z2
X1 X2
U1 indep. of Z1 | X1, Y0
U1, U2 indep. of Z2 | X1, X2, Y0, Y1, Z1
16
Strategies for Adjustment
• Covariance adjustment will fail• Propensity score stratification will fail• Inverse probability of treatment
weighting holds promise– Robins, Hernan, and Brumback Epidemiology
(2000)– Hong and Raudenbush, Journal of Educational
and Behavioral Statistics (2008)
17
4. Estimating and Applying IPTW
• Time 0
• Time 1
• Time 2
10 w
),|Pr(
)Pr(
0111
111 YXzZ
zZw
),|Pr(),,,,|Pr(
)Pr()|Pr(
011111102122
1111222 YXzZzZYYXXzZ
zZzZzZw
18
5. Illustrative Example
19
Longitudinal Evaluation of School Change and Performance (LESCP), 1997-1999
U.S. Department of EducationPlanning and Evaluation Service
Sample
- Longitudinal cohort of students: Grades 3 - 5- 4,216 students, 72.3% eligible for free lunch- 190 classrooms 3 years- 67 Title I schoolsOutcome- Stanford Achievement Test 9 Close-ended Math
20
Table 1 Sample Response Pattern
Observations Grade 3 Grade 4 Grade 5 Total
a. All the three years 953 953 953 2859
b. Grade 3 and 4 only 730 730 1460
c. Grade 3 and 5 only 127 127 254
d. Grade 4 and 5 only 363 363 726
e. Grade 3 only 1490 1490
f. Grade 4 only 435 435
g. Grade 5 only 118 118
Total 3300 2481 1561 7342
21
Table 2 Propensity Model Results
Grade 4 treatment Grade 5 treatmentPredictor SE () p SE () p
Average grade 3 content difficulty 1.215 .501 .015 -.654 .421 .120Average grade 3 math time -.249 .520 .632 .558 .466 .231% having grade 4 intensive math ---- ---- ---- 1.171 .656 .075Average math pretest score .032 .223 .885 .090 .238 .706Class size -.032 .049 .517 .137 .048 .004% low achievers receiving services -.379 .993 .702 -1.440 .968 .137Teacher’s educational degree .046 .619 .940 .118 .495 .811Teaching experience -.046 .031 .144 .007 .024 .761Teacher’s gender -1.566 .780 .045 .355 .580 .541African American teacher .941 .710 .185 -.092 .588 .876Teacher of other non-white ethnicity .761 1.123 .498 .356 .969 .713School size -.006 .003 .015 -.001 .002 .701% free-lunch students in school -3.392 2.100 .106 .029 .021 .168% black students in school .721 1.094 .510 -.569 .994 .567% Hispanic students in school 4.607 1.989 .021 -2.166 1.890 .252School-wide Title I program .043 .869 .960 .817 .896 .362% missing grade 3 instruction info -2.969 1.211 .014 -.695 .859 .419% missing grade 4 treatment info ---- ---- ---- -1.322 .739 .073Missing at least one other covariate -6.799 1.947 .000 3.528 1.792 .049
22
Table 3 Treatment Effect Estimation Results from Weighted Multi-level Growth Model
Fixed effects Coefficient SE tYear-1 status 610.178 2.004 304.456Growth rate 21.253 1.175 18.094Year-1 treatment on Year-1 outcome 3.089 2.255 1.209Year-2 treatment on Year-2 outcome 7.518 2.408 3.123Year-1 treatment on Year-2 outcome 0.042 3.834 0.011Two-way interaction of Year-1 andYear-2 treatments on Year-2 outcome
-0.280 4.577 -0.061
Variance components EstimateWithin students Variation 254.017Between students Var (year-1 status) 781.219
Var (growth rate) 51.597Corr (status, growth) -0.134
Between schools Var (year-1 status) 179.245Var (growth rate) 30.660Corr (status, growth) .419
Between classrooms Variation 168.970
23580
590
600
610
620
630
640
650
660
Year 0 Year 1 Year 2
Ma
th A
ch
ieve
me
nt
(0,0)
(1,0)
(0,1)
(1,1)
24
Table 4 Stability Analysis for Grade 5 Treatment Effect Estimation
Treatment Effect
Standard Error
t ratio
Model1
Naïve 12.70 4.15 3.06
Model2
Propensity as covariate
7.17 3.38 2.12
Model3
Propensity stratification
8.11 3.67 2.21
25
Table 5 Sensitivity Analysis for Grade 5 Treatment Effect Estimation
Hypothetical standardized mean difference in a covariate
Hypothetical standardized regression coefficient
Adjusted treatment effect
95% CI with standard error equal to 1.972
Minimum standard error for rejecting the null hypothesis
.1 .152 6.61 (2.00, 11.22) 3.37
.2 .152 6.62 (1.61, 10.83) 3.18
.3 .152 5.83 (1.22, 10.44) 2.98
.4 .152 5.45 (0.84, 10.06) 2.78
.5 .152 5.06 (0.45, 9.67) 2.58
.6 .152 4.67 (.06, 9.28) 2.38
.7 .152 4.28 (-.33, 8.89) 2.18
26
Table 5 Sensitivity Analysis for Grade 5 Treatment Effect Estimation (Continue)
Hypothetical standardized mean difference in a covariate
Hypothetical standardized regression coefficient
Adjusted treatment effect
95% CI with standard error equal to 1.972
Minimum standard error for rejecting the null hypothesis
.48 .10 5.77 (1.16, 10.38) 2.95
.48 .20 4.55 (-.06, 9.16) 2.32
.48 .30 3.32 (-1.29, 7.93) 1.69
.48 .40 2.09 (-2.52, 6.70) 1.07
.48 .50 0.87 (-3.74, 5.48) .44