causal inference for time-varying instructional treatments stephen w. raudenbush university of...

Causal Inference for Time-varying Instructional Treatments

Stephen W. RaudenbushUniversity of Chicago

Joint Work with Guanglei HongThe research reported here was supported by a grant from the Spencer Foundation entitled “Improving Research on Instruction: Models Designs, and Analytic Methods;” and a grant from the W.T. Grant Foundation entitled “Building Capacity for Evaluating Group-Level Interventions.” See Hong and Raudenbush, 2008, Journal of Educational and Behavioral Statistics

2

Outline1. Cumulative effects of sequences of instruction

* The changing social structure of instruction* Potential outcomes and causal effects

2. Statistical inference* Under sequential randomization* Given time-varying confounding

3. Estimating and applying IPTW

4. Illustrative results

3

1. Instructional Regimes

• Explicit

– Connor, Morrison, Fishman, Schatschneider, and Underwood, Science (2007)

– Borman, Slavin, Cheung, Chamberlain, Madden, and Chambers Educational Evaluation and Policy Analysis (2005)

– Bryk and Kerbow

• Implicit

4

2. Cumulative Effects of Sequences of Regimes

• Changing Social Structure of Instruction

5

Special Crossed and Nested StructureChild School 1 School 2 …

Teacher 1 Teacher 2 … Teacher 1 Teacher 2 …

1 X X

2 X X

.

.

.

N X x

6

Figure 3 Potential Outcomes in a 2-year Study of Binary Treatments, Z1 and Z2

z1 = 1

z1 = 0

Y1(1)

Y1(0)

z2 = 1

z2 = 0

z2 = 1

z2 = 0

Y2(1,1)

Y2(1,0)

Y2(0,1)

Y2(0,0)

7

Potential Outcomes

• Year-0 Outcome

• Year-1 Outcome

• Year-2 Outcome

00ijY

)(110 11 jjij zY

),(21210 211 jjjjij zzY

8

Causal Effects of Time-Varying Treatments

Grade-1 treatment on grade-1 outcome: E[Y1(1) – Y1(0)] = 1

Grade-1 treatment on grade-2 outcome: E [Y2(1,0) – Y2(0,0)] = 21

Grade-2 treatment on grade-2 outcome: E [Y2(0,1) – Y2(0,0)] = 22

“Amplifying Effect:” E [Y2(1,1) – Y2(0,0)] – (21 + 22)= *

*212222112212 )0,0(),(, zzzzYzzYor

9

Figure 4 Causal Effects of Z1, Z2 in a Randomized 2-year Study

U1 U2

Y1 Y2

Z1 Z2

Y0

U1, Y0 indep. of Z1; U1, U2, Y0, Y1 , Z1 indep. of Z2

10

3. Hierarchical Model for Observed Data• Observed Year-0 outcome

• Observed Year-1 outcome

growth yr1 treat yr 1 teacher

• Observed Year-2 Outcome

ikkjikkij cY 000 00

ikkjkjkkjikikkjij cczY 111101 10110

*21222211102 2121210

2 kjkjkkjkkjikikkjjij zzzzY

ikkjjjkjjkj ccc 2210100

growth yr1 treat in yr2 yr2 treat synergy

yr 2 teacher

11

Hierarchical Model (continued)

),0(~

,0

0~,,

0

0~

,,,

,

1110

0100

1

0

1110

0100

1

0

**22222121

11

1111

0000

cj

bb

bb

ik

ik

dd

dd

k

k

kkk

k

ikkik

ikkik

Nc

Nb

bN

d

d

bd

bd

t

13

Figure 4 Causal Effects of Z1, Z2 in a Randomized 2-year Study

U1 U2

Y1 Y2

Z1 Z2

Y0

U1, Y0 indep. of Z1; U1, U2, Y0, Y1 , Z1 indep. of Z2

14

Statistical Inference given time-varying confounding

15

Figure 5 Causal Effects of Z1, Z2 in a Non-Randomized 2-year Study Assuming Strongly Ignorable Treatment

AssignmentU1 U2

Y0 Y1 Y2

Z1 Z2

X1 X2

U1 indep. of Z1 | X1, Y0

U1, U2 indep. of Z2 | X1, X2, Y0, Y1, Z1

16

Strategies for Adjustment

• Covariance adjustment will fail• Propensity score stratification will fail• Inverse probability of treatment

weighting holds promise– Robins, Hernan, and Brumback Epidemiology

(2000)– Hong and Raudenbush, Journal of Educational

and Behavioral Statistics (2008)

17

4. Estimating and Applying IPTW

• Time 0

• Time 1

• Time 2

10 w

),|Pr(

)Pr(

0111

111 YXzZ

zZw

),|Pr(),,,,|Pr(

)Pr()|Pr(

011111102122

1111222 YXzZzZYYXXzZ

zZzZzZw

18

5. Illustrative Example

19

Longitudinal Evaluation of School Change and Performance (LESCP), 1997-1999

U.S. Department of EducationPlanning and Evaluation Service

Sample

- Longitudinal cohort of students: Grades 3 - 5- 4,216 students, 72.3% eligible for free lunch- 190 classrooms 3 years- 67 Title I schoolsOutcome- Stanford Achievement Test 9 Close-ended Math

20

Table 1 Sample Response Pattern

Observations Grade 3 Grade 4 Grade 5 Total

a. All the three years 953 953 953 2859

b. Grade 3 and 4 only 730 730 1460

c. Grade 3 and 5 only 127 127 254

d. Grade 4 and 5 only 363 363 726

e. Grade 3 only 1490 1490

f. Grade 4 only 435 435

g. Grade 5 only 118 118

Total 3300 2481 1561 7342

21

Table 2 Propensity Model Results

Grade 4 treatment Grade 5 treatmentPredictor SE () p SE () p

Average grade 3 content difficulty 1.215 .501 .015 -.654 .421 .120Average grade 3 math time -.249 .520 .632 .558 .466 .231% having grade 4 intensive math ---- ---- ---- 1.171 .656 .075Average math pretest score .032 .223 .885 .090 .238 .706Class size -.032 .049 .517 .137 .048 .004% low achievers receiving services -.379 .993 .702 -1.440 .968 .137Teacher’s educational degree .046 .619 .940 .118 .495 .811Teaching experience -.046 .031 .144 .007 .024 .761Teacher’s gender -1.566 .780 .045 .355 .580 .541African American teacher .941 .710 .185 -.092 .588 .876Teacher of other non-white ethnicity .761 1.123 .498 .356 .969 .713School size -.006 .003 .015 -.001 .002 .701% free-lunch students in school -3.392 2.100 .106 .029 .021 .168% black students in school .721 1.094 .510 -.569 .994 .567% Hispanic students in school 4.607 1.989 .021 -2.166 1.890 .252School-wide Title I program .043 .869 .960 .817 .896 .362% missing grade 3 instruction info -2.969 1.211 .014 -.695 .859 .419% missing grade 4 treatment info ---- ---- ---- -1.322 .739 .073Missing at least one other covariate -6.799 1.947 .000 3.528 1.792 .049

22

Table 3 Treatment Effect Estimation Results from Weighted Multi-level Growth Model

Fixed effects Coefficient SE tYear-1 status 610.178 2.004 304.456Growth rate 21.253 1.175 18.094Year-1 treatment on Year-1 outcome 3.089 2.255 1.209Year-2 treatment on Year-2 outcome 7.518 2.408 3.123Year-1 treatment on Year-2 outcome 0.042 3.834 0.011Two-way interaction of Year-1 andYear-2 treatments on Year-2 outcome

-0.280 4.577 -0.061

Variance components EstimateWithin students Variation 254.017Between students Var (year-1 status) 781.219

Var (growth rate) 51.597Corr (status, growth) -0.134

Between schools Var (year-1 status) 179.245Var (growth rate) 30.660Corr (status, growth) .419

Between classrooms Variation 168.970

23580

590

600

610

620

630

640

650

660

Year 0 Year 1 Year 2

Ma

th A

ch

ieve

me

nt

(0,0)

(1,0)

(0,1)

(1,1)

24

Table 4 Stability Analysis for Grade 5 Treatment Effect Estimation

Treatment Effect

Standard Error

t ratio

Model1

Naïve 12.70 4.15 3.06

Model2

Propensity as covariate

7.17 3.38 2.12

Model3

Propensity stratification

8.11 3.67 2.21

25

Table 5 Sensitivity Analysis for Grade 5 Treatment Effect Estimation

Hypothetical standardized mean difference in a covariate

Hypothetical standardized regression coefficient

Adjusted treatment effect

95% CI with standard error equal to 1.972

Minimum standard error for rejecting the null hypothesis

.1 .152 6.61 (2.00, 11.22) 3.37

.2 .152 6.62 (1.61, 10.83) 3.18

.3 .152 5.83 (1.22, 10.44) 2.98

.4 .152 5.45 (0.84, 10.06) 2.78

.5 .152 5.06 (0.45, 9.67) 2.58

.6 .152 4.67 (.06, 9.28) 2.38

.7 .152 4.28 (-.33, 8.89) 2.18

26

Table 5 Sensitivity Analysis for Grade 5 Treatment Effect Estimation (Continue)

Hypothetical standardized mean difference in a covariate

Hypothetical standardized regression coefficient

Adjusted treatment effect

95% CI with standard error equal to 1.972

Minimum standard error for rejecting the null hypothesis

.48 .10 5.77 (1.16, 10.38) 2.95

.48 .20 4.55 (-.06, 9.16) 2.32

.48 .30 3.32 (-1.29, 7.93) 1.69

.48 .40 2.09 (-2.52, 6.70) 1.07

.48 .50 0.87 (-3.74, 5.48) .44