sreb march 2010 5

New Evidence: Proving Online PD Increases Teacher

PerformanceBarbara TreacyDirector, EdTech Leaders Online (ETLO)Education Development Center (EDC)

Today’s Plan

Discuss e-Learning for Educators program EDC/EdTech Leaders Online

background Sample courses and model used in

research Research results Comments/Questions

Some questions

Does PD “work”? When is it effective? When isn’t it? How do we know? Does online PD “work”?

Effective PD

Should: focus on student learning be intensive, ongoing, connected to practice address teaching of specific curriculum

content align with school improvement priorities &

goals build strong working relationships among

teachers-Linda Darling-Hammond, NSDC (2009) http://www.nsdc.org/stateproflearning.cfm

http://www.nsdc.org/stateproflearning.cfm

National Ed Tech Plan Draft

“Episodic and ineffective professional development is replaced by professional learning that is collaborative, coherent, and continuous and that blends more effective in-person courses and workshops with the expanded opportunities, immediacy, and convenience enabled by online learning.”

-US Dept of Ed, March 5, 2011

Online PD“Some online tools have some

affordances that, if the training takes advantage of them, can help with some of the classic issues of professional development.” --Dr. Chris Dede, EdWeek, 10/1/2009

“The illiterate of the 21st

century will not be thosewho cannot read and

write,but those who cannot

learn,unlearn, and relearn.”

-Alvin Toffler

Goal: Build capacity to use online learning to meet local goals

For states, districts, universities, non-profits, others

Over 3000 online specialists trained in over 36 states

Focus on educator PD & Virtual Schools Online facilitator & course design programs 60+ online workshops Custom course development Learning Community Model Based at EDC

EdTech Leaders Online

E-Learning for Educators Goal: build state online teacher PD programs

focused on content, pedagogy, student achievement

10 state consortium: AL, DE, KY,MD, MO, MS, NC, NH, PA, WV

7 SREB states Funded in 2005 by USED Unique state/PTV partnership Alabama Public Television lead agency ETLO facilitator training, developer training,

workshops Content sharing across states

Impact

In first four years 480 workshop facilitators trained 1536 workshops delivered 21,628 participants completed workshops 278 course developers trained 110 workshops developed 91% teachers rated workshops

excellent/very good 96% facilitators rated training

excellent /very good

Research Study

Goal: examine the effect of online PD on teacher knowledge, practices, and student learning

Conducted by: Boston College Includes: 4 large-scale randomized experiments

using 3 workshops/grade designed by EDC for teachers in:– 4th grade English/Language Arts– 5th grade Math– 7th grade English/Language Arts– 8th grade Math

Workshop Topics 4th Grade ELA

Vocabulary, Writing, Reading Comprehension 5th Grade Math

Fractions, Algebraic Thinking, Measurement 7th Grade ELA

Vocabulary, Writing, Reading Comprehension

8th Grade Math Functions, Proportional Reasoning,

Geometric Measurement

Timeline for One Experiment

Students

Teachers

Background Survey

Student Instruments

Background Survey

Student Instruments

Student Instruments

Background Survey

Teacher Instruments

OPD Course 3

OPD Course 2

OPD Course 1

Teacher Instruments

Spring SemesterFall SemesterSpring Semester

Year 2Year 1

Online Teacher Instruments Include:Knowledge of Content TestInstructional Practices SurveyPedagogical Beliefs Survey

Online Student Instruments Include:Learning Outcomes TestInstructional Practices Survey

Summary of Research Findings for Teachers (ELA)

4th grade: Experimental group teachers had

significantly higher knowledge and practice scores than the control group teachers for vocabulary, reading comprehension, writing, and total ELA scores


significantly higher knowledge and practice scores than the control group teachers for vocabulary, writing and total ELA scores (reading comprehension was not significant)

Summary of Research Findings for Students (ELA)

4th grade: Experimental group students had

significantly higher knowledge and practice scores than the control group for vocabulary knowledge and writing practice scores, and for the total ELA knowledge scores


significantly higher practice scores than the control group for writing practice only

Summary of Research Findings for Teachers (Mathematics) 5th grade:

Experimental group teachers had significantly higher knowledge and practice scores than the control group teachers for fractions, measurement, algebraic thinking and total mathematics scores

8th grade: Experimental group teachers had significantly higher

knowledge scores than the control group teachers for proportional reasoning, geometric measurement and total mathematics scores

Experimental group teachers had significantly higher practice scores than the control group teachers for proportional reasoning, geometric measurement, functions and total mathematics scores

Summary of Research Findings for Students (Mathematics)

5th grade: No treatment effects were observed


significantly higher knowledge scores than the control group for geometric measurement and functions scores, and for the total mathematics scores

4th Grade Teacher Results- Knowledge (Sample)

InstrumentExperimental

Group (n = 36)

Control Group(n = 49)

ELAPre 37% 36%

Post 52% 38%

VocabularyPre 16% 14 %

Post 22% 14 %

Reading Comprehension

Pre 47% 46%

Post 65% 49%

WritingPre 49% 49%

Post 65% 47%

Online facilitator training

Graduate level facilitated online course (EDC)

Practicum follows training

Log in: http://moodle.etlo.org/course/view.php?id=430

Username: srebdemo Password: opd

http://moodle.etlo.org/course/view.php?id=430

http://moodle.etlo.org/course/view.php?id=430

“The interactivity and connectivity that this training course has provided has exceeded even my wildest expectations. It's our charge to provide this kind of community- building, knowledge-sharing experience for all KY teachers. I can't wait.”

-Kentucky facilitator training participant

Example Workshop: Functions

6 sessions / 30 hours Includes: online reading, activities,

videos, interactives, online discussions, project

Log in: http://edc.blackboard.com

Username: srebdemo Password: opd

http://edc.blackboard.com/

A Final Comment“My work as an elearning facilitator as well as an

elearningCourse developer has made me an ambassador of

onlineprofessional development. There is no better way to implement 21st century learning than to experience theprocess of an online course. If we want our students tosucceed in a global environment, then we, as educators,

mustbe engaged in 21st century content, context, tools, thinking skills, and assessment.

Thanks to online learning I am a 21st century life- longlearner!”

-West Virginia facilitator and course developer

Thank you!

Contact:Barbara [email protected]

EdTech Leaders Onlinehttp://edtechleaders.org

Education Development Centerhttp://edc.org

mailto:[email protected]

http://edtechleaders.org/

http://edc.org/

Research Data and Procedures

Analysis Procedures

For reliability Calculated internal consistency and test-

retest reliability For test scores

Calculated percent correct for experimental and control groups

For practice scores Calculated mean scores for experimental

and control groups

Analysis Procedures

For teacher outcomes: Conducted analysis of covariance

(ANCOVA) to test the significance of the difference between experimental and control group teachers’ posttest scores after controlling for pre-test scores

Calculated effect sizes (standardized differences between experimental and control group teachers’ posttest score)

Analysis Procedures

For student outcomes: Conducted analysis of covariance

(ANCOVA) to test the significance of the difference between experimental and control group students’ posttest scores after controlling for pre-test scores

Present effect sizes (standardized differences between experimental and control group students’ posttest score)

Analysis Procedures

For student outcomes: Formulated hierarchical linear models

(HLM) because Teachers were randomly assigned to

experimental and control groups Students were nested within teachers Statistical model provides more accurate

estimation of the impact of the treatment (i.e., these findings are more accurate than the ANCOVAs for the student outcomes)

4TH GRADE RESULTS

4th Grade Teacher Instruments

InstrumentNumber of Items

Pre-Test Reliability

Post-Test Reliability

Knowledge

Total ELA 21 .805 .833

Vocabulary 8 .820 .862

Reading Comprehensi

on7 .658 .722

Writing 6 .511 .497

Practice


Reading Comprehensi

on16 .879 .879

Writing 14 .911 .915

4th Grade Teacher Results- Knowledge


Group (n = 36)


ELAPre 37% 36%

Post 52% 38%

VocabularyPre 16% 14 %

Post 22% 14 %


Pre 47% 46%

Post 65% 49%

WritingPre 49% 49%

Post 65% 47%

4th Grade Teacher Results- Practices

Instrument(scale 1-4)

Experimental Group

(n = 36)

Control Group

(n = 49)

Vocabulary

Pre 2.81 2.76

Post 3.16 2.81

Reading Comprehensio

n

Pre 3.11 3.16

Post 3.36 3.21

Writing

Pre 2.89 2.76

Post 3.31 2.89

4th Grade Teacher Results- Significance

SubjectANCOVA

SignificanceEffect Size

Knowledge

Total ELAYes

(p<.01)Medium

(.53)

Vocabulary Yes

(p<.05)Small(.27)


Yes(p<.01)

Medium(.55)

WritingYes

(p<.01)Medium

(.75)

Practice

VocabularyYes

(p<.01)Small(.45)


Yes(p<.01)

Small(.32)

WritingYes

(p<.01)Small(.43)

4th Grade Student Instruments




Knowledge

Total ELA 29 .829 .852


Reading Comprehensi

on10 .659 .728

Writing 9 .59 .60

Practice


Reading Comprehensi

on10 .744 .760

Writing 10 .723 .746

4th Grade Student Results- Knowledge


Group (n = 831)


ELAPre 54% 53%

Post 63% 60%

VocabularyPre 64% 64%

Post 73% 69%


Pre 51% 50%

Post 58% 54%

WritingPre 47% 46%

Post 59% 56%

4th Grade Student Results- Practices


GroupControl Group

Vocabulary

Pre (scale 0-3)

1.84 n = 762

1.85n = 919

Post (scale 0-3)

1.89n = 750

1.89n = 902


Pre (scale 0-3)

2.14 n = 765

2.11n = 918

Post (scale 0-3)

2.20n = 757

2.16n = 919

Writing

Pre (scale 1-3)

2.36n = 757

2.33n = 913

Post(scale 1-3)

2.44n = 749

2.38n = 908

4th Grade Student Results- Significance

SubjectANCOVA

Significance

Effect SizeHLM

Significance

Knowledge

Total ELAYes

(p<.01)None(.10)

Yes

Vocabulary

Yes (p<.01)

None(.09)

Yes

Reading Comprehens

ion

Yes (p<.01)

None(.08)

No

WritingYes

(p<.01)None(.09)

No

Practice

Vocabulary No - No

Reading Comprehens

ion

Yes (p<.05)

None(.06)

No

WritingYes

(p<.05)None(.08)

Yes

5TH GRADE RESULTS





Knowledge

Total Mathematic

s31 .72 .80

Fractions 9 .54 .48

Algebraic Thinking

8 .55 .60

Measurement

14 .55 .67

Practice

Fractions 4 .61 .60

Algebraic Thinking

8 .82 .82

Measurement

26 .82 .88



Group (n = 34)


MathematicsPre 46% 45%

Post 58% 44%

FractionsPre 60% 57%

Post 63% 54%

Algebraic Thinking

Pre 47% 50%

Post 61% 49%

MeasurementPre 36% 34%

Post 53% 34%


Instrument(scale 1-4)

Experimental Group

(n = 34)

Control Group

(n = 45)

Fractions

Pre 2.74 2.87

Post 3.16 2.68

Algebraic Thinking

Pre 2.87 3.01

Post 3.42 2.98

Measurement

Pre 2.37 2.59

Post 2.91 2.74


SubjectANCOVA


Knowledge

Total Mathematics

Yes(p<.01)

Large(.71)

FractionsYes

(p<.05)Small(.29)

Algebraic Thinking

Yes(p<.01)

Medium(.43)

Measurement

Yes(p<.01)

Large(.93)

Practice

FractionsYes

(p<.01)Large(.88)

Algebraic Thinking

Yes(p<.01)

Large(.75)

Measurement

Yes(p<.01)

Large(.57)





Knowledge

Total Mathematic

s29 .73 .85

Fractions 8 .49 .68

Algebraic Thinking

10 .60 .73

Measurement

11 .48 .59



Group (n = 648)



Post 54% 53%

FractionsPre 30% 32%

Post 48% 46%

Algebraic Thinking

Pre 43% 45%

Post 54% 53%

MeasurementPre 50% 51%

Post 60% 59%


SubjectANCOVA

Significance

Effect SizeHLM

Significance

Knowledge

Total Mathematic

s

Yes (p<.01)

None(.10)

No

FractionsYes

(p<.05)None(.09)

No

Algebraic Thinking

Yes (p<.05)

None(.07)

No

Measurement

No - No

7TH GRADE RESULTS





Knowledge

Total ELA 22 .731 .728


Reading Comprehensi

on5 .475 .485

Writing 4 .120 .214

Practice


Reading Comprehensi

on13 .871 .886

Writing 17 .836 .833



Group (n = 36)


ELAPre 34% 30%

Post40%

30%


Post 31% 22%


Pre 37% 35%

Post 45% 36%

WritingPre 47% 41%

Post 54% 40%


Instrument (scale 1-4)

Experimental Group

(n = 35)

Control Group

(n = 45)

Vocabulary

Pre 2.9 3.1

Post 3.2 3.2

Reading Comprehensio

n

Pre 2.6 2.6

Post 2.8 2.7

Writing

Pre 3.0 3.1

Post 3.3 3.2


SubjectANCOVA


Knowledge

Total ELAYes

(p<.01)Medium

(.48)

VocabularyYes

(p<.05)Medium

(.34)


No -

WritingYes

(p<.01)Medium

(.47)

Practice

VocabularyYes

(p<.05)Small(.30)


No -

WritingYes

(p<.05)Small(.28)





Knowledge

Total ELA 24 .807 .833


Reading Comprehensi

on10 .695 .720

Writing 4 .262 .438

Practice

Reading Comprehensi

on12 .702 .773

Writing 21 .844 .863



Group (n = 831)


ELAPre 64% 66%

Post 67% 66%


Post 72% 71%


Pre 62% 63%

Post 62% 61%

WritingPre 58% 59%

Post 67% 66%

7th Grade Student Results- Practices

InstrumentExperimental Group

Control Group


Pre (scale 0-1)

.44 n = 789

.43n = 1154

Post (scale 0-1)

.50n = 797

.46n = 1162

Writing

Pre (scale 1-3)

2.25n = 830

2.22*n = 1224

Post(scale 1-3)

2.33n = 826

2.24n = 1225

* There was a significant difference between the pre-scores of the experimental and control groups.


SubjectANCOVA

Significance

Effect Size

HLM Significance

Knowledge

Total ELAYes

(p<.01)None(.08)

No

Vocabulary

Yes (p<.05)

None(.05)

No

Reading Comprehensi

on

Yes (p<.05)

None(.07)

No

Writing No - No

Practice

Reading Comprehensi

on

Yes (p<.01)

None(.11) Yes

WritingYes

(p<.01)None(.14)

No

8TH GRADE RESULTS





Knowledge

Total Mathematic

s46 .809 .847

Proportional Reasoning

29 .690 .772

Geometric Measureme

nt10 .430 .568

Functions 7 .597 .657

Practice


16 .843 .883

Geometric Measureme

nt12 .772 .757




Group (n = 28)



Post 42% 35%


Pre 35% 33%

Post 40% 34%


Pre 41% 36%

Post 47% 34%

FunctionsPre 47% 37%

Post 49% 40%


Instrument (scale 1-4)

Experimental Group

(n = 28)

Control Group

(n = 43)


Pre 2.7 2.7

Post 2.9 2.6


Pre 2.6 3.8

Post 2.9 3.8

Functions

Pre 2.5 4.0

Post 2.9 3.9


SubjectANCOVA


Knowledge

Total Mathematics

Yes(p<.01)

Small(.29)


Yes(p<.05)

Small(.28)


Yes(p<.01)

Medium(.43)

Functions No -

Practice


Yes(p<.01)

Large(.54)


Yes(p<.05)

Medium(.34)

FunctionsYes

(p<.01)Large(.51)





Knowledge

Total Mathematic

s36 .704 .758


10 .416 .440

Geometric Measureme

nt10 .493 .642




Group (n = 799)



Post 52% 48%


Pre 59% 56%

Post 55% 53%


Pre 43% 40%

Post 47% 43%

FunctionsPre 43% 40%

Post 52% 48%


SubjectANCOVA

Significance

Effect Size

HLM Significanc

e

Knowledge

Total Mathematics

Yes (p<.01)

None(.01)

Yes


No - No


Yes (p<.01)

None(.07)

Yes

FunctionsYes

(p<.01)None(.13)

Yes

Summary of Research Findings for Teachers (ELA)


significantly higher knowledge and practice scores than the control group teachers for vocabulary, reading comprehension, writing, and total ELA scores


significantly higher knowledge and practice scores than the control group teachers for vocabulary, writing and total ELA scores (reading comprehension was not significant)

Summary of Research Findings for Teachers (Mathematics)


knowledge and practice scores than the control group teachers for fractions, measurement, algebraic thinking and total mathematics scores


knowledge scores than the control group teachers for proportional reasoning, geometric measurement and total mathematics scores (function scores were not significant)

Experimental group teachers had significantly higher practice scores than the control group teachers for proportional reasoning, geometric measurement, functions and total mathematics scores

Summary of Research Findings for Students (ELA)

We focus on the results of the HLM analyses as they provide a more accurate representation of the treatment effect


significantly higher knowledge and practice scores than the control group for vocabulary knowledge and writing practice scores, and for the total ELA knowledge scores


significantly higher practice scores than the control group for writing practice only

Summary of Research Findings for Students (Mathematics)

5th grade: No treatment effects were observed


significantly higher knowledge scores than the control group for geometric measurement and functions scores, and for the total mathematics scores

Next steps…..

Analysis is ongoing