sreb march 2010 5
Post on 26-Jan-2015
375 Views
Preview:
DESCRIPTION
TRANSCRIPT
New Evidence: Proving Online PD Increases Teacher
PerformanceBarbara TreacyDirector, EdTech Leaders Online (ETLO)Education Development Center (EDC)
Today’s Plan
Discuss e-Learning for Educators program EDC/EdTech Leaders Online
background Sample courses and model used in
research Research results Comments/Questions
Some questions
Does PD “work”? When is it effective? When isn’t it? How do we know? Does online PD “work”?
Effective PD
Should: focus on student learning be intensive, ongoing, connected to practice address teaching of specific curriculum
content align with school improvement priorities &
goals build strong working relationships among
teachers-Linda Darling-Hammond, NSDC (2009) http://www.nsdc.org/stateproflearning.cfm
National Ed Tech Plan Draft
“Episodic and ineffective professional development is replaced by professional learning that is collaborative, coherent, and continuous and that blends more effective in-person courses and workshops with the expanded opportunities, immediacy, and convenience enabled by online learning.”
-US Dept of Ed, March 5, 2011
Online PD“Some online tools have some
affordances that, if the training takes advantage of them, can help with some of the classic issues of professional development.” --Dr. Chris Dede, EdWeek, 10/1/2009
“The illiterate of the 21st
century will not be thosewho cannot read and
write,but those who cannot
learn,unlearn, and relearn.”
-Alvin Toffler
Goal: Build capacity to use online learning to meet local goals
For states, districts, universities, non-profits, others
Over 3000 online specialists trained in over 36 states
Focus on educator PD & Virtual Schools Online facilitator & course design programs 60+ online workshops Custom course development Learning Community Model Based at EDC
EdTech Leaders Online
E-Learning for Educators Goal: build state online teacher PD programs
focused on content, pedagogy, student achievement
10 state consortium: AL, DE, KY,MD, MO, MS, NC, NH, PA, WV
7 SREB states Funded in 2005 by USED Unique state/PTV partnership Alabama Public Television lead agency ETLO facilitator training, developer training,
workshops Content sharing across states
Impact
In first four years 480 workshop facilitators trained 1536 workshops delivered 21,628 participants completed workshops 278 course developers trained 110 workshops developed 91% teachers rated workshops
excellent/very good 96% facilitators rated training
excellent /very good
Research Study
Goal: examine the effect of online PD on teacher knowledge, practices, and student learning
Conducted by: Boston College Includes: 4 large-scale randomized experiments
using 3 workshops/grade designed by EDC for teachers in:– 4th grade English/Language Arts– 5th grade Math– 7th grade English/Language Arts– 8th grade Math
Workshop Topics 4th Grade ELA
Vocabulary, Writing, Reading Comprehension 5th Grade Math
Fractions, Algebraic Thinking, Measurement 7th Grade ELA
Vocabulary, Writing, Reading Comprehension
8th Grade Math Functions, Proportional Reasoning,
Geometric Measurement
Timeline for One Experiment
Students
Teachers
Background Survey
Student Instruments
Background Survey
Student Instruments
Student Instruments
Background Survey
Teacher Instruments
OPD Course 3
OPD Course 2
OPD Course 1
Teacher Instruments
Spring SemesterFall SemesterSpring Semester
Year 2Year 1
Online Teacher Instruments Include:Knowledge of Content TestInstructional Practices SurveyPedagogical Beliefs Survey
Online Student Instruments Include:Learning Outcomes TestInstructional Practices Survey
Summary of Research Findings for Teachers (ELA)
4th grade: Experimental group teachers had
significantly higher knowledge and practice scores than the control group teachers for vocabulary, reading comprehension, writing, and total ELA scores
7th grade: Experimental group teachers had
significantly higher knowledge and practice scores than the control group teachers for vocabulary, writing and total ELA scores (reading comprehension was not significant)
Summary of Research Findings for Students (ELA)
4th grade: Experimental group students had
significantly higher knowledge and practice scores than the control group for vocabulary knowledge and writing practice scores, and for the total ELA knowledge scores
7th grade: Experimental group students had
significantly higher practice scores than the control group for writing practice only
Summary of Research Findings for Teachers (Mathematics) 5th grade:
Experimental group teachers had significantly higher knowledge and practice scores than the control group teachers for fractions, measurement, algebraic thinking and total mathematics scores
8th grade: Experimental group teachers had significantly higher
knowledge scores than the control group teachers for proportional reasoning, geometric measurement and total mathematics scores
Experimental group teachers had significantly higher practice scores than the control group teachers for proportional reasoning, geometric measurement, functions and total mathematics scores
Summary of Research Findings for Students (Mathematics)
5th grade: No treatment effects were observed
8th grade: Experimental group students had
significantly higher knowledge scores than the control group for geometric measurement and functions scores, and for the total mathematics scores
4th Grade Teacher Results- Knowledge (Sample)
InstrumentExperimental
Group (n = 36)
Control Group(n = 49)
ELAPre 37% 36%
Post 52% 38%
VocabularyPre 16% 14 %
Post 22% 14 %
Reading Comprehension
Pre 47% 46%
Post 65% 49%
WritingPre 49% 49%
Post 65% 47%
Online facilitator training
Graduate level facilitated online course (EDC)
Practicum follows training
Log in: http://moodle.etlo.org/course/view.php?id=430
Username: srebdemo Password: opd
“The interactivity and connectivity that this training course has provided has exceeded even my wildest expectations. It's our charge to provide this kind of community- building, knowledge-sharing experience for all KY teachers. I can't wait.”
-Kentucky facilitator training participant
Example Workshop: Functions
6 sessions / 30 hours Includes: online reading, activities,
videos, interactives, online discussions, project
Log in: http://edc.blackboard.com
Username: srebdemo Password: opd
A Final Comment“My work as an elearning facilitator as well as an
elearningCourse developer has made me an ambassador of
onlineprofessional development. There is no better way to implement 21st century learning than to experience theprocess of an online course. If we want our students tosucceed in a global environment, then we, as educators,
mustbe engaged in 21st century content, context, tools, thinking skills, and assessment.
Thanks to online learning I am a 21st century life- longlearner!”
-West Virginia facilitator and course developer
Thank you!
Contact:Barbara TreacyBtreacy@edc.org
EdTech Leaders Onlinehttp://edtechleaders.org
Education Development Centerhttp://edc.org
Research Data and Procedures
Analysis Procedures
For reliability Calculated internal consistency and test-
retest reliability For test scores
Calculated percent correct for experimental and control groups
For practice scores Calculated mean scores for experimental
and control groups
Analysis Procedures
For teacher outcomes: Conducted analysis of covariance
(ANCOVA) to test the significance of the difference between experimental and control group teachers’ posttest scores after controlling for pre-test scores
Calculated effect sizes (standardized differences between experimental and control group teachers’ posttest score)
Analysis Procedures
For student outcomes: Conducted analysis of covariance
(ANCOVA) to test the significance of the difference between experimental and control group students’ posttest scores after controlling for pre-test scores
Present effect sizes (standardized differences between experimental and control group students’ posttest score)
Analysis Procedures
For student outcomes: Formulated hierarchical linear models
(HLM) because Teachers were randomly assigned to
experimental and control groups Students were nested within teachers Statistical model provides more accurate
estimation of the impact of the treatment (i.e., these findings are more accurate than the ANCOVAs for the student outcomes)
4TH GRADE RESULTS
4th Grade Teacher Instruments
InstrumentNumber of Items
Pre-Test Reliability
Post-Test Reliability
Knowledge
Total ELA 21 .805 .833
Vocabulary 8 .820 .862
Reading Comprehensi
on7 .658 .722
Writing 6 .511 .497
Practice
Vocabulary 16 .887 .904
Reading Comprehensi
on16 .879 .879
Writing 14 .911 .915
4th Grade Teacher Results- Knowledge
InstrumentExperimental
Group (n = 36)
Control Group(n = 49)
ELAPre 37% 36%
Post 52% 38%
VocabularyPre 16% 14 %
Post 22% 14 %
Reading Comprehension
Pre 47% 46%
Post 65% 49%
WritingPre 49% 49%
Post 65% 47%
4th Grade Teacher Results- Practices
Instrument(scale 1-4)
Experimental Group
(n = 36)
Control Group
(n = 49)
Vocabulary
Pre 2.81 2.76
Post 3.16 2.81
Reading Comprehensio
n
Pre 3.11 3.16
Post 3.36 3.21
Writing
Pre 2.89 2.76
Post 3.31 2.89
4th Grade Teacher Results- Significance
SubjectANCOVA
SignificanceEffect Size
Knowledge
Total ELAYes
(p<.01)Medium
(.53)
Vocabulary Yes
(p<.05)Small(.27)
Reading Comprehension
Yes(p<.01)
Medium(.55)
WritingYes
(p<.01)Medium
(.75)
Practice
VocabularyYes
(p<.01)Small(.45)
Reading Comprehension
Yes(p<.01)
Small(.32)
WritingYes
(p<.01)Small(.43)
4th Grade Student Instruments
InstrumentNumber of Items
Pre-Test Reliability
Post-Test Reliability
Knowledge
Total ELA 29 .829 .852
Vocabulary 10 .708 .738
Reading Comprehensi
on10 .659 .728
Writing 9 .59 .60
Practice
Vocabulary 6 .287 .336
Reading Comprehensi
on10 .744 .760
Writing 10 .723 .746
4th Grade Student Results- Knowledge
InstrumentExperimental
Group (n = 831)
Control Group(n = 1225)
ELAPre 54% 53%
Post 63% 60%
VocabularyPre 64% 64%
Post 73% 69%
Reading Comprehension
Pre 51% 50%
Post 58% 54%
WritingPre 47% 46%
Post 59% 56%
4th Grade Student Results- Practices
InstrumentExperimental
GroupControl Group
Vocabulary
Pre (scale 0-3)
1.84 n = 762
1.85n = 919
Post (scale 0-3)
1.89n = 750
1.89n = 902
Reading Comprehension
Pre (scale 0-3)
2.14 n = 765
2.11n = 918
Post (scale 0-3)
2.20n = 757
2.16n = 919
Writing
Pre (scale 1-3)
2.36n = 757
2.33n = 913
Post(scale 1-3)
2.44n = 749
2.38n = 908
4th Grade Student Results- Significance
SubjectANCOVA
Significance
Effect SizeHLM
Significance
Knowledge
Total ELAYes
(p<.01)None(.10)
Yes
Vocabulary
Yes (p<.01)
None(.09)
Yes
Reading Comprehens
ion
Yes (p<.01)
None(.08)
No
WritingYes
(p<.01)None(.09)
No
Practice
Vocabulary No - No
Reading Comprehens
ion
Yes (p<.05)
None(.06)
No
WritingYes
(p<.05)None(.08)
Yes
5TH GRADE RESULTS
5th Grade Teacher Instruments
InstrumentNumber of Items
Pre-Test Reliability
Post-Test Reliability
Knowledge
Total Mathematic
s31 .72 .80
Fractions 9 .54 .48
Algebraic Thinking
8 .55 .60
Measurement
14 .55 .67
Practice
Fractions 4 .61 .60
Algebraic Thinking
8 .82 .82
Measurement
26 .82 .88
5th Grade Teacher Results- Knowledge
InstrumentExperimental
Group (n = 34)
Control Group(n = 45)
MathematicsPre 46% 45%
Post 58% 44%
FractionsPre 60% 57%
Post 63% 54%
Algebraic Thinking
Pre 47% 50%
Post 61% 49%
MeasurementPre 36% 34%
Post 53% 34%
5th Grade Teacher Results- Practices
Instrument(scale 1-4)
Experimental Group
(n = 34)
Control Group
(n = 45)
Fractions
Pre 2.74 2.87
Post 3.16 2.68
Algebraic Thinking
Pre 2.87 3.01
Post 3.42 2.98
Measurement
Pre 2.37 2.59
Post 2.91 2.74
5th Grade Teacher Results- Significance
SubjectANCOVA
SignificanceEffect Size
Knowledge
Total Mathematics
Yes(p<.01)
Large(.71)
FractionsYes
(p<.05)Small(.29)
Algebraic Thinking
Yes(p<.01)
Medium(.43)
Measurement
Yes(p<.01)
Large(.93)
Practice
FractionsYes
(p<.01)Large(.88)
Algebraic Thinking
Yes(p<.01)
Large(.75)
Measurement
Yes(p<.01)
Large(.57)
5th Grade Student Instruments
InstrumentNumber of Items
Pre-Test Reliability
Post-Test Reliability
Knowledge
Total Mathematic
s29 .73 .85
Fractions 8 .49 .68
Algebraic Thinking
10 .60 .73
Measurement
11 .48 .59
5th Grade Student Results- Knowledge
InstrumentExperimental
Group (n = 648)
Control Group(n = 790)
MathematicsPre 42% 44%
Post 54% 53%
FractionsPre 30% 32%
Post 48% 46%
Algebraic Thinking
Pre 43% 45%
Post 54% 53%
MeasurementPre 50% 51%
Post 60% 59%
5th Grade Student Results- Significance
SubjectANCOVA
Significance
Effect SizeHLM
Significance
Knowledge
Total Mathematic
s
Yes (p<.01)
None(.10)
No
FractionsYes
(p<.05)None(.09)
No
Algebraic Thinking
Yes (p<.05)
None(.07)
No
Measurement
No - No
7TH GRADE RESULTS
7th Grade Teacher Instruments
InstrumentNumber of Items
Pre-Test Reliability
Post-Test Reliability
Knowledge
Total ELA 22 .731 .728
Vocabulary 12 .703 .700
Reading Comprehensi
on5 .475 .485
Writing 4 .120 .214
Practice
Vocabulary 4 .473 .506
Reading Comprehensi
on13 .871 .886
Writing 17 .836 .833
7th Grade Teacher Results- Knowledge
InstrumentExperimental
Group (n = 36)
Control Group(n = 49)
ELAPre 34% 30%
Post40%
30%
VocabularyPre 26% 21%
Post 31% 22%
Reading Comprehension
Pre 37% 35%
Post 45% 36%
WritingPre 47% 41%
Post 54% 40%
7th Grade Teacher Results- Practices
Instrument (scale 1-4)
Experimental Group
(n = 35)
Control Group
(n = 45)
Vocabulary
Pre 2.9 3.1
Post 3.2 3.2
Reading Comprehensio
n
Pre 2.6 2.6
Post 2.8 2.7
Writing
Pre 3.0 3.1
Post 3.3 3.2
7th Grade Teacher Results- Significance
SubjectANCOVA
SignificanceEffect Size
Knowledge
Total ELAYes
(p<.01)Medium
(.48)
VocabularyYes
(p<.05)Medium
(.34)
Reading Comprehension
No -
WritingYes
(p<.01)Medium
(.47)
Practice
VocabularyYes
(p<.05)Small(.30)
Reading Comprehension
No -
WritingYes
(p<.05)Small(.28)
7th Grade Student Instruments
InstrumentNumber of Items
Pre-Test Reliability
Post-Test Reliability
Knowledge
Total ELA 24 .807 .833
Vocabulary 10 .701 .739
Reading Comprehensi
on10 .695 .720
Writing 4 .262 .438
Practice
Reading Comprehensi
on12 .702 .773
Writing 21 .844 .863
7th Grade Student Results- Knowledge
InstrumentExperimental
Group (n = 831)
Control Group(n = 1225)
ELAPre 64% 66%
Post 67% 66%
VocabularyPre 70% 71%
Post 72% 71%
Reading Comprehension
Pre 62% 63%
Post 62% 61%
WritingPre 58% 59%
Post 67% 66%
7th Grade Student Results- Practices
InstrumentExperimental Group
Control Group
Reading Comprehension
Pre (scale 0-1)
.44 n = 789
.43n = 1154
Post (scale 0-1)
.50n = 797
.46n = 1162
Writing
Pre (scale 1-3)
2.25n = 830
2.22*n = 1224
Post(scale 1-3)
2.33n = 826
2.24n = 1225
* There was a significant difference between the pre-scores of the experimental and control groups.
7th Grade Student Results- Significance
SubjectANCOVA
Significance
Effect Size
HLM Significance
Knowledge
Total ELAYes
(p<.01)None(.08)
No
Vocabulary
Yes (p<.05)
None(.05)
No
Reading Comprehensi
on
Yes (p<.05)
None(.07)
No
Writing No - No
Practice
Reading Comprehensi
on
Yes (p<.01)
None(.11) Yes
WritingYes
(p<.01)None(.14)
No
8TH GRADE RESULTS
8th Grade Teacher Instruments
InstrumentNumber of Items
Pre-Test Reliability
Post-Test Reliability
Knowledge
Total Mathematic
s46 .809 .847
Proportional Reasoning
29 .690 .772
Geometric Measureme
nt10 .430 .568
Functions 7 .597 .657
Practice
Proportional Reasoning
16 .843 .883
Geometric Measureme
nt12 .772 .757
Functions 37 .947 .951
8th Grade Teacher Results- Knowledge
InstrumentExperimental
Group (n = 28)
Control Group(n = 43)
MathematicsPre 38% 34%
Post 42% 35%
Proportional Reasoning
Pre 35% 33%
Post 40% 34%
Geometric Measurement
Pre 41% 36%
Post 47% 34%
FunctionsPre 47% 37%
Post 49% 40%
8th Grade Teacher Results- Practices
Instrument (scale 1-4)
Experimental Group
(n = 28)
Control Group
(n = 43)
Proportional Reasoning
Pre 2.7 2.7
Post 2.9 2.6
Geometric Measurement
Pre 2.6 3.8
Post 2.9 3.8
Functions
Pre 2.5 4.0
Post 2.9 3.9
8th Grade Teacher Results- Significance
SubjectANCOVA
SignificanceEffect Size
Knowledge
Total Mathematics
Yes(p<.01)
Small(.29)
Proportional Reasoning
Yes(p<.05)
Small(.28)
Geometric Measurement
Yes(p<.01)
Medium(.43)
Functions No -
Practice
Proportional Reasoning
Yes(p<.01)
Large(.54)
Geometric Measurement
Yes(p<.05)
Medium(.34)
FunctionsYes
(p<.01)Large(.51)
8th Grade Student Instruments
InstrumentNumber of Items
Pre-Test Reliability
Post-Test Reliability
Knowledge
Total Mathematic
s36 .704 .758
Proportional Reasoning
10 .416 .440
Geometric Measureme
nt10 .493 .642
Functions 16 .499 .531
8th Grade Student Results- Knowledge
InstrumentExperimental
Group (n = 799)
Control Group(n = 1090)
MathematicsPre 50% 47%
Post 52% 48%
Proportional Reasoning
Pre 59% 56%
Post 55% 53%
Geometric Measurement
Pre 43% 40%
Post 47% 43%
FunctionsPre 43% 40%
Post 52% 48%
8th Grade Student Results- Significance
SubjectANCOVA
Significance
Effect Size
HLM Significanc
e
Knowledge
Total Mathematics
Yes (p<.01)
None(.01)
Yes
Proportional Reasoning
No - No
Geometric Measurement
Yes (p<.01)
None(.07)
Yes
FunctionsYes
(p<.01)None(.13)
Yes
Summary of Research Findings for Teachers (ELA)
4th grade: Experimental group teachers had
significantly higher knowledge and practice scores than the control group teachers for vocabulary, reading comprehension, writing, and total ELA scores
7th grade: Experimental group teachers had
significantly higher knowledge and practice scores than the control group teachers for vocabulary, writing and total ELA scores (reading comprehension was not significant)
Summary of Research Findings for Teachers (Mathematics)
5th grade: Experimental group teachers had significantly higher
knowledge and practice scores than the control group teachers for fractions, measurement, algebraic thinking and total mathematics scores
8th grade: Experimental group teachers had significantly higher
knowledge scores than the control group teachers for proportional reasoning, geometric measurement and total mathematics scores (function scores were not significant)
Experimental group teachers had significantly higher practice scores than the control group teachers for proportional reasoning, geometric measurement, functions and total mathematics scores
Summary of Research Findings for Students (ELA)
We focus on the results of the HLM analyses as they provide a more accurate representation of the treatment effect
4th grade: Experimental group students had
significantly higher knowledge and practice scores than the control group for vocabulary knowledge and writing practice scores, and for the total ELA knowledge scores
7th grade: Experimental group students had
significantly higher practice scores than the control group for writing practice only
Summary of Research Findings for Students (Mathematics)
5th grade: No treatment effects were observed
8th grade: Experimental group students had
significantly higher knowledge scores than the control group for geometric measurement and functions scores, and for the total mathematics scores
Next steps…..
Analysis is ongoing
top related