interaction with and reflection on the noyce program evaluation data frances lawrenz pey-yan liou...
Post on 20-Dec-2015
216 views
TRANSCRIPT
Interaction with and Reflection on the
Noyce Program Evaluation Data
Frances LawrenzPey-Yan Liou
Christina MadsenChristopher Desjardins
Allison Kirchhoff
University of Minnesota
Noyce Program Evaluation Group
• Our evaluation project has four major components:– Preparation of an extensive literature review
pertaining to recruitment and retention– Thematic synthesis through content analysis of
project information– Statistical analyses to produce quantitative models
of the program development– Execution of an overall program evaluation plan
through collaboration (participatory approach) with existing projects.
Methodology: Data sources• Surveys
– PI online survey (N=66)– Scholar online survey (N=555)– Disciplinary faculty online survey (N=80)
• Interviews– Scholars in progress (N=12)– Districts in progress (N= 17)
• Monitoring data collected for NSF• Self-response bias
Methodology: Analyses
• Frequencies, cross tabs, means • Open-ended responses categorization• Factor and cluster analyses• Regression and HLM analyses using 3 outcome
variables:– Factor score: Commitment to teaching in high needs school (Influence
of Noyce)
– Would you have become a teacher if you hadn’t received the Noyce scholarship?
– Would you have decided to teach in a high need school if you hadn’t participated in the Noyce scholarship program?
Components of the PI survey
• Project overview
• Noyce money
• Program characteristics and organization
• Contacts and partners
Project overview
75 active Noyce projects with 88% of the projects responding– Of the 9 non-responding PIs, 7 were in their first or second year and
subsequently had no data to report
• Projects in 29 states responded, with 12 responding from Texas
• Total of 141 strands reported with an average 1 or 2 teacher education strands per responding PI– 8 strands was the maximum– Strand breakdown:
• 49 Undergraduate program leading to a bachelor’s degree (34.8%)• 17 Teaching credential (no degree) (12.1%)• 27 Post-bac or graduate program (no master’s awarded) (19.1%)• 27 Graduate program (19.1%)• 21 Other (14.9%)
Noyce money
• 55% indicated that Noyce scholarship paid for over 75% of scholars’ tuition (n = 66) (PI Sec II Q4)
• 60% indicated that their teaching preparation program would continue mostly unchanged without the Noyce funding (n = 65) (PI Sec II Q7)
– In contrast, 62% reported that the funding made their teacher certification program different (n = 65) (PI Sec II Q8)
• In general, PIs indicated that Noyce funding:– Greatly increased their ability to recruit a variety of students– Had a lesser effect on perceptions about teaching careers– Had a lesser effect on relationships with community, districts, STEM
faculty, and industry. (PI Sec II Q9)
Funded activities
33%
37%
39%
54%
34%
54%
58%
67%
52%
82%
97%
0.0% 20.0% 40.0% 60.0% 80.0% 100.0%
Mentoring by individuals not in school
Mentoring by experienced teachers
Mentoring by teachers in the same discipline
Mentoring in the schools
Meetings with district personnel and/or teachers
Meetings of the Noyce scholars after the program
Meeting with university or other instructional faculty
Meetings of Noyce scholars during the program
Special educational programs or sessions
Data collected about scholars
Scholarships or stipends
Miscellaneous Meetings Mentoring
PI Sec II Q 1, number of respondents for each item ranged from 65 to 56.
Program characteristics• Required activities (PI Sec III Q2)
– Most: Teaching methods classes specific to candidate’s subject area; student teaching experience
– Least: Mentoring during the candidate’s second year of teaching
• Required field experience (PI Sec III Q3)
– Most: Supervised actual classroom teaching in high needs schools (mode # of hrs > 121).
– Least: Education field experience working outside of schools with young people like those who attend high needs schools in your area (mode # of hrs 1-40)
• Collaborations with school districts (PI Sec III Q5)
– Most: School districts providing practicum sites for scholars– Least: The school districts agreeing to hire all scholar who successfully
complete the program.
Screening criteria for selecting Noyce scholars
Criteria n % responding yes
GPA 65 98.5
Personal statement 64 97.0
Letters of recommendation 60 90.9
Undergraduate status 66 81.8
Structured interviews 63 73.0
Bachelor’s degree 65 72.3
Previous experience 64 68.8
Standardized tests 63 36.5
STEM courses beyond required for bachelor’s 63 36.5
Subject matter test 63 30.2
Attitude tests or measures 63 17.7
Observation of a teaching event 64 14.1
PI Sec III Q 3
Components of the scholar survey
• Project overview• Program characteristics and
organization• Teaching environment and
experience• The decision to become a teacher• Background and experience• Overall experience
Project overview
• Of the 555 scholars responding to the survey:– 46% were teaching full-time/part-time
– 31% were still in their certification program but not yet a full-time teacher
– 13% were still in their program but also teaching full-time
– 8% completed a program but never taught
– 1% left their program without completing certification
– 1% taught after being certified and were working in education but not as teachers
– 1% taught after being certified but were no longing working in education
Program characteristics
• Features of teacher certification program (Scholar Sec II Q1)
– Most: Classes in teaching methods specific to their subject area.
– Least: Guaranteed job at a participating school district.
• Developing specific strategies for teaching students from diverse racial and ethnic backgrounds was the most extensively explored opportunity across the programs (Scholar Sec II Q4)
Teaching environment
• 83% of the 320 responding scholars are currently teaching in a high needs school (Scholar Sec III Q1)
• 68% of the 324 responding scholars are in schools where over 50% of students receive free or reduced lunch (Scholar Sec III Q3)
• 41% of the 322 responding scholars are in schools where there has been 15% teacher attrition over the last 3 years (Scholar Sec III Q3)
• 78% of the 318 responding scholars have held leadership positions within the last 3 years
(Scholar Sec III Q5)
Decision to become a teacher
• 53% of scholars decided to become a STEM teacher during adulthood (23 or older) (n = 540) (Scholar Sec IV Q1)
• 87% of scholars learned about the Noyce scholarship after they decided to become a teacher (n = 543) (Scholar Sec IV Q4)
• 61% of scholars learned about the Noyce scholarship from their advisor (n = 586) (Scholar Sec IV
Q5)
Factors influencing STEM teacher decision
Scholar Sec IV Q3, respondents may have multiple selections (respondents vary from 516 to 535)
72.3%
76.0%
76.7%
89.9%
93.1%
96.6%
97.0%
0.0% 20.0% 40.0% 60.0% 80.0% 100.0%
I like having summers off
I like the flexibility and/or autonomy of STEM teaching
I feel a teaching career is conducive to my family life
I feel that I have talent for STEM teaching
I like sharing my subject with others
I like working with young people
I feel this career allows me to 'make a difference' in theworld
Effect of Noyce scholarship on teaching
Would you have become a teacher if you had not received the Noyce Scholarship? Would you have taught in a high needs school if you hadn’t participated in the Noyce program?
Yes Possibly No I will not teach in a high needs school
Scholar Sec IV Q6 (n = 543) & Q7 (n = 542)
Faculty survey results
• 91% were familiar with the Noyce program (n = 79) (Fac Q1)
• 44% interact with STEM education faculty daily and 25% interact weekly (n = 79) (Fac Q2)
– Half of those interacting daily were in mathematics
• Faculty encouraged students to pursue teaching because of the students’ interest or because of the students’ personalities (Fac Q6a)
• Most suggested response from faculty to students interested in teaching was that they should get actual experience teaching and that teaching is a ‘noble’ profession (Fac Q6b)
• 57% indicated that availability of scholarships affects advice they give to STEM majors (n = 69) (Fac Q6c)
3.8%
61.3%
20.0%
15.0%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
Higher ability Similar ability Lower ability Cannot compare
Ability of STEM majors who intend to teach compared to
those that do not
Fac Q4 (n = 80)
7.0%
62.0%
19.7%
4.2%7.0%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
Strong positive Moderate positive Minimal positive Too early to tell Other
Perceived effect of Noyce on STEM teacher certification
program
Fac Q1 (n = 71)
Group work 1: Data analysis
• Work in small groups or individually for 10 minutes to develop a question that you would like us to explore
• At the end of the 10 minutes hand in your question and we’ll select a suggestion(s) to analyze
• We’ll discuss these results and implications after our next group work activity
PI Suggestions1. Career/non-career changers
2. Timing of teaching decision
3. Unique program characteristics
Career/non career changers
• Overview
• Demographics
• High needs teaching
Overview
• Career changers– ½ of all responding scholars indicated they were
‘career changers’ (n=532)– Of the scholars indicating being career changers:
• All scholars at seven institutions indicated they were ‘career changers’
• 25 scholars at one institution indicated they were ‘career changers’
– Of the scholars indicating not being career changers:
• All scholars at four institutions indicated they were ‘non-career changers’
• 20 scholars at one institution indicated they were ‘non-career changers’
Demographics
• Of the 267 ‘career changing’ scholars: – Program type
• 67% are/were in graduate programs (masters)• 14% are/were in teaching credential programs (no degree)
– Teaching status• 47% are/were teaching full-time• 5% completed a teacher program, but never taught
• Of the 265 ‘non career changing’ scholars:– Program type
• 37% are/were in undergraduate programs• 9% are/were in teaching credential programs (no degree)
– Teaching status• 43% are/were teaching full-time• 10% completed a teacher program, but never taught
High needs teaching
Teaching high needs school Teaching another type of school
Career changers Non-career changers
Scholar Sec III Q1 & Sec V Q4, career changers (n=267), non-career changers (n=265)
Timing of teaching decision
Scholars were asked, “Did you first learn about the Noyce scholarship before or after you decided to
become a teacher?”
• Program type
• Recruitment strategies
Program type
21%19% 19%
37%
4%
29%
10%
22%
34%
6%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
Undergraduateprogram
Teachingcredential
Post-bac orgrad program(no masters)
Grad program(mastersawarded)
Other
Before After
Scholar Sec I Q1 & Sec IV Q 4, Before (n=74), After (n=469)
Recruitment strategies
18%
53%
9%
20%
12%
62%
3%
23%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
Website Advisor Career center Other
Before After
Scholar Sec IV Q4 & Q5, Before (n=74), After (n=469)
Unique program characteristics
•Experiences in teacher certification program
•Required field experiences
Experiences• Scholars responded to 14 experiences in their teacher
certification program (Scholar Sec II Q1)
– Of the 14 experiences, 2 were unique:• Guaranteed job at a participating school district
– 27% of scholars indicated they were guaranteed a job (n=544)
– Of the scholars indicating yes, 38% were from graduate programs (masters) whereas 9% were from teaching credential programs (n=144)
• Mentoring provided by certification program during the second year
– 32% of scholars indicated they received second year mentoring by their program (n=524)
– Of the scholars indicating yes, 38% were from graduate programs (masters) whereas 16% were from teaching credential programs (n=165)
Required field experiences• Scholars responded to 5 required field experiences in their
teacher certification program (Scholar Sec II Q 2)
– Of the 5 field experiences, 2 were unique:• Education field experience working outside of schools (e.g.
summer camp) with young people like those who attend high needs schools in the area
– 11% of scholars indicated they received education field experience outside of schools (n=541)
– 39% of the scholars were in undergraduate program leading to a bachelors degree whereas 12% were from teaching credential programs (n=57)
• Research field experience in your subject area– 27% of scholars indicated they received research field experience
(n=539)
– 44% of scholars were in graduate programs (masters) whereas 11% were in teaching credential programs (n=143)
Group Work 2: Discussion
• Spend a few minutes thinking about the discussion questions included in your packet.
• Then in a small group, you’ll discuss either the first or second question.
• We’ll then reconvene and discuss as a large group ideas that were raised during the small group discussion.
Factor analysis
• Factor analysis was utilized to discover underlying factors and as a psychometric procedure for the development and refinement of the questionnaire
• One main goal is to combine many items into a construct, and use this construct to do more analyses
Factor analysis results of Noyce scholar surveyConstructed Factor Cronbach’s
AlphaNumber of Items
Commitment to teaching in a high needs school (influence of Noyce)
0.893 6
Preparation for high needs school 0.730 13
Path to teaching 0.722 7
District/school high needs environment
0.716 5
Personal beliefs towards teaching 0.611 8
School teaching environment 0.775 4
Mentoring experience 0.724 6
Commitment to teaching in high needs school
(influence of Noyce)
• Become a teacher (Scholar Sec IV Q8a)
• Complete the certification program (Scholar Sec IV Q8b)
• Take a teaching job (Scholar Sec IV Q8c)
• Teach in a high needs school (Scholar Sec IV Q8d)
• Remain teaching in a high needs school for the full term of your commitment (Scholar Sec IV Q8e)
• Remain teaching in a high needs school beyond the full term of your commitment (Scholar Sec
IV Q8f)
1. teach in a high needs school (Scholar Sec IV Q8d)
2. remain teaching in a high needs school for the full term of your commitment (Scholar Sec IV Q8e)3. remain teaching in a high needs school beyond the full term of your commitment (Scholar Sec IV Q8f)4. become a teacher (Scholar Sec IV Q8a)5. take a teaching job (Scholar Sec IV Q8c)6. complete the certification program (Scholar Sec IV Q8b)
Hierarchical cluster analysis of commitment to teaching in a high needs school
123456
Influence of Noyce vs. timing Commitment to teaching in a high needs school (Influence of Noyce scholarship)
Did you first learn about the Noyce scholarship before or after you decided to become a teacher?
Commitment to teaching in a high needs school (Influence of Noyce scholarship)
Spearman’s rho
N
Did you first learn about the Noyce scholarship before or after you decided to become a teacher?
Spearman’s rho
N
-.250**
530
**. Correlation is significant at the 0.01 level (2-tailed).
1: before2: after
(Scholar Sec IV Q4)
AfterBefore
Commitment to high needs teaching
Low
High
Spearman’s rho = -0.250
Influence of Noyce vs. timing
Did you first learn about the Noyce scholarship before or after you decided to become a teacher?
* Not the actual relationship, just a visual representation
(Scholar Sec IV Q4)
Influence of funding on timing of teaching decision
21.6%
1.5%
31.1%
14.1%
47.3%
84.4%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Before After
Did you first learn about the Noyce scholarship before or after you decide to become a teacher?
Yes
Possible
No
2
Would you have becomea teacher if you had not received the Noyce scholarship?
(2, N=542) = 83.296, p<0.001
(Scholar Sec IVQ4)
(Scholar Sec IVQ6)
Hierarchical Linear Models (HLM)
• Also known as multi-level analysis, is a more advanced form of simple linear regression and multiple linear regression
Why is HLM used in analyzing Noyce data?• The program-level variables help to explain
between-program variance rather than within-program variance
• Scholars in the same program tend to be more alike than in other projects
Three outcome variables in HLM
• Factor score: Commitment to teaching in a high needs school (influence of Noyce)
• Would you have become a teacher if you hadn’t received the Noyce scholarship? (Scholar Sec IV Q6)
• Would you have decided to teach in a high needs school if you hadn’t participated in the Noyce scholarship program? (Scholar Sec IV Q7)
Group Work 3: Searching for HLM PI predictors
• Brief overview of predictor analyses
• Spend a few minutes reading through the PI level variables handout
• Work in small groups or individually for 10 minutes to discuss the PI level variables that you think they would influence the outcome variable “Commitment to teaching in a high needs school (influence of Noyce)”
1: Commitment to teaching in a high needs school
• What proportion of the total tuition scholars need to pay is provided by the Noyce scholarship funding? (0.10, p=0.013) (PI Sec II Q4)
• Your ability to recruit participants from ethnicities under-represented in STEM into your certification/licensure program (0.10, p=0.047) (PI Sec II Q9f)
• Your relationships with businesses or industry (0.13, p=0.049) (PI Sec II Q9l)
• Interactions with children from different cultures (-0.33, p=0.01) (PI Sec III Q2b)
• Teaching methods class(es) specific to the candidate’s subject area (0.36, p=0.032) (PI Sec III Q2d)
• Student teaching experience (-0.29, p=0.015) (PI Sec III Q2g)
1: Commitment to teaching in a high needs school
• What proportion of the total tuition scholars need to pay is provided by the Noyce scholarship funding? (0.10, p=0.013) (0.12, p=0.001) (PI Sec II Q4)
• Your ability to recruit participants from ethnicities under-represented in STEM into your certification/licensure program (0.10, p=0.047) (0.11, p=0.001) (PI Sec II Q9f)
• Your relationships with businesses or industry (0.13, p=0.049) (PI Sec II Q9I)
• Interactions with children from different cultures (-0.33, p=0.01) (PI Sec III Q2b)
• Teaching methods class(es) specific to the candidate’s subject area (0.36, p=0.032) (PI Sec III Q2d)
• Student teaching experience (-0.29, p=0.015) (PI Sec III Q2g)
We would like to thank NSF and all those involved with the Noyce project!!
Thank you for your participation and valuable ideas!