Evaluating YourEvaluating YourRegional CompetitionRegional Competition

Laura FlorenceLaura Florence, University of Michigan, University of Michigan

Regional Coordinator, Great Lakes BowlRegional Coordinator, Great Lakes Bowl

Why Evaluate your Regional?Why Evaluate your Regional?

Are we meeting our goals and objectives?Are we meeting our goals and objectives? Are we using the “right approach”? (focusing Are we using the “right approach”? (focusing

on correct objectives? correct methods?)on correct objectives? correct methods?) Show successes, strengths of programShow successes, strengths of program Demonstrate concrete results to funding Demonstrate concrete results to funding

sourcessources Improve your competitionImprove your competition A good place to start in program developmentA good place to start in program development

Developing a Program=PIEDeveloping a Program=PIE

Implementation

Evaluation

Planning

My Environmental Education Evaluation Resource Assistant: MEERA

www.meera.snre.umich.edu

Evaluation ProcessEvaluation Process• Used MEERA website as a guide:Used MEERA website as a guide:

Step 1: Before You Get StartedStep 1: Before You Get StartedStep 2: Clarify Program LogicStep 2: Clarify Program LogicStep 3: Set Goals and IndicatorsStep 3: Set Goals and IndicatorsStep 4: Choose Design and ToolsStep 4: Choose Design and ToolsStep 5: Collect DataStep 5: Collect DataStep 6: Analyze DataStep 6: Analyze DataStep 7: Report ResultsStep 7: Report ResultsStep 8: Improve ProgramStep 8: Improve Program

Step 1: Before You Get StartedStep 1: Before You Get Started

•What types of resources will I need to invest in the evaluation?What types of resources will I need to invest in the evaluation?

•How do I find and work with an internal or external evaluator?How do I find and work with an internal or external evaluator?

•How do I involve program managers, staff and others?How do I involve program managers, staff and others?

•How do I obtain approval for the evaluation and consent How do I obtain approval for the evaluation and consent from participants?from participants?

Step 2: Clarify Program LogicStep 2: Clarify Program Logic

•What is a logic model?What is a logic model?

•Why should I develop a logic model?Why should I develop a logic model?

•How do I get started?How do I get started?

Step 3: Set Goals and IndicatorsStep 3: Set Goals and Indicators

•What are your goals for the evaluation?What are your goals for the evaluation?

•How do I develop evaluation questions?How do I develop evaluation questions?

•How do I answer my evaluation questions?How do I answer my evaluation questions?

NOSB Evaluation Goals

Focus: Improving recruitment and retention of teams to the Great Lakes Bowl

1) What motivates teachers to participate in NOSB?

2) What factors are important for continued participation?

3) What factors would prevent teams from returning to NOSB?

4) What do NOSB teachers think are effective ways of advertising NOSB?

5) What can be the role of current NOSB teachers in recruiting new teams?

Step 4: Choose Design & ToolsStep 4: Choose Design & Tools

•What type of data should I collect?What type of data should I collect?•Which tool should I use to collect data?Which tool should I use to collect data?•When and from whom should I collect data?When and from whom should I collect data?

Step 5: Collect Data

•How can I best manage the data collection process?•How do I select evaluation participants?•How big should my sample be?•How do I prepare my data for analysis?

For NOSB evaluation:For NOSB evaluation:

Qualitative and quantitative data were collected from Qualitative and quantitative data were collected from 2009 Great Lakes Bowl teachers:2009 Great Lakes Bowl teachers:

•Focus group at the eventFocus group at the event•Follow-up survey completed after the eventFollow-up survey completed after the event

14 teachers: 7 in focus group, 10 in survey, only 1 14 teachers: 7 in focus group, 10 in survey, only 1 teacher not included in eitherteacher not included in either focus group or survey

Step 6: Analyze Data

•What type of analysis do I need?•How do I analyze quantitative and qualitative data?•What software can I use?

For NOSB Evaluation:

•Content analysis for qualitative focus group and survey dataContent analysis for qualitative focus group and survey data•Descriptive statistics (counts, frequencies, percentages, mean, Descriptive statistics (counts, frequencies, percentages, mean, median, mode, variability) for quantitative survey datamedian, mode, variability) for quantitative survey data

Step 7: Report Results

•How do I develop conclusions and recommendations?

•How should I report my results and conclusions?

•How should I organize my report?

•How do I use graphics to illustrate results?

Key results from NOSB evaluation:Key results from NOSB evaluation:

•100% of teachers surveyed are willing to help promote NOSB, 80% are willing to make a presentation at a professional meeting

•Need for additional preparation materials was not cited as a factor in retention or recruitment

•Focus recruitment efforts at teacher meetings and workshops

•Advertising– electronic and paper mailings are important

Key results from NOSB evaluation:Key results from NOSB evaluation:

•Factors motivating participation:

Personal interestStudent enrichmentLow/no costNOSB supports science curriculumAugments existing club activitiesStudent interest

Key results from NOSB evaluation:Key results from NOSB evaluation:

Educational opportunities are desirable to teachers Educational opportunities are desirable to teachers and students, but not vital for retention.and students, but not vital for retention.

However, once teachers or their students participate However, once teachers or their students participate in educational opportunities (teacher workshops, in educational opportunities (teacher workshops, internships, scholarships) they greatly value them as a internships, scholarships) they greatly value them as a program component.program component.

Early-career teachers are more likely to participate in Early-career teachers are more likely to participate in professional development opportunities that offer professional development opportunities that offer CEUs.CEUs.

Step 8: Improve Program

•How can I use my evaluation results to benefit my program?

•How can I ensure the evaluation is used to improve my program?

•In what other ways can I get the most from the evaluation?

NOSB Program Improvement Plans:NOSB Program Improvement Plans:

•Adjust 2009-2010 recruitment efforts to reflect the results of Adjust 2009-2010 recruitment efforts to reflect the results of this evaluation:this evaluation:

-Empower teachers to promote NOSB-Empower teachers to promote NOSB-Re-institute paper mailings, newsletter articles-Re-institute paper mailings, newsletter articles-Explore opportunities to present and/or exhibit at -Explore opportunities to present and/or exhibit at

teacher meetings and workshopsteacher meetings and workshops

•Plan a 2010 evaluation to provide additional data on efficacy Plan a 2010 evaluation to provide additional data on efficacy of these adjustments.of these adjustments.

•Share evaluation results with other NOSB regions and NOSB Share evaluation results with other NOSB regions and NOSB national office.national office.

More about MEERA:More about MEERA:

•What I liked about MEERAWhat I liked about MEERA

•Recommendations for use—Recommendations for use—- great for inexperienced evaluators and in-house - great for inexperienced evaluators and in-house

evaluationsevaluations- good to have a contact during the process- good to have a contact during the process

•Aspects that were particularly helpful—Aspects that were particularly helpful—- logic model- logic model- data analysis- data analysis- reporting results- reporting results- program improvement- program improvement

Questions/DiscussionQuestions/Discussion

Questions about MEERA, my evaluation? Questions about MEERA, my evaluation? Share regional evaluation experiencesShare regional evaluation experiences Success stories? What hasn’t worked?Success stories? What hasn’t worked? Ideas for what you would want to evaluate?Ideas for what you would want to evaluate? Share tips, strategies, techniquesShare tips, strategies, techniques