Governor’s Prevention Initiative for Youth: Evaluation

Jane A. Ungemack, Dr.P.H.Evaluator

University of Connecticut Health Center

Evaluation

• Systematic efforts to collect and use information:– Document program implementation– Describe target populations/participants– Inform and improve program performance– Access program effectiveness– Increase accountability– Increase understanding

Governor’s Prevention Initiative for Youth

Long-term Goal: Reduce substance use among adolescents

Intermediate Goal: Reduce risk factors and increase protective factors for

substance use in the individual, peer, family, school and community domains

Target Group: 12-17 year old youth

Evaluation Framework OverviewII. Contextual Conditions

(Economic, Cultural, Risk Conditions, ATOD use)

2. State System 4. State-Level 6. State-Level 8. State-LevelCharacteristics/ Collaborative Intermediate Systems

Dynamics Strategies/ Outcomes ChangeActivities

Training/TA Training/TA

1. SIGMobilization

3. Sub-RecipientCharacteristics

Training/TA

5a. SubRecipient

Planning forScience-based

PreventionInterventions

7a. Sub-Recipient/CommunityImmediate

Outcomes & systemchanges

9. CommunityIntermediateOutcomes:Risk and

ProtectiveFactors

10. Long-termOutcomes:Behavioral

Impacts

5b. ProgramInterventions:

Activities/Actions

7b. ProgramOutcomes: Risk and

Protective Factor

Evaluation Approach

Process evaluation:• documents program implementation and

activitiesOutcome evaluation: • assesses program effects or impacts

Capacity-Building for Evaluation

Science-based approachEvaluation and assessment as integral parts of

the program design– Community-level – Program-level

Evaluation Team Relationship with Grantees

• Training and technical assistance• Instruments and administration protocols• Consultation and collaboration• Statewide coordination

Grantee Responsibilities

• Develop a program plan based on the logic model • Specify measurable program objectives• Cooperate and collaborate with UConn Evaluation

Team• Coordinate community survey• Collect and submit process data • Collect and submit outcome data • Commit time and effort to evaluation activities

Assessing Community-Level Outcomes: CSAP Requirements

School surveyUse of core substance use, risk and protective

factor measures

Community-Level Assessment: School Survey

• Mandated • 7th-10th grade students • Representative sample (minimum n=500;

125/grade level)• Year 1 and Year 3• First administration: February-April, 2000

School Survey

• Self-administered during a classroom period

• Anonymous and confidential• Parental consents • Sampling, instrument and administration

protocols provided by UConn Evaluation Team

School Survey

Measures– Demographic characteristics– Lifetime and current use of ATOD– Risk and protective factors– Limited community-specific items

School Survey

Grantee responsibilities:– Planning/coordination with UConn Evaluation

Team– Planning/coordination with school personnel– Instrument duplication– Data cleaning– Data entry– Analysis

Assessing Program-Level Outcomes: CSAP Requirements

• Select a minimum of three programs (for each of three domains)

• Measure program outcomes using core measures• Include a sufficient sample size for analysis• Collect pre- and post-test data

Program-Level Evaluation

Process evaluation:• Document program implementation and activities Outcome evaluation: • Assess program outcomes

Program-Level Evaluation: Process Evaluation

Each program will be responsible for reporting:– Prevention strategies– Types of activities– Dosage– Number served– Participant characteristics (age, gender,

race/ethnicity, etc.).

Process Evaluation

• Minimum Data Set (MDS) • Instruments, protocols, and training

provided by the UConn Evaluation Team

Program-Level Evaluation:Outcome Evaluation

• Based on the logic model, identify measurable objectives that you will address in your program

• Program objectives should be selected from one or more of the risk/protective factors included in the RFP list of Connecticut Intermediate Outcomes

Outcome Evaluation

• UConn Evaluation Team staff will work with each grantee to finalize program-specific objectives and measures

• All grantees will be asked to participate in a pre/post-test assessment of age-eligible participants as appropriate

Outcome Evaluation

Pre- and Post-Test Assessments • Youth participation will be voluntary• Confidential• Informed consents • Standardized instrument plus optional

program-specific items• Minimum sample size = 50

Considerations for Estimating Evaluation Costs

• Personnel (.25 FTE minimum recommended)

• Computer equipment • Photocopying • Office supplies • Data collection and cleaning• Data entry

Evaluation Themes for the Governor’s Prevention Initiative

• Evaluation at all levels• Science-based• Capacity-building • Collaboration• Coordination • Sustainability