austin youth program quality: a snapshot

A joint venture between the Forum and High/Scope.

Austin Youth Program Quality: A Snapshot

Dr. Raphael Travis – Texas State University School of Social Work

Raphael Travis, Jr., DrPH, LCSW 2

Overview

Five waves of data collection and analysis Sixth wave coming soon 219 assessments entered and analyzed

Only possible from a dedicated few volunteering time, energy and resources … you!


A Snapshot

1. Average levels of program quality areas across programs Successes and Opportunities

2. Analyses across each wave of data3. Distinctions between self-assessments and external

assessments4. A spotlight on what predicts high youth engagement

(our most desirable indicator of quality)


The Pyramid of Program Quality

5

PlanMake choices

Reflect

Partner with adults

Lead and mentorBe in small groups

Experience belonging

Engagement

Reframing conflictEncouragementSkill building

Active engagementSession flow

Welcoming atmosphereSupportive

Environment

Interaction

Safe Environment Healthy food and drinks

Program space and furniture Emergency proceduresPsychological and emotional safety

Physically safe environment

Professional Learning CommunityYouth Voice and Governance


Score Reports: Keep in Mind

Observation scores represent a snapshot –this has limitations and value.

These are aggregates where multiple observations occurred.

The overall story is more important than the actual numbers… But, today we will talk about numbers

What you do with it matters (how you step up to action).

Contemplation

Preparation

Action


Wave 5:National Sample vs. External Assessments

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

Safe Support Interact Engage

Scor

e External Reports

National Sample


Wave 5:Self Assessment vs. External Assessment

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

Safe Support Interact Engage

Scor

e Self Reports

External Reports


Wave 4 vs. Wave 5 (All Assessments)

Safe Support Interact Engage0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

WAVE 4 (Nov 2010) WAVE 5Sc

ore


Wave 5 Safe Environment: Very High Scores


Wave 5 Supportive Environment:Very High Scores – Even Higher (~5.0)


Wave 5 Interaction: Big Spread in Programs Occur; Scores Dip Below 2 and 3


Wave 5 Engagement:Substantial Variability; But Majority are 3 and Below


Predicting Engagement

Certain processes are consistently more important than others in predicting high levels of engagement


Safety

SUCCESS: SAFEB3/4 – Ventilation, lighting, temperature okay

OPPORTUNITY: SAFEC4/C6 – Appropriate safety and emergency equipment;

access to outdoor space is supervised


Supportive

SUCCESS: SUPPORTF2/F3/J1/J2 – Warm tone, friendly, smile; staff

actively involved, acknowledge accomplishments specifically OPPORTUNITY:

SUPPORTK2/K3 – (Conflict) Staff seek input from both youth about cause and solution of negative behavior; encourage youth to reflect on actions and consequences when conflict


Interaction

SUCCESS: INTERACTL3/O2 – Youth identify with program (shared

language, gestures, and traditions); staff provides explanations, guidelines and directions

OPPORTUNITY: INTERACTM2/N2/N3 – Staff use two or more ways to form

groups; youth have chances to mentor others and lead groups


Engagement

SUCCESS: ENGR1/R4 – All youth engage in intentional reflecting on

activities; staff provide structure for youth feedback on activities

OPPORTUNITY: ENGP2/Q2 – Two or more strategies used to plan projects; all

youth can make at least open-ended choice during program processes


Explaining High Youth Engagement

Safety, supportive, and interaction explained half of all the variability in engagement scores, in addition, even more (58%) when looking only at self-

assessments.


Significant Predictors of Engagement (All)

Wave 1: SUPPORTH Wave 2: SUPPORTI, SUPPORTJ* Wave 3: SUPPORTG* Wave 4: SUPPORTH, SUPPORTJ, INTERACTN Wave 5: SUPPORTK ALL: SAFED, SUPPORTH, SUPPORTI, SUPPORTK,

INTERACTN


Significant Predictors of Engagement (Self and External Assessments)

Predictors of Engagement (Self) ALL: SUPPORTG, SUPPORTH, SUPPORTI, SUPPORTK,

INTERACTL, INTERACTN

Predictors of Engagement (External) ALL: SUPPORTH, INTERACTN


What Explains Engagement THE MOST?

SUPPORTH: Active Engagement Youth can create, combine, reform materials/ideas toward

tangible products; reflect on what they do SUPPORTI: Skill-Building

Support youth in building new skills; persistence SUPPORTJ: Encouragement

Acknowledge what youth do; actively involved with youth; open-ended questions

INTERACTN: Leading and Mentoring Youth have opportunities to act as group facilitators and

mentors


So What? What Does This Mean for You? For assessors

it means continuing to do the great work you are doing. You have collected high quality, meaningful information – both self-

assessors and external assessors. For methods trainers

it means there may be indicator training areas to highlight even if that is not the focus of your particular workshop

(e.g., active engagement, skill-building, encouragement, leading and mentoring).

These processes seem to help distinguish programs that are highest in engagement from those that are not.

For program leadership and staff it means continuing to pay attention to how your program operates as much as you

pay attention to the results. Highlight and pay attention to colleagues that promote these influential processes

and model effective program facilitation. For others, continue to support the work!

austin youth program quality: a snapshot

Documents