scaling up innovations dean l. fixsen, ph.d. & karen a. blase, ph.d. university of north...

Scaling Up Innovations

Dean L. Fixsen, Ph.D. & Karen A. Blase, Ph.D.

University of North Carolina - Chapel Hill

Rob Horner, Ph.D.

University of Oregon

George Sugai, Ph.D.

University of Connecticut

OR Superintendents’ Forum April 2010

Scaling Up Effectiveness

OR Public Schools

Students: 565,000

Schools: 1,800

School Districts: 196

Counties: 36

Budget: $8.5 Billion

“Bottom 20%”

113,000

360

39

36

Challenges

Follow Through Programs

Figure 1: This figure shows the average effects of nine Follow Through models on measures of basic skills (word knowledge, spelling, language, and math computation), cognitive-conceptual skills (reading comprehension, math concepts, and math problem solving) and self-concept. This figure is

adapted from Engelmann, S. and Carnine, D. (1982), Theory of Instruction: Principles and applications. New York: Irvington Press.

Hattie (2009) recently reported a meta-analysis of 816 meta-analyses

52,649 research studies in education involving over 83 million students, teachers, staff, parents, and others.

"It is what teachers get the students to do in the class that emerged as the strongest component of the accomplished teachers' repertoire."

Hattie (2009)

Achieving Student Benefits

Increasing opportunities to respond and the amount/ accuracy of feedback is an important correlate of student achievement

The feedback to the teachers about what students can and cannot do is more powerful than feedback to the student

This requires a change in the conception of what it means to be a teacher – not a solo performer

Hattie (2009)


We now know a lot about WHAT to do to educate students

We can improve education for students – on purpose!


Science “to” Service

SCIENCE SERVICEGAPIMPLEMENTATION

Challenges

Science to Service Gap

What is known is not what is adopted to help students

Implementation Gap

What is adopted is not used fully and effectively in practice


Good Intentions

Actual SupportsYears 1-3

Outcomes

Every Teacher Trained

Fewer than 50% of the teachers received some training

Fewer than 10% of the schools used the CSR as intended

Every Teacher Continually Supported

Fewer than 25% of the teachers received support

Vast majority of students did not benefit

Aladjem & Borman, 2006; Vernez, Karam, Mariano, & DeMartini, 2006

Longitudinal Studies of a Variety of Comprehensive School Reforms

You are not alone!

Superintendents across the nation are facing the same problems:

Lack of consistency across teachers, schools, and years

Lack of capacity to make meaningful changes and sustain them

Challenges

In 2007

State of Oregon participated in a process to select States to create an infrastructure for implementation of innovations statewide.

1 of 36 interested States

1 of 16 applicant States

1 of 6 chosen States that met the selection criteria and site visit criteria (IL,MI, MN, MO, OR, VA)

SISEP Center

SISEP Center

State Implementation and Scaling up of Evidence-based Practices (SISEP)

www.scalingup.org

Dean Fixsen and Karen BlaseNational Implementation Research Network, University of North Carolina at Chapel Hill

Rob HornerUniversity of Oregon

George SugaiUniversity of Connecticut

http://www.scalingup.org/

The SISEP Center – Intensive and focused activity to build state capacity and align system structures, roles, and functions

Use implementation science and best practices across programs and innovations

Large scale, real time change

Capacity Building

Students cannot benefit from interventions they do not experience

Teachers and staff have to change if students are to benefit

Dobson & Cook (1980)

Challenges

Know-WHAT

Knowledge of the intervention

Know-HOW

Knowledge of implementation

Tucker, Edmondson, & Nembhard (2005)

Implementation Science

Know WHAT

Choose Interventions Wisely Meaningful Improvement

Must be “worth the effort” to scale up (e.g. EBISS)

Eventually want to see educationally and socially significant changes in student outcomes across the State

Know HOW

Effective NOT Effective

Effective

NOT Effective

IMPLEMENTATION

INT

ER

VE

NT

ION Student Benefits

Highly variable, often ineffective, sometimes harmful to students,

families, and adults

(Institute of Medicine, 2000; 2001; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)

Poor Outcomes

Implementation

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Download all or part of the monograph at:

http://www.fpg.unc.edu/~nirn/resources/detail.cfm?resourceID=31

Implementation Research: A Synthesis of the Literature


Data Show These Methods, When Used Alone, Do Not Result In Uses of Innovations As Intended:

Diffusion/ Dissemination of information

Training

Passing laws/ mandates/ regulations

Providing funding/ incentives

Organization change/ reorganization


Data Show These Methods, When Used Alone, Do Not Result In Uses of Innovations As Intended

We know a lot about ineffective methods because they are the ones we use!

Implementation science will improve as implementation practices improve (create a better “laboratory”)


Student Benefits

Technical

Integrated & Compensatory

Performance Assessment (Fidelity)

Coaching

Training

Selection

Systems Intervention

Facilitative Administration

Decision Support Data System

Innovation Organization

Leadership

Adaptive

•Exploration (Sustainability)

•Installation (Sustainability)

•Initial Implementation

•Full Implementation (Effectiveness & Sustainability)

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

2 – 4 Years

Implementation Takes Time

Implementation Drivers Common features of successful

supports to help make full and effective uses of a wide variety of innovations


© Fixsen & Blase, 2007

Integrated & Compensatory

Performance Assessment (Fidelity)

Coaching

Training

Selection

Staff Competence

Systems Intervention

Facilitative Administration

Decision Support Data System

Organization Supports

TechnicalLeadership

Adaptive

Reliable Benefits for Students

Consistent uses of Innovations

OUTCOMES(% of Participants who Demonstrate Knowledge, Demonstrate

new Skills in a Training Setting, and Use new Skills in the Classroom)

TRAININGCOMPONENTS

KnowledgeSkill

DemonstrationUse in the Classroom

Theory and Discussion

10%

5% 0%

..+Demonstration in Training

30%20%

0%

…+ Practice & Feedback in Training

60% 60% 5%

…+ Coaching in Classroom

95% 95% 95%

Joyce and Showers, 2002

Staff Coaching

Support Implementation

Students cannot benefit from education practices they do not experience

Support implementation practices within schools and districts

Capacity Building Scaling up = at least 60% of

the students who could benefit from an innovation have access to that innovation

Achieve significant benefits to students and society

Letting it happen Recipients are accountable

Helping it happen Recipients are accountable

Making it happen Purposeful use of implementation

practices and science

Implementation teams are accountable

Based on Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou, 2004

Capacity Building

Letting it happen

Recipients are accountable

Helping it happen

Recipients are accountable

Making it happen

Implementation Teams are accountable: THEY DO THE WORK (Heart of Scaling)

Based on Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou, 2004

Capacity Building

Implementation Team

Minimum of three people (four or five preferred) to promote effective, efficient, and sustainable implementation, organization change, and system transformation work

Tolerate turnover; teams are sustainable even when the players come and go

Implementation Team

A group that knows the innovations very well (formal and craft knowledge)

A group that knows implementation very well (formal and craft knowledge)

A group that knows improvement cycles to make intervention and implementation methods more effective and efficient over time

Implementation Team

School & District Supports

Management (leadership, policy)

Administration (HR, structure)

Supervision (nature, content)

Teacher & Staff Competence

State and Community Supports

Regional Authority Supports

Imp

lem

enta

tio

n T

eam

Simultaneous, Multi-Level Interventions

Implementation Team

Implementation Team

Prepare Communities

Prepare schools and staff

Work with Researchers

Assure Implementation

Prepare Regions Assure Student Benefits

Create Readiness

Parents and Stakeholders


Impl. Team NO Impl. Team

Effective

Effective use of Implementation Science & Practice

IMPLEMENTATION

INT

ER

VE

NT

ION

80%, 3 Yrs 14%, 17 Yrs

Balas & Boren, 2000Fixsen, Blase, Timbers, & Wolf, 2001


Letting it Happen Helping it Happen

School Wide PBS

SWPBS # of Schools

0

2,000

4,000

6,000

8,000

10,000

12,000

Aug 04 Aug 05 Aug 06 Aug 07 Aug 08 Aug 09

School Wide PBS

SWPBS % of Schools

010,00020,00030,00040,00050,00060,00070,00080,00090,000

Aug 04 Aug 05 Aug 06 Aug 07 Aug 08 Aug 09

12% in 17 Years (1992-2009)

Costs and Savings

Implementation Costs & Savings(Inflation Adjusted)

60

70

80

90

100

110

120

1 Yr Pre During PostYear 1

PostYear 2

PostYear 3

Ch

ang

e in

Bu

dg

et (

Per

cen

t)

Short-Term Investment in Imple. Capacity

Realize Long-Term Benefits

This year’s success pays for next years increase in capacity

Barber & Fullan (2005)

Costs and Savings

Change Systems

To scale up, we need to:

Turn policy into effective practice

Create an infrastructure for implementation of innovations

Turn effective practice into policy

OR Public Schools

Students: 565,000

Schools: 1,800

School Districts: 196

Counties: 36

Budget: $8.5 Billion

“Bottom 20%”

113,000

360

39

36

Challenges

PROBLEM: The “bottom 20%” is distributed throughout the state and shifts each year

SOLUTION: Plan capacity to reach ALL schools


N = 565,000All Students & All Students & FamiliesFamilies

All Students & All Students & FamiliesFamilies

School Teachers School Teachers and Staffand Staff


StateStateDepartment Department LeadershipLeadership


District Leaders District Leaders and Staffand Staff

District Leaders District Leaders and Staffand Staff


N = 565,000All Students & All Students & FamiliesFamilies

All Students & All Students & FamiliesFamilies





1 for each School(N = 1,800 School Teams)

School School Implementation Implementation

Team (N=4)Team (N=4)

School School Implementation Implementation

Team (N=4)Team (N=4)N = 7,200

N = 6

N = 288

Implementation-Skilled Workforce

N =

76 <

0.1%

$$

N = 70

Re

-Pu

rpo

se

““District” District” Implementation Implementation Teams (N=4)Teams (N=4)

““District” District” Implementation Implementation Teams (N=4)Teams (N=4)

1 for every group of 25 Schools (N = 72 “District” Teams)

Regional Regional Implementation Implementation Teams (N=5)Teams (N=5)

Regional Regional Implementation Implementation Teams (N=5)Teams (N=5)

1 for every group of 5 “Districts” (N = 14 Regional Teams)

1 for every 10 RITs(N = 1 State Teams)

State State Transformation Transformation


State State Transformation Transformation


Intensive Development

Saturation

State Capacity Development

0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10 11 12

Years

Reg

iona

l Im

pl. T

eam

s RITs

STTs

Intensive Development

Oregon Districts and Student Enrollment

ODE Report Card 2008-2009

Scale Up

To scale up interventions we must first scale up implementation capacity

Building implementation capacity is essential to maximizing the statewide use of EBPs and other innovations

Oregon needs

About 14 Regional Implementation Teams (and support staff)

One State Transformation Team

Annual cost about $8 million

About $40,000 per year per district (or $14 per year per student)

Scale Up

System Change

Innovative practices do not fare well in existing organizational structures and systems

An infrastructure for implementation does not exist

Organizational and system changes are essential to successful use of innovations

Legacy Systems A legacy system is a system or

application that continues to be used despite its poor competitiveness and compatibility with modern equivalents

Difficult to integrate new systems into legacy systems because it is a difficult and time intensive process to understand current system functionalities

Legacy methods create a huge conversion challenge for implementation teams

Ashok R. B. Samuel (2009)

EXISTING SYSTEM

EFFECTIVE INNOVATIONS

ARE CHANGED TO

FIT THE SYSTEM

EXISTING SYSTEM IS

CHANGED TO SUPPORT

THE EFFECTIVENESS OF

THE INNOVATION

EFFECTIVE INNOVATION

System Change

Compliance and Crises, Urgent, Time Sensitive!!• Services not meeting Standards• Deal with urgent and high profile issues

Best PracticesImplemented Fully With Good Outcomes

System Supports & Stability • Regulatory roles• Basic Data Systems• Financing and Fiscal Accountability• Accreditation/ Licensing Standards• HR rules and regulations• Safety Standards• Work with Legislature• Inclusion of Stakeholders

System Supports & Stability

Mandates, System Supports,

Foundational Polices & Regulations

Leadership Responsibilities and Leverage PointsLeadership Responsibilities and Leverage Points

Thanks to Tom Bellamy

Implementation Team

Management Team

TeachersInnovations

Students

Po

licy En

abled

P

ractice (PE

P)

Pra

ctic

e In

form

ed

Po

licy

(P

IP)

Sys

tem

C

han

ge

“Ext

ern

al”

Sys

tem

Ch

ang

e S

up

po

rt Adaptive Challenges

•RFP methods

• IHE curricula

• Salaries

• Funding

• Credentialing

• Licensing

• Time/ scheduling

• Union contracts

• Duplication

• Fragmentation

• Hiring criteria

• Federal/ State laws

SYSTEM ALIGNMENT

State Department

Districts

Schools

Teachers/ Staff

Effective Practices

AL

IGN

ME

NT

Federal Departments

Imp

lem

enta

tio

n T

eam

s

FORM SUPPORTS FUNCTION

The End in Mind With the purposeful use of

implementation science, we can:

Make statewide use of good instruction, evidence-based practices, and other innovations…

To produce increasingly effective outcomes for all students…

For the next 50 years.


Global Implementation Conference 2011 www.implementationconference.org

Integrate the science, practice, policy of implementation, organization change, and system transformation

Call for Applications

Do you know of an organization/coalition currently implementing an evidence based practice in their local community?

Is the implementation of this program/innovation producing beneficial outcomes to the community?

Are they a role model for moving the evidence-based practice from science to service?

If you answered “YES” to all of the above, check out: http://www.samhsa.gov/scienceandservice

Nominate your own organization or someone you know!!

Science and Service Award Program

For More InformationDean L. Fixsen, Ph.D.

919-966-3892

[email protected]

Karen A. Blase, Ph.D.

919-966-9050

[email protected]

Frank Porter Graham Child Development Institute

University of North Carolina

Chapel Hill, NC

http://nirn.fpg.unc.edu/

For More InformationState Implementation and Scaling up of Evidence-

based Practices (SISEP)

Dean Fixsen, Karen Blase, Rob Horner, George Sugai

www.scalingup.org

“Resources” Tab

Concept paper

Annotated bibliography

Data on scaling up

Scaling up Briefs

Evidence-based

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Download all or part of the monograph at:


Implementation Research: A Synthesis of the Literature

Thank You for your Support

Annie E. Casey Foundation (EBPs and cultural competence)

William T. Grant Foundation (implementation literature review)

Substance Abuse and Mental Health Services Administration (implementation strategies grants; national implementation awards)

Centers for Disease Control & Prevention (implementation research)

National Institute of Mental Health (research and training grants)

Juvenile Justice and Delinquency Prevention (program development and evaluation grants

Office of Special Education Programs (Scaling up Capacity Development Center)

Administration for Children and Families (Child Welfare Leadership Development)

Duke Endowment (Child Welfare Reform)

scaling up innovations dean l. fixsen, ph.d. & karen a. blase, ph.d. university of north...

Documents