results drive accountability ppt 8-27-12 final

1National Center on Educational

Outcomes

National Center on Educational Outcomes (NCEO)

August 27, 2012

Using Assessment Data as Part of a Results-Driven Accountability System

Input from the NCEO Core Teamand

Sample Approaches


Outcomes

1. Summary of NCEO Core Team Report2. Clarifying Questions3. Overview of Sample Approaches4. Discussion

.

Overview


Outcomes

Core Team

• Peggy Carr, National Center for Education Statistics

• Alan Coulter, Data Accountability Center

• Candace Cortiella, The Advocacy Institute

• David Egnor, Office of Special Education Programs

• Jack Fletcher, University of Houston

• Lynn Fuchs, Vanderbilt University

• Brian Gong, Center for Assessment

• Colleen Riley, Kansas Department of Education


Outcomes

Resource Group

•Rolf Blank, Council of Chief State School Officers

•Anne Chartrand, Sourtheast RRC

•Karen Denbroeder, Florida Department of Education

• Judy Elliott, Consultant

•David Francis, University of Houston

•Michael Kolen, University of Iowa

•Elizabeth Kozleski, University of Kansas

•Rachel Quenemoen, Nat’l Center State Collaborative


Outcomes

Three major sections:

Input of the NCEO Core Team

1. Framing Considerations2. Core Team Suggestions3. Example Reporting Format


Outcomes

1. Public transparency and understandability are critical features of a results-driven accountability system and must be reflected in measures used to review states on student performance.

2. Multiple measures must be included. No single measure should be used in making decisions about student performance results.

3. The use of measures of student performance should provide appropriate incentives to states, particularly in relation to identified values (e.g., inclusion in the general assessment).

Framing Considerations


Outcomes

4. The measures should provide a flag to look deeper into areas that need improvement.

5. A plan should be developed and steps taken to monitor, validate, and improve the use of measures by OSEP and others; additional variables may be appropriate to enhance the measures in the future.



Outcomes

6. Variables that may be related to student performance but that have inconsistent interpretations and reliability should not be included in measures that are used for reviewing states on the performance of their students with disabilities.



Outcomes

7. No increased burden on states to collect additional data should result from the shift to reviewing student performance results. The developed measures need to fit within what states are doing as they review districts, and should be compatible with and reflective of the state’s overall accountability system used for school improvement.



Outcomes

1.Use a reporting format that ensures that multiple measures are considered for students with disabilities receiving special education services.2.Provide data for reading and mathematics separately.3.Include participation of students with disabilities in state assessments.

Core Team Suggestions


Outcomes

4.Include participation of students with disabilities in the general state assessment.5.Include performance of students with disabilities on the general state assessment.6.Include the relative difficulty of state assessments.7.Include the gap in general assessment performance between students with disabilities and students without disabilities.



Outcomes

8.Include improvement in performance over time.


These 8 suggestions guided the development of a set of 6 tables to display the data.


Outcomes

Table 1: Reading General State Assessment

Example Reporting Format


Outcomes

Table 2: Mathematics General State Assessment



Outcomes

Table 3: Reading and Math Overall Performance and Targets



Outcomes

Table 4: Reading Alternate Assessments



Outcomes

Table 5: Mathematics Alternate Assessments



Outcomes

Table 6: Participation Rates for Students with Disabilities in Reading and Mathematics Assessments



Outcomes

Two Sample Approaches:

Sample Approaches for Using Assessment Data

1. Decision Matrix (with 3 options)

2. Decision-Making Steps


Outcomes

Important Note

The sample approaches include possible thresholds for deciding whether a state exceeds, meets, or does not meet expectations. OSEP and stakeholders should discuss/consider whether adjustments to these example thresholds are needed.

Stakeholders, experts, and OSEP will need to be involved in determining appropriate thresholds for any elements that are used in reviewing assessment results.


Outcomes

Decision Matrix (Includes State Proficiency Target)

Sample Approach 1a

• Reading and math combined• Data from Core Team Tables 1-3

• Element 1: Participation in general assessment• Element 2: Improvement in percent proficient• Element 3: Gap in proficiency between students

with disabilities and students without disabilities• Element 4: Percent proficient or above• Element 5: Gap in proficiency target and actual


Outcomes

Sample Approach 1a


Outcomes

Benefits (Pros) and Challenges (Cons)

Sample Approach 1a

Benefits (Pros) Challenges (Cons)• Combines variables that are

difficult to look at separately• Easy to see where state falls

across two content areas

• Complex and may lack transparency

• Each element has issues that need to be considered


Outcomes

Decision Matrix (Without State Proficiency Target)

Sample Approach 1b

• Reading and math combined• Data from Core Team Tables 1-2

• Element 1: Participation in general assessment• Element 2: Improvement in percent proficient• Element 3: Gap in proficiency between students

with disabilities and students without disabilities• Element 4: Percent proficient or above


Outcomes

Sample Approach 1b


Outcomes


Outcomes


Sample Approach 1b



across two content areas




Outcomes

Decision Matrix (Without State Proficiency Target)

Sample Approach 1c

• Reading and math combined• Data from Core Team Tables 1-2, with

additional alternate assessment data• Elements 1-4 same as for Approaches 1a and 1b• Element 5: Gap between percent proficient on

general state assessment and percent proficient on AA-AAS


Outcomes

Detailed information on this approach is not included because additional data are needed to make calculations.

A decision matrix approach similar to that in Sample Approaches 1a and 1b would be used.

Sample Approach 1c


Outcomes


Sample Approach 1c



across two content areas• Explicitly includes students in

the AA-AAS




Outcomes

Decision-Making Steps

Sample Approach 2

• Reading and math separate• Four steps

• Participation in general assessment• Gap in performance between students with

disabilities and students without disabilities• Proficient rates on general assessment in

relation to the difficulty of the state’s assessment• Alternate assessment participation and

performance


Outcomes

Sample Approach 2

Decision-Making Recording Sheet


Outcomes

Sample Approach 2


Outcomes


Sample Approach 2

Benefits (Pros) Challenges (Cons)• Considers all examples of

reporting tables by Core Team

• Provides greater transparency than some approaches

• Allows for adjustment of steps and variable for policy shifts

• Must be completed one state at a time

• More subjective than some approaches

• Comparability is a challenge