aascu academic affairs summer meeting portland oregon ▪ july 28-31, 2011 christine m keller vsa...

17
Using the Voluntary System of Accountability to Measure Learning Outcomes Issues and Opportunities AASCU Academic Affairs Summer Meeting Portland Oregon July 28-31, 2011 Christine M Keller VSA Executive Director

Upload: gary-powers

Post on 17-Jan-2016

214 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

Using the Voluntary System of Accountability to Measure Learning Outcomes

Issues and Opportunities

AASCU Academic Affairs Summer Meeting

Portland Oregon ▪ July 28-31, 2011

Christine M Keller VSA Executive Director

Page 2: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

VOLUNTARY SYSTEM OF ACCOUNTABILITY (VSA)

on the undergraduate student experience through a common web report – the College Portrait.

Initiative by public 4-year universities to supply comparable, transparent information

AASCU & APLUwww.colle

geportrai

ts.org

Page 3: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

VSA GOALS• Provide a streamlined college search tool

for prospective students, families, and high school counselors

• Provide a mechanism for public institutions to demonstrate transparency and accountability

• Support institutions in the measurement and reporting of student learning outcomes through original research and by providing a forum for collaboration and exchange

Page 4: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

SLO PILOT PROJECT• 2007-08: Directly measure, publicly

report student learning gains at institution level using a common method– Skills: critical thinking, analytic reasoning,

problem solving, written communication

– Tests: CAAP, CLA, ETS Proficiency Profile

– Reported: At, Above, Below Expected

• 2012: 4-year trial period ends

• 2011-12: Evaluate what works, what doesn’t work, next steps

Page 5: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director
Page 6: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

EVALUATION FRAMEWORK• SLO measurement and reporting will

continue to be essential element of VSA

• How can the section be improved to better meet the needs of participating institutions and external stakeholders within the current environment?

Page 7: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

EVALUATION ACTIVITIES• Interviews with key internal and

external stakeholders– National Institute for Learning Outcomes

Assessment (NILOA)

• Focus Groups, Conference Sessions• Surveys• Advisory panels, working groups

Page 8: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

CLA N=80

ETS PP

N=18

CAAPN=11

• 34% or 109 College Portraits have published SLO results as of July 2011

SLO PUBLISHED ON CP BY TEST

Page 9: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

VSA SLO SURVEY• Short survey to gauge progress,

gather feedback from VSA contacts

• ~200 responses in June/July

– 61% institutional research

– 16% academic affairs

– 13% assessment

– 10% other

Page 10: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

SURVEY: TEST ADMINISTRATION• 83% administered at least one test

– 23% CAAP– 37% ETS PP– 51% CLA

• 60% posted results on College Portrait

• 92% plan to continue using one of tests

Page 11: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

VALUE OF VSA APPROACHHelpful as overall indicator of student progress … powerful stimulus to other more local efforts

Benchmarks to compare skill levels of our students with other campuses

Good faith effort to be transparent

Selected Comments

VSA SLO Survey 7/5/2011

Dovetails with institutional priorities

Page 12: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

VALUE OF VSA APPROACHLack of faculty buy in … small sample sizes provide limited value at operational level

Major limitation of VSA, please change this section … add AAC&U rubrics Selected Comments

VSA SLO Survey 7/5/2011

Testing protocol is negative because of cost (cash/staff time) with no useful results

Unfortunately I don’t think many people look at this section

Page 13: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

SURVEY: OTHER ASSESSMENTSUsed Effective

AAC&U VALUE Rubrics 39% (52 of 135)

53% (43 of 80)

Electronic Portfolios 68% (105 of 155)

58% (71 of 122)

ETS Major Field Test 61% (90 of 147)

64% (68 of 102)

Professional Licensure/Certification

93% (161 of 173)

87% (135 of 155)

Page 14: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

NEXT STEPS• Gather feedback on efficacy of the SLO

pilot study (Summer/Fall 2011)

• Assemble brainstorming group to consider learning outcomes in current context (Fall 2011)

• Convene technical workgroup to evaluate options (early Spring 2012)

• Work with senior advisory panel and VSA Board to recommend next steps and future directions (late Spring 2012)

Page 15: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

QUESTIONS & DISCUSSION

• Questions about evaluation plan or current status of VSA/SLO

• Your input on strengths, weaknesses, recommendations for future for student learning measurement and reporting within the VSA

Page 16: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

DISCUSSION• What are the benefits of the current

VSA approach to measuring/reporting SLO? – Has it been effective with external

stakeholder groups?

• What are some of the challenges or barriers of the current approach?

• How can the current approach be expanded or modified to better address current needs, challenges, external demands, etc?

Page 17: AASCU Academic Affairs Summer Meeting Portland Oregon ▪ July 28-31, 2011 Christine M Keller VSA Executive Director

MORE INFORMATION

Websites www.voluntarysystem.org

www.collegeportraits.org

Christine Keller, VSA Executive Director [email protected]

Wendell Hall, VSA Assistant Director [email protected]

Elspeth Payne, VSA Project Coordinator [email protected]