Using the Voluntary System of Accountability to Measure Learning OutcomesIssues and Opportunities

AASCU Academic Affairs Summer Meeting

Portland Oregon ▪ July 28-31, 2011

Christine M Keller VSA Executive Director

VOLUNTARY SYSTEM OF ACCOUNTABILITY (VSA)

on the undergraduate student experience through a common web report – the College Portrait.

Initiative by public 4-year universities to supply comparable, transparent information

AASCU & APLUwww.collegeportr

ai

ts.org

VSA GOALS• Provide a streamlined college search tool

for prospective students, families, and high school counselors

• Provide a mechanism for public institutions to demonstrate transparency and accountability

• Support institutions in the measurement and reporting of student learning outcomes through original research and by providing a forum for collaboration and exchange

SLO PILOT PROJECT• 2007-08: Directly measure, publicly

report student learning gains at institution level using a common method– Skills: critical thinking, analytic reasoning,

problem solving, written communication– Tests: CAAP, CLA, ETS Proficiency Profile– Reported: At, Above, Below Expected

• 2012: 4-year trial period ends• 2011-12: Evaluate what works, what

doesn’t work, next steps

EVALUATION FRAMEWORK• SLO measurement and reporting will

continue to be essential element of VSA

• How can the section be improved to better meet the needs of participating institutions and external stakeholders within the current environment?

EVALUATION ACTIVITIES• Interviews with key internal and

external stakeholders– National Institute for Learning Outcomes

Assessment (NILOA)• Focus Groups, Conference Sessions• Surveys• Advisory panels, working groups

CLA N=80

ETS PP

N=18

CAAPN=11

• 34% or 109 College Portraits have published SLO results as of July 2011

SLO PUBLISHED ON CP BY TEST

VSA SLO SURVEY• Short survey to gauge progress,

gather feedback from VSA contacts• ~200 responses in June/July

– 61% institutional research– 16% academic affairs– 13% assessment– 10% other

SURVEY: TEST ADMINISTRATION• 83% administered at least one test

– 23% CAAP– 37% ETS PP– 51% CLA

• 60% posted results on College Portrait

• 92% plan to continue using one of tests

VALUE OF VSA APPROACHHelpful as overall indicator of student progress … powerful stimulus to other more local efforts

Benchmarks to compare skill levels of our students with other campuses

Good faith effort to be transparent

Selected Comments VSA SLO Survey 7/5/2011

Dovetails with institutional priorities

VALUE OF VSA APPROACHLack of faculty buy in … small sample sizes provide limited value at operational level

Major limitation of VSA, please change this section … add AAC&U rubrics Selected Comments

VSA SLO Survey 7/5/2011

Testing protocol is negative because of cost (cash/staff time) with no useful results

Unfortunately I don’t think many people look at this section

SURVEY: OTHER ASSESSMENTSUsed Effective

AAC&U VALUE Rubrics 39% (52 of 135)

53% (43 of 80)

Electronic Portfolios 68% (105 of 155)

58% (71 of 122)

ETS Major Field Test 61% (90 of 147)

64% (68 of 102)

Professional Licensure/Certification

93% (161 of 173)

87% (135 of 155)

NEXT STEPS• Gather feedback on efficacy of the SLO

pilot study (Summer/Fall 2011) • Assemble brainstorming group to

consider learning outcomes in current context (Fall 2011)

• Convene technical workgroup to evaluate options (early Spring 2012)

• Work with senior advisory panel and VSA Board to recommend next steps and future directions (late Spring 2012)

QUESTIONS & DISCUSSION• Questions about evaluation plan or

current status of VSA/SLO• Your input on strengths, weaknesses,

recommendations for future for student learning measurement and reporting within the VSA

DISCUSSION• What are the benefits of the current

VSA approach to measuring/reporting SLO? – Has it been effective with external

stakeholder groups?• What are some of the challenges or

barriers of the current approach?• How can the current approach be

expanded or modified to better address current needs, challenges, external demands, etc?

MORE INFORMATIONWebsites

www.voluntarysystem.orgwww.collegeportraits.org

Christine Keller, VSA Executive Director ckeller@aplu.org

Wendell Hall, VSA Assistant Director whall@aplu.org

Elspeth Payne, VSA Project Coordinator epayne@aplu.org