10/18/17
1
AssessingCriticalThinkingSkillsDAVIDK.KNOX,PH.D.,DIRECTOROFCLEMSONTHINKS2(CT2)
κριτικός (Kritikos), relatingtojudging,fitforjudging,skilledinjudging,abletodiscern (Plato, Plutarch, Lucian,others).Withthegenitiveoftheobject,forexample:κριτικὸς ἐνθυμήσεων καί ἐννοιῶν καρδίας- abletojudgethe thoughtsandintentions oftheheart.
Whatis”Critical”inCriticalThinking?
Attitude+Knowledge+ThinkingSkills=CriticalThinking
“Proficiencyinreading,writing,andarithmetichastraditionallybeentheentry-levelthresholdtothejobmarket,butthenewworkplacerequiresmorefromitsemployees.Employeesneedtothinkcritically,solveproblems,innovate,collaborate,andcommunicatemoreeffectively.”–– AmericanManagementAssociation,2010(quotedinDianeHalpern:ThoughtandKnowledge,page5.)
10/18/17
2
HartResearchAssociatesStudy:2009*• Intellectualandpracticalskills• Theabilitytocommunicateeffectively,orallyandinwriting(89%)• Criticalthinkingandanalyticalreasoningskills(81%)• Theabilitytoanalyzeandsolvecomplexproblems(75%)• Teamworkskillsandtheabilitytocollaboratewithothersindiversegroupsettings(71%)
• Theabilitytoinnovateandbecreative(70%)• Theabilitytolocate,organize,andevaluateinformationfrommultiple• sources(68%)• Theabilitytoworkwithnumbersandunderstandstatistics(63%)
*http://www.aacu.org/sites/default/files/files/LEAP/2009_EmployerSurvey.pdf
HartResearchAssociatesStudy:2009*
*http://www.aacu.org/sites/default/files/files/LEAP/2009_EmployerSurvey.pdf
CriticalThinking,Defined“Criticalthinkingistheprocessofpurposeful,reflective
judgment.Criticalthinkingmanifestsitselfingivingreasonedandfair-mindedconsiderationtoevidence,conceptualizations,methods,contexts,andstandards inordertodecidewhatto
believeorwhattodo.”(Facione,2010).“Theintellectuallydisciplinedprocessofactively andskillfullyconceptualizing,applying,analyzing,synthesizing,and/orevaluatinginformationgatheredfrom,orgeneratedby,
observation,experience,reflection,reasoning,orcommunication,asaguidetobeliefandaction”(ScrivenandPaul,1987).
10/18/17
3
CriticalThinking,Defined“Criticalthinkingistheuseofthosecognitiveskillsorstrategiesthatincreasetheprobabilityofadesiredoutcome.Itisusedtodescribethinkingthatispurposeful,reasoned,andgoal
directed– thekindofthinkinginvolvedinsolvingproblems,formulatinginferences,calculatinglikelihoods,andmakingdecisions,whenthethinkerisusingskillsthatare
thoughtfulandeffectivefortheparticularcontextandtypeofthinkingtask.”(Halpern,2014)
“Beingacriticalthinkerinvolvesmorethancognitiveactivitiessuchaslogicalreasoningorscrutinizingargumentsforassertionsunsupportedbyempiricalevidence.Thinkingcriticallyinvolvesourrecognizingtheassumptionsunderlyingourbeliefsandbehaviors.Itmeanswecangivejustificationsforourideasandactions.Mostimportant,perhaps,itmeanswetrytojudgetherationalityofthesejustifications.Wecandothisbycomparingthemtoarangeofvaryinginterpretationsandperspectives.Wecanthinkthrough,project,andanticipatetheconsequencesofthoseactionsthatarebasedonthesejustifications.Andwecantesttheaccuracyandrationalityofthesejustificationsagainstsomekindofobjectiveanalysisofthe
"real"worldasweunderstandit.”(Brookfield,1987)
CriticalThinking,Defined“Itisthemarkofaneducatedmindtobeabletoentertainathoughtwithoutacceptingit.”―Aristotle,Metaphysics“Allthatweareistheresultofwhatwehavethought:itisfoundedonourthoughtsandmadeupofourthoughts.”–– Buddha,Dhammapada“Learningwithoutthoughtislaborlost;thoughtwithoutlearningisperilous”–– Confucius,Analects“Readnottocontradictandconfute;nortobelieveandtakeforgranted;nortofindtalkanddiscourse;buttoweighandconsider”–– FrancisBacon,“OfStudies”
CriticalThinkingMindset:EagernessandSkills“Apersonwithastrongdisposition towardcriticalthinkinghastheconsistentinternalmotivationtoengageproblemsandmakedecisionsbycriticalthinking.Operationallythismeansthreethings:Thepersonconsistentlyvalues criticalthinking,believes thatusingcriticalthinkingskillsoffersthegreatestpromiseforreachinggoodjudgments,andintends toapproachproblemsanddecisionsbyapplyingcriticalthinkingskillsasbestasheorshecan.Thiscombinationofvalues,beliefs,andintentionsformsthehabitsofmindthatdisposethepersontowardcriticalthinking.”–– PeterFacione,ThinkCritically,Chapter2.1
10/18/17
4
CriticalThinkingMindset:EagernessandSkills“Everyonethinks;itisournaturetodoso.Butmuchofourthinking,lefttoitself,isbiased,distorted,partial,uninformed,ordown-rightprejudiced.Yetthequalityofourlifeandthatofwhatweproduce,make,orbuilddependspreciselyonthequalityofourthought.Shoddythinkingiscostly,bothinmoneyandinqualityoflife.Excellenceinthought,however,mustbesystematicallycultivated”–– RichardPaulandLindaElder,CriticalThinkingConceptsandTools,pg.2
CriticalThinkingMindset:EagernessandSkillsUniversalIntellectualStandards:Clarity: Couldyouelaboratefurtheronthatpoint?Couldyouexpressthatpointinanotherway?Couldyougivemeanillustration?”Couldyougivemeanexample?Accuracy: Isthatreallytrue?Howcouldwecheckthat?Howcouldwefindoutifthatistrue?Precision: Couldyougivememoredetails?Couldyoubemorespecific?Relevance: Howisthatconnectedtothequestion?Howdoesthatbearontheissue?
CriticalThinkingMindset:EagernessandSkillsUniversalIntellectualStandards(cont’d)
Depth: Howdoesyouransweraddressthecomplexitiesinthequestion?Howareyoutakingintoaccounttheproblemsinthequestion?Areyoudealingwiththemostsignificantfactors?Breadth: Doweneedtoconsideranotherpointofview?Isthereanotherwaytolookatthisquestion?WhatwouldthislooklikefromthepointofviewofX?Logic: Doesthisreallymakesense?Doesthisfollowfromwhatyousaid?Howdoesthatfollow?Fairness: Areweconsideringallrelevantviewpointsingoodfaith?Arewedistortingsomeinformationtomaintainourbiasedperspective?Arewemoreinterestedinourvestedintereststhanthecommongood?–– RichardPaulandLindaElder,CriticalThinkingConceptsandTools,pgs.8,9
10/18/17
5
CT2 PointsofAssessment:Classes•PreandPostCaliforniaCriticalThinkingSkillsTests
•CTArtifacts
•AdditionalquestionsforCT2 classesonEvaluationofInstructorforms
•TransferrableSkillsNSSEmodule
•ETSProficiencyProfile(FreshmenandSeniors)
•ClemsonEducationalProfile
CT2 PointsofAssessment:
CriticalThinkingTestsHigherEducation:General-Content,Multi-Aspect*ACTCAAPTestModule:CriticalThinking. .http://www.act.org/caap/index.html (retiringinJanuary2018)
TheCaliforniaCriticalThinkingSkillsTest(CCTST): CollegeLevel.http://www.insightassessment.com
TheCaliforniaCriticalThinkingDispositionsInventory (CCTDI). http://www.insightassessment.com
CambridgeThinkingSkillsAssessment.CambridgeAssessmenthttp://www.admissionstests.cambridgeassessment.org.uk/adt/tsacambridge/about
CollegiateLearningAssessment+ (CLA+).http://www.collegiatelearningassessment.org/
TheCriticalThinkingAssessmentTest (CAT).http://www.tntech.edu/cat/using/CornellCriticalThinkingTest,LevelZ.https://www.criticalthinking.com/cornell-critical-thinking-test-level-z.html
ETS® ProficiencyProfile. http://www.ets.org/proficiencyprofile/about
*Adaptedfrom:ANANNOTATEDLISTOFENGLISH-LANGUAGECRITICALTHINKINGTESTS: RobertH.EnnisandGeoffreyScottChattin,UniversityofIllinoisUC
10/18/17
6
CriticalThinkingTestsHigherEducation:General-Content,Multi-Aspect*HalpernCriticalThinkingAssessment(HCTA)https://sites.google.com/site/dianehalperncmc/home/research/halpern-critical-thinking-assessment
HolisticCriticalThinkingScoringRubric(HCTSR).http://www.insightassessment.com/
InternationalCriticalThinkingBasicConcepts&UnderstandingOnlineTest:AssessingInitialUnderstandingofBasicCriticalThinkingConceptsandPrinciples.http://www.criticalthinking.org/pages/online-critical-thinking-basic-concepts-test/679
InternationalCriticalThinkingEssayTest.http://www.criticalthinking.org/store/products/international-critical-thinking-test/185
JamesMadisonTestofCriticalThinking(partoftheJamesMadisoncriticalthinkingcourse),https://www.criticalthinking.com/james-madison-critical-thinking-course-student-book.html.
NewJerseyTestofReasoningSkills.http://cehs.montclair.edu/academic/iapc/
OxfordThinkingSkillsAssessment.CambridgeAssessment.http://www.admissionstests.cambridgeassessment.org.uk/adt/tsaoxford/about
Watson-GlaserIICriticalThinkingAppraisal.https://us.talentlens.com/watson-glaser-critical-thinking-test
*Adaptedfrom:ANANNOTATEDLISTOFENGLISH-LANGUAGECRITICALTHINKINGTESTS: RobertH.EnnisandGeoffreyScottChattin,UniversityofIllinoisUC
CriticalThinkingTestsTheCaliforniaCriticalThinkingSkillsTest(CCTST)TheCaliforniaCriticalThinkingSkillsTest(CCTST)isanobjectivemeasureofthecorereasoningskillsneededforreflectivedecisionmakingconcerningwhattobelieveorwhattodo.TheCCTSTisdesignedtoengagethetest-taker’sreasoningskills.Multiplechoiceitemsuseeverydayscenarios,appropriatetotheintendedtest-takergroup.Eachitemrequiresthatthetest-takermakeanaccurateandcompleteinterpretationofthequestion. Anyspecializedinformationneededtorespondcorrectlyisprovidedinthequestionitself.Thetestitemsrangeindifficultyandcomplexity.Differentquestionsprogressivelyinvitetest-takerstoanalyzeortointerpretinformationpresentedintext,charts,orimages;todrawaccurateandwarrantedinferences;toevaluateinferencesandexplainwhytheyrepresentstrongreasoningorweakreasoning;ortoexplainwhyagivenevaluationofaninferenceisstrongorweak.Theinstrumentistypicallyadministeredin45-50minutes;thelengthoftheinstrumentissettopermitmaximumperformancewithintherangeofpossibleeffortfortheintendedtest-takergroup.
CriticalThinkingTestsTheCaliforniaCriticalThinkingSkillsTest(CCTST)ScaleDescriptions
ReasoningSkills- Overall:
TheReasoningSkillsOveralldescribesoverallstrengthinusingreasoningtoformreflectivejudgmentsaboutwhattobelieve or whattodo.Toscorewelloverall,thetest-takermustexcelinthesustained,focusedandintegratedapplicationofcorereasoningskillsincludinganalysis,interpretation,inference,evaluation,explanation,inductionanddeduction.The Overallscorepredictsthecapacityforsuccessineducationalorworkplacesettingswhichdemandreasoneddecisionmakingandthoughtfulproblemsolving.
Analysis:
Analyticalreasoningskillsenablepeopletoidentifyassumptions,reasonsandclaims,andtoexaminehowtheyinteractintheformationofarguments. Weuseanalysistogatherinformationfromcharts,graphs,diagrams,spokenlanguageanddocuments. Peoplewithstronganalyticalskillsattendto patternsandtodetails.Theyidentifytheelementsofasituationanddeterminehowthosepartsinteract. Stronginterpretationskillscansupporthighqualityanalysisbyprovidinginsightsintothesignificanceofwhatapersonissayingorwhatsomethingmeans.
Inference:
Inferenceskillsenableustodrawconclusionsfromreasonsandevidence.Weuseinferencewhenweofferthoughtfulsuggestionsandhypotheses. Inferenceskillsindicatethenecessaryortheveryprobableconsequencesofagivensetoffactsandconditions. Conclusions,hypotheses,recommendationsordecisionsthatarebasedonfaultyanalyses,misinformation,baddataorbiasedevaluationscanturnouttobemistaken,eveniftheyhavebeenreachedusingexcellentinferenceskills.
10/18/17
7
CriticalThinkingTestsTheCaliforniaCriticalThinkingSkillsTest(CCTST)ScaleDescriptions,Continued
Evaluation:
Evaluativereasoningskillsenableustoassessthecredibilityofsourcesofinformationandtheclaimstheymake. And,weuse theseskillstodeterminethestrengthorweaknessofarguments. Applyingevaluationskillswecanjudgethequalityofanalyses,interpretations,explanations,inferences,options,opinions,beliefs,ideas,proposals,anddecisions. Strongexplanationskillscansupporthighqualityevaluationbyprovidingtheevidence,reasons,methods,criteria,orassumptionsbehindtheclaimsmadeandtheconclusionsreached.
Deduction:
Decisionmakinginpreciselydefinedcontextswhererules,operatingconditions,corebeliefs,values,policies,principles,proceduresandterminologycompletelydeterminetheoutcomedependsonstrongdeductivereasoningskills.Deductivereasoningmoveswithexactingprecisionfromtheassumedtruthofasetofbeliefstoaconclusionwhichcannotbefalseifthosebeliefsaretrue.Deductivevalidityisrigorouslylogicalandclear-cut. Deductivevalidityleavesno roomforuncertainty,unlessonealtersthemeaningsofwordsorthegrammarofthelanguage.
Induction:
Decisionmakingincontextsofuncertaintyreliesoninductivereasoning. Weuseinductivereasoningskillswhenwedrawinferencesaboutwhatwethinkmustprobablybetruebasedonanalogies,casestudies,priorexperience,statisticalanalyses,simulations,hypotheticals,andfamiliarcircumstancesandpatternsofbehavior. Aslongasthereisthepossibility,howeverremote,thatahighlyprobableconclusionmightbemistaken,thereasoningisinductive.Althoughit doesnotyieldcertainty,inductivereasoningcanprovideasolidbasisforconfidenceinourconclusions.
10/18/17
8
Pre-Test
Post-Test
CoreCriticalThinkingSkillsMeasuredbytheCCTST
CriticalThinkingTestsTheCriticalThinkingAssessmentTest(CAT)TheCATinstrument,developedatTennesseeTech,isauniquetooldesignedtoassessandpromotetheimprovementofcriticalthinkingandreal-worldproblemsolvingskills.Theinstrumentistheproductofextensivedevelopment,testing,andrefinementwithabroadrangeofinstitutions,faculty,andstudentsacrossthecountry.TheNationalScienceFoundationhasprovidedsupportformanyoftheseactivities.TheCATinstrumentisdesignedtoassessabroadrangeofskillsthatfacultyacrossthecountryfeelareimportantcomponentsofcriticalthinkingandrealworldproblemsolving.Thetestwasdesignedtobeinterestingandengagingforstudents.Allofthequestionsarederivedfromrealworldsituations.Mostofthequestionsrequireshortansweressayresponses,andadetailedscoringguidehelpsensuregoodscoringreliability.
10/18/17
9
CriticalThinkingTestsTheCriticalThinkingAssessmentTest(CAT)
SkillsAssessedbyCATInstrument
EvaluatingInformation
•Separatefactualinformationfrominferences.
•Interpretnumericalrelationshipsingraphs.
•Understandthelimitationsofcorrelationaldata.
•Evaluateevidenceandidentifyinappropriateconclusions.
CreativeThinking
•Identifyalternativeinterpretationsfordataorobservations.
•Identifynewinformationthatmightsupportor
contradictahypothesis.
•Explainhownewinformationcanchangeaproblem.
LearningandProblemSolving
•Separaterelevantfromirrelevantinformation.
•Integrateinformationtosolveproblems.
•Learnandapplynewinformation.
•Usemathematicalskillstosolvereal-worldproblems.
•Communication
•Communicateideaseffectively.
CriticalThinkingTestsResponseInteractingwithpeoplefromacrosscampusInteractionwithcolleaguesandfellowfacultymembers.Thecollegialityandopportunitytoworkwithpeoplefromoutsideofmydiscipline.Theconversationswehadatourtable,bothformalandinformal.InteractingwithotherfacultyfromthisgroupTheinteractionwiththeotherfacultymembersmadethescoringsessionsquiteenjoyable.Thecamaraderieandtheintellectualdiscussions
Thediscussionsatthetableaboutthescoring,etc.Talkingabouthowtoapplytherubricinparticularcaseswas,attimes,eye-opening.
CollegialityamongalltheparticipantsTheopportunitytoexchangeideaswithothercolleagues.Meetingotherfacultyfromotherdepartments.Interactionwithotherlike-mindedCT2faculty.InteractionwithfacultywhilesharingacommongoalThecamaraderieandthequestions.interactionwithfacultyfromvariousdisciplinesEngagingindialoguetoreachconsensuswithcolleagues
Q.WhatdidyoulikemostabouttheCATTestScoringSession?
CriticalThinkingTests
10/18/17
10
Assessments:CCTST vs.CAT• CaliforniaCriticalThinkingSkillsTest(CCTST)• 34multiplechoicequestions• 45minutes• Onlineorpaper• Gradedimmediately(online)• Multi-LingualVersions• $10pertest
**Studentsaretestedduringthefirstandfinalweeksofclasses
• CriticalThinkingAssessmentTest(CAT)• Essay:15questions• Onehour• Paperandpencil• Facultygraded• Facultydevelopmentpotential• $9.95pertestand$300annualfee
Assessments:ETS®ProficiencyProfileAstheonlyintegratedtestofgeneraleducationskills,theETSProficiencytestassessesfourcoreskillareas- criticalthinking,reading,writingandmathematics- inasingletestthatthe VoluntarySystemofAccountability(VSA)hasselectedasagaugeofgeneraleducationoutcomes.
ClemsonUniversity usesthe ETSProficiencyProfile testto:• Functionas“bookends”intheCT2 assessmentprocess• measureanddocumentprogrameffectivenesstomeetrequirementsforaccreditationandprogramfunding• assessstudentproficiencyincoreacademicskillareastoidentifystrengths,weaknessesandopportunitiestoimprovecurriculumand
instruction• compareperformanceagainstprogramsatmorethan380institutionsnationwide• conductbenchmarkandtrendanalysestoevaluateprogramimprovementeffortsandoveralllearningoutcomes
ClemsonUniversitybeganinFall2011toadministertheETSProficiencyProfiletoallincomingfreshmen.Thesesamestudentsaretestedwhentheyareseniorsandwewillbeabletolongitudinallytracktheirprogress.Thisprovidesuswithalevelofinsightintothe“value-added” aspectofaClemsoneducationneverpossiblebefore.
Assessments:ETS®ProficiencyProfileETSProficiencyProfileStandardForm
TheStandardformoftheETSProficiencyProfiletestisintendedtoprovideinformationaboutindividualstudentsaswellasgroupsofstudents.Itconsistsof108questions,dividedintotwosectionsof54questionseach.Thetwosectionsmaybeadministeredeitherinasingletwo-hourtestingsessionorintwoseparate,one-hourtestingsessions.TheStandardformincludes:• 27questionstestingcriticalthinkingskills• 27questionstestingreadingskills• 27questionstestingwritingskills• 27questionstestingmathematicsskills
Additionally,studentstakingtheStandardformcanearna CertificateofAchievement basedonhowwelltheyperform.
10/18/17
11
Assessments:ETS®ProficiencyProfileETSProficiencyProfileAbbreviatedForm
TheAbbreviatedFormoftheETSProficiencyProfiletestisintendedtoprovideinformationaboutgroupsofatleast30students.Itisadministeredinasingle40-minutetestingsession.TheAbbreviatedFormdoesnotprovideinformationaboutindividualstudents.
TheAbbreviatedformiscreatedbydividingtheStandardFormintothreesmallerformsof36questionseach.ThethreeAbbreviatedFormsarepackagedinalternatingsequence,sothateachversionistakenbyone-thirdofthestudents.The108questionsintheStandardFormareassignedtothethreeAbbreviatedFormstomakeeachofthemaminiatureversionoftheStandardForm.
EachofthethreeAbbreviatedFormsincludes:• Ninequestionsmeasuringcriticalthinkingskills• Ninequestionsmeasuringreadingskills• Ninequestionsmeasuringwritingskills• Ninequestionsmeasuringmathematicsskills
Artifacts• Defined:ar·ti·fact noun \ˈär-ti-ˌfakt\ a: somethingcreatedbyhumansusuallyforapracticalpurpose;especially : anobjectremainingfromaparticularperiod.
• Eachcoursesubmitsatleastonetypeofartifact
• Cansubmitdifferentartifacts
• Manyclasseshaveaspecificassignmentthatwillbe
designatedthecriticalthinkingartifact.
• Randomsamplefromclassisscored
• ScoringisperformedbyCT2 FacultyScholarsusingthe
AAC&UCriticalThinkingrubric
• Examples:• Essay• Project• Critique• Video• PosterPresentation• Portfolio
Artifacts:AAC&UCriticalThinkingVALUERubric
10/18/17
12
Artifacts:“VisualEssay”
ThankYou!Pleasevisitourwebsiteforfurtherinformation:
http://www.clemson.edu/thinks2
2016ClemsonThinks2 FacultyInstitute