國外藥學教育成效評量

林香汶 Hsiang-Wen (Margaret) Lin, Ph.D.

中國醫藥大學藥學系暨碩博士研究所、中國醫藥大學附設醫院藥劑部

Behavior learning objectives After completing this lecture, you will be

able to: Appreciate Bloom taxonomy Compare and contrast common educational

methods used in US pharmacy education Be familiar with the concept of assessment Compare and contrast common assessment

methods used in US pharmacy education

Bloom Taxonomy Knowledge: arrange, define, duplicate, label, list,

memorize, name, order, recognize, relate, recall, repeat, reproduce state.

Comprehension: classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, restate, review, select, translate,

Application: apply, choose, demonstrate, dramatize, employ, illustrate, interpret, operate, practice, schedule, sketch, solve, use, write.

Analysis: analyze, appraise, calculate, categorize, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test.

Synthesis: arrange, assemble, collect, compose, construct, create, design, develop, formulate, manage, organize, plan, prepare, propose, set up, write.

Evaluation: appraise, argue, assess, attach, choose compare, defend estimate, judge, predict, rate, core, select, support, value, evaluate.

Benjamin Bloom, 1956

Common educational methods used in US pharmacy education

Lecture Lecture-based cases Case-based lectures Problem-based learning Journal Club/ Paper critique Learning by doing (i.e., presentations) Experiential training Lab/Recitation More…

What are better reasons for assessment?

To improve courses To improve academic programs To improve teaching To enhance student learning

What is assessment?

Assessment the process of documenting, usually in

measurable terms, i.e., knowledge, skills, attitudes and beliefs

a component of any instructional system

Goals of assessment Provide direction and motivation for future learning, including knowledge,skills, and professionalism Protect the public by upholding high professional standards and screeningout trainees who are incompetent Meet public expectations of self-regulation Choose among applicants for advanced training

What to assess Habits of mind and behavior Acquisition and application of knowledge and skills Communication Professionalism Clinical reasoning and judgment in uncertain situations Teamwork Practice-based learning and improvement Systems-based practice

Principles of assessment (I)

Epstein RM, 2007

Principles of assessment (II) How to assess

Use multiple methods and a variety of environments and contexts to capture different aspects of performance

Organize assessments into repeated, ongoing, contextual, and developmental programs

Balance the use of complex, ambiguous real-life situations requiring reasoning and judgment with structured, simplified, and focused assessments of knowledge, skills, and behaviors

Include directly observed behavior Use experts to test expert judgment Use pass–fail standards that reflect appropriate developmental levels Provide timely feedback and mentoring

Epstein RM, 2007

Principles of assessment (III) Cautions

Be aware of the unintended effects of testing Avoid punishing expert examinees who use shortcuts Do not assume that quantitative data are more reliable, valid, or useful t

han qualitative data

Epstein RM, 2007

Type of assessment methods

The most important distinctions are: (1) formative vs. summative(2) objective vs. subjective(3) referencing (criterion-referenced, norm-

referenced)(4) informal and formal (5) Whether it is practical, valid and/or reliable

Common assessment methods (I) Written exercises

Multiple-choice questions single-best-answer extended matching format

Key-feature and script-concordance questions Short-answer questions Structured essays

Assessments by supervising clinicians Global ratings with comments at end of rotation Structured direct observation with checklists for ratings

(e.g., mini–clinical-evaluation exercise or video review) Oral examinations

Epstein RM, 2007

Common assessment methods (II) Clinical simulations

Standardized patients and objective structured clinical examinations (OSCEs)

Incognito standardized patients High-technology simulations

Multisource (“360-degree”) assessments Peer assessments Patient assessments Instructor assessments Self-assessments Portfolios

Epstein RM, 2007

Teaching triangulation

Student performance

Peer assessment Self assessment

Instructor assessment

Not only one answer…No right or wrong answer…

Objective structured clinical examinations (OSCEs)

a performance-based tool comprised of a series of stations through which

all examinees rotate on a timed basis. In each station, the examinee is faced with a

simulated task or problem the examinee is required to perform specific functions to

complete the task or address the problem each examinee in an interactive station is observed and

assessed by a trained examiner using a standardized marking key Austin Z, 2003

Type of OSCE stations Interactive: with

“standardized patients": actors who have been specially trained to portray patients with specific medical conditions or drug-related problems

“standardized clients”: actors or other health professionals who have been specially trained to portray allied health professionals in an interdisciplinary health care context

Non-interactive: written responses to tasks or problems and involve no direct

observation and assessment Austin Z, 2003

Ideal subject content for OSCEs

Communication and interpersonal skills Ethical and professional judgment Complex ethical problem identification and

resolution skills

Austin Z, 2003

Features of OSCEs

Both the process and the product are tested The scoring using both analytical and global rating

is objective Questions being asked in response stations are

always objective Simulations can be used for acute cases and there

is scope for immediate feedback Patient variability and examiner variability are

eliminatedAustin Z, 2003

Advantages of OSCEs

Covering a broad range of clinical skills much wider than a conventional examination

Integrating the teaching and evaluation, and variety maintains student's interest

Increasing faculty-student interaction Testing a large number of students within a

relatively short time

Austin Z, 2003

Disadvantages of OSCEs High costs Time consuming to develop Have difficulties associated with developing and

administering Risk of observer fatigue if the observer has to record the

performance of several candidates on lengthy check lists All stations must invariably demand only equal time Breaking clinical skills into individual competencies is

artificial and not meaningful

Austin Z, 2003

Assessment Instruments used in OSCEs

Need both analytical and global rating scales Analytical checklist

a series of performance-based observations Including specific, observable, and measurable tasks e.g., include asking a specific question related to allergies,

or providing specific information to the patient regarding side effects of a medication.

Global assessment capture the candidate’s performance in important skills su

ch as communication and integrative problem solving

Austin Z, 2003

Example of global assessment

Austin Z, 2003

Performance Reports of OSCEs

Austin Z, 2003

OSCEs related lectures in 2008 AACP conference

Section: Performance-based teaching, learning and assessmentChallenges and Opportunities in Effectively imple

menting performance-based AssessmentOSCEology 101: overview of OSCEHow to develop and implement OSCEs in your sch

ool

Development of an OSCE Blueprinting Station development Case writing Case review and validation Standard setting Piloting Assembly of OSCE

Austin Z, 2008 AACP handout

Blueprinting

Process of defining content domain and process skills to be assessed

Based upon outcomes, expectations, or standards Level: defines complexity of performance expected Range: describes scope of performance expected

Austin Z, 2008 AACP handout

Levels and ranges

Simple Patient

Simple Problem

Complex Patient

Simple Problem

Simple Patient

Complex Problem

Complex Patient

Complex Problem

Austin Z, 2008 AACP handout

Defining levels and ranges

Attributes of “complex patient” include: Psycho-social issues AffectLanguage barrier Physical abilities Cultural Issues Financial Issues

Austin Z, 2008 AACP handout

Defining levels and ranges

Attributes of “complex problem” include: Multiple disease states Multiple DRPsUnusual/rare medical conditions Unusual/rare medications

Austin Z, 2008 AACP handout

Levels and ranges Patient has a cough requiring self-medication and has no allergies, medical conditions or complicating factors

Patient with a language barrier has a cough requiring self-medication and has no allergies, or medical conditions

Patient has a cough and is taking ACE-inhibitors and a statin but also has a low-grade fever and muscle aches/pains

Patient with multiple disease states unable to swallow oral medications is experiencing a cough

Austin Z, 2008 AACP handout

“The minimally competent student”

A factious concept, but one based on reality May be an amalgam of various students Able to meet ALL performance standards/

expectations but at the most basic level

Austin Z, 2008 AACP handout

Reflections

Is it a good idea to go for the OSCEs in Taiwan? As for high stakes vs. low stakes assessment ? As for summative vs. formative assessment ?

Based on the pharmacy education environment in Taiwan, are we ready to get into this kind of performance-based assessments?

What kinds of education reform are needed in Taiwan to facilitate pharmacy students to get ready to be a practitioners?

… More

Assignment

To work individually or work as a group to appraise the assessment methods used in the following two articles

Am. J. Pharm. Educ 2006; 70(5) Article 117

JAMA 2007;298(9):1010-1022

References Epstein RM. Assessment in Medical Education, N Engl J Med 2007;356:387-96. Cisnerosa RM, Salisbury-Glennonb JD, Anderson-Harpera HM. Status of Problem-Based Lea

rning Research in Pharmacy Education: A Call for Future Research. Am. J. Pharm. Educ., 2000. 66, 19–26

Austin Z, Gregory P, Tabak D. Simulated Patients vs. Standardized Patients in Objective Structured Clinical Examinations. Am. J. Pharm. Educ 2006; 70 (5) Article 119.

Austin Z,O'Byrne C, Pugsley J, Munoz LQ. Development and Validation Processes for an Objective Structured Clinical Examination (OSCE) for Entry-to-Practice Certification in Pharmacy: The Canadian Experience. Am. J. Pharm. Educ., 2003; 67 (3) Article 76

Lin HW, Pickard AS, Mahady GB, Popovich, NG. The development of a measure to assess pharmacists’ knowledge of herbal and dietary supplements. Am. J. Pharm. Educ., 2008; 72(3): Article 59.

Windish DM, Huot SJ. Green ML. Medicine Residents’ Understanding of the Biostatistics and Results in the Medical Literature. JAMA. 2007;298(9):1010-1022.

Austin Z, Marini A, MacLeod Glover N, Tabak D,Peer-Mentoring Workshop for Continuous Professional Development. Am. J. Pharm. Educ 2006; 70(5) Article 117

Garavalia LS, Marken PA, Sommi RW. Selecting Appropriate Assessment Methods: Asking the Right Questions. Am. J. Pharm. Educ., 2002. 66, 108-112

Mackellar A,Ashcroft DM, Bell et al. Criteria for the Assessment of Pharmacy Students’ Communication Skills With Patients. Am. J. Pharm. Educ. 2007; 71 (3) Article 50.

Ananthakrishnan N. Objective structured clinical/practical examination (OSCE/OSPE). J Postgrad Med 1993;39:82-4

Thank your for your attention

Questions???