© NAC / MCC 2013

André F. De Champlain, PhD, Acting Director of Research & Development

Paper presented at the 2013 IPAC Conference

Can a Standardized Set of Assessments and Markers Predict Family Medicine Specialty Certification Status?

Wednesday, October 2nd , 2013Queenstown, New Zealand

Acknowledgements

2

• Carlos Brailovsky (CFPC)

• Sirius Qin

• Marguerite Roy, PhD

• Steve Slade (CAPER)

• Sydney Smee, PhD

• Cindy Streefkerk

• Fang Tian

• Claire Touchie, MD

• 25% of practising physicians in Canada are international medical graduates (IMGs)

• Despite more training positions, we still rely on IMGs

• Rural & remote areas are the most underserviced

Need for IMGs in Canada

3

External Drivers

4

• Agreement on Internal Trade (AIT – 1994)

• AIT aims to reduce barriers to the movement of persons, goods, services and investments within Canada

• Need to develop comparable licensing across Canada to support IMG physician mobility

• Report of the Canadian Task Force on Licensure of International Medical Graduates (2004)

• Standardize licensure processes and requirements

National Assessment Collaboration (NAC)

• The NAC is an alliance of Canadian organizations streamlining the evaluation process for IMGs seeking a licence to practise medicine in Canada

• 2 main focuses

• NAC Exam (since 2011)

• OSCE which assesses readiness of IMGs for entry into Canadian residency programs

• Provides residency program directors with objective information on the skills, attitudes and knowledge level of applicants

© NAC / MCC 2013

5

National Assessment Collaboration (NAC)• The NAC is an alliance of 18 members representing

organizational & jurisdictional stakeholders across Canada

• Mandated to streamline the evaluation process for IMGs seeking a licence to practise medicine in Canada

• 2 main focuses

• Practice Ready Assessment (PRA)

• Partners from across the country are developing a pan-Canadian practice ready assessment process for international medical graduates (IMGs)

• Route available to IMGs seeking a provisional licence to enter independent practice

• Currently, 7 PRA programs across the country

© NAC / MCC 2013

6

Initial Screening(MCCEE/CLBA/

Credentials)

Practice Ready Assessment

Provisional Licensure

IMG Orientation

Summative Assessment

Licensure

(Full)

Practice Eligible Route

for Certification

Seek Alternative Career Path

Seek Alternative Career Path

Canadian

Residency

Training Required

Selection Decision

Other Routes

(i.e., Credentials (Accredited

Qualifications), Other Programs)

Clinical Assessment

(NAC OSCE)

Supervised (Monitored)

Practice

Competencies

Entry-to-Residency

Entry-to-Practice

7

Canadian

• LMCC

• Certification

IMG Physician Routes to Practice!

Assessments

(Over-time)

Common

Co

mp

arab

le P

roce

sse

sFuture: Pan-Canadian PRA Process

Practice Ready Route

Selection AssessmentsPractice-

ReadyDecision

•Minimum Eligibility•Sponsorship (potential future job)•Ranking Tools

IMG Orientation

Over-Time

•Common Toolkit•Assessor Training

Competencies & Context

Screening

Screening assessments (Point-in-Time)

ExperienceCredentials

IMG PGT not in Canada and/or has practised outside of Canada

8

NAC-PRA Predictive Validity Study

9

• Goal• Assess the extent to which a common

set of screening tools and markers are useful in predicting a number of outcomes for a cohort of Family Medicine PRA candidates including:• P/F status on the SOO and SAMPS

components of the College of Family Physicians of Canada (CFPC) certification exam

• Scores on the SOO and SAMPS components of the College of Family Physicians of Canada (CFPC) certification exam

• Time to passing of SOO&SAMPS

NAC-PRA Predictive Validity Study

10

• Usefulness of Study• Will provide valuable information with

respect to informing a pan-Canadian PRA strategy • How might we develop a framework which

concurrently allows for (1) standardization across PRA programs (via common screening tools) and; (b) tailoring of over-time (practice based) assessment tools to address local needs and requirements?

• Allow PRA programs to allocate resources to these tailored assessments

Sample - Eligibility

11

• Criteria

• IMGs who were admitted to a NAC PRA program between 2007-2011

• IMGs who had attempted the CFPC certification examination by end of 2012

• IMGs who had complete records on all variables of interest

Measurement Instruments

12

• MCC Evaluating Exam (MCCEE)• Four-hour CBT examination

composed of 180 single-best answer MCQs

• Offered at >500 sites in 80 countries

• Eligibility pre-requisite for the MCC Qualifying Exams (IMGs & US DOs)

• 50-500 score range (M~271; SD~50)

• Cut-score = 250

Measurement Instruments

13

• MCC Qualifying Exam Part I (MCCQE Part I)

• MCCQE Part I administered in 2 multi-week windows at >12 dedicated secure sites across Canada

• First part is composed of 196 single-best answer computer-delivered MCQs (3.5 hours – 75% of composite score)

• Second part contains ~ 60 clinical decision-making cases (CDMs) – SMs and write-ins (4.0 hours – 25% of composite score)

• Cut-score = 390; score range: 50-950

• Requirement for enrollment into the Canadian Medical Registry as a licentiate of the Medical Council of Canada (LMCC)

Measurement Instruments

14

• CFPC Certification Exam in Family Medicine (pre-harmonization) included:• Five 15-minute simulated office oral exams (SOOs)

• Assesses both the definition and management of health problems in a patient-centered approach

• Physician examiner acts both as patient and examiner

• 30-40 short-answer management problems (SAMPs)

• Measure a candidate’s problem solving skills and knowledge in the context of a clinical situation (3-4 questions/case)

• Overall %-correct score was computed

Analyses

15

• Predictors• MCCEE score (most recent attempt)• MCCQE Part I score (most recent attempt)• Gender • Age at start of PRA process• Years since obtaining MD degree at start of PRA• Degree at English language medical school?

• Outcomes• CFPC P/F status on SOO and SAMPS components• CFPC score on SOO and SAMPS components• Time to successful completion of CFPC SOO & SAMPS

• Models• Logistic regression models• Multiple linear regression models• Survival data analysis model (logit-linear model)

Results - Overview

16

• Sample characteristic “highlights” • N=132

• Gender: Females (51 or 38.6%); Males (81 or 61.4%)

• MD in English: Yes (45 or 34.1%); No (87 or 65.9%)

• Age at entry into PRA: Mean=40.86; SD=6.66

• Years from MD to entering PRA: Mean=16.15; SD=6.49

• Sample performance “highlights” • EE score: Mean=317.3; SD=42.9

• MCCQE Part I: Mean=469.7; SD=73.9

• SOO: Mean=59.32; SD=6.3

• SAMPS: Mean=67.04; SD=5.7

Results – Predicting SOO

17

• Predicting SOO P/F status

• Very weak predictive model

• None of the IVs were significantly associated with passing or failing the SOO component of the CFPC exam

• Predicting SOO %-correct scores

• Very weak predictive model (R2=0.07)

• Only gender significantly predicted SOO scores

• On average women outperformed men by about 3.3% (1/2 SD)

Results – Predicting SAMPS P/F –Final Model

18

Effect Chi-square Type I error Odds ratio

Gender 5.60 0.02 0.194

MCCQE Part 1 score 6.21 0.01 1.020

Age at entry into PRA 4.44 0.03 0.917

Pass rate: 78%

Model fit: L2=24.03, p<0.0001

Results – Predicting SAMPS P/F by QE1 Section – Final Model

19

Effect Chi-square

Type I error Odds ratio

Gender 6.32 0.01 0.177

MCCQE Part 1 MCQ score 5.92 0.01 1.016

Age at entry into PRA 3.98 0.05 0.922

Pass rate: 78%

Model fit: L2=22.74, p<0.0001

Results – Predicting SAMPS Scores Final Model

20

Effect Parameter estimate

t-value Type I error

Gender -2.57 -3.19 0.002

MCCQE Part 1 score 0.04 7.49 0.0001

Age at entry into PRA -0.17 -2.84 0.005

Adjusted R-squared: 0.38

Model fit: F=27.91, p=<0.0001

Results – Predicting SAMPS Scores by QE1 Section – Final Model

21

Effect Parameter estimate

t-value Type I error

Gender -2.51 -3.15 0.002

MCCQE Part 1 MCQ score 0.03 5.73 0.0001

MCCQE Part 1 CDM score 0.02 3.28 0.001

Age at entry into PRA -0.18 -3.03 0.003

Adjusted R-squared: 0.40

Model fit: F=22.60, p=<0.0001

Results – Predicting Time to Attaining SOO & SAMPS P/F Status

22

• Time to SOO P/F status

• Gender & number of attempts are significant predictors time to SOO P/F status

• Better performance on the 2nd attempt

• Time to SAMPS P/F status

• Gender & MCCQE Part I (MCQ) score significantly predict time to passing SAMPS

• Number of attempts is not a significant predictor of SAMPS P/F status

What Does This All Mean? Examples

23

Gender Age MCCQE Part 1 Score

Probability of passing SAMPS

Predicted SAMPS score

Male 30 290 0.20 60.69%

Male 30 390 0.65 64.69%

Male 30 490 0.93 68.69%

Male 50 290 0.04 57.29%

Male 50 390 0.23 61.29%

Male 50 490 0.69 65.29%

What Does This All Mean? Examples –Part Deux

• For each increase of 1 point on the MCCQE Part I (50-950), my odds of passing the CFPC SAMPS section increase by 2%

• If my colleague obtains a score of 350 on the MCCQE Part I and I obtain a score of 500, my odds of passing the SAMPS component increase by 300%!

• For each increase of 1 point on the MCCQE Part I (50-950), my SAMPS %-correct score increases by 0.04%

• If my colleague obtains a score of 350 on the MCCQE Part I and I obtain a score of 500, my SAMPS score will be 6% higher than his

© NAC / MCC 2013

24

Conclusions

25

• Weak predictive models for CFPC SOO component

• Gender predicts SOO score (again, very weak model)

• Gender and number of attempts significant predictors of time to passing the SOO component

• Candidates do better on their 2nd SOO attempt

Conclusions

26

• SAMPS component of the CFPC exam• Very strong predictive model

• MCCQE Part I scores, gender and age at entry into PRA are strong predictors of SAMPS P/F status & score• MCQ component of MCCQE Part I strong predictor of

SAMPS P/F status

• Both MCQ and CDM components predict SAMPS scores

• Gender and MCCQE Part I scores are strong predictors of time to passing SAMPS

• # of attempts does not predict SAMPS P/F status

Implications

27

• MCC EE scores do not significantly predict P/F status nor performance on the CFPC exam

• Caveat: Restriction of range effect on EE scores!

• Consider using MCCQE Part I scores as one common screening tool to predict “knowledge/problem solving skills” competency of NAC PRA candidates

• Use other predictors judiciously!

• Female and younger PRA candidates do better than older male applicants on knowledge/problem solving competencies

Implications

28

• Need an additional common, standardized screening tool to capture SOO competencies

• NAC examination?

Next Steps

29

• Further explore relationships of screening tools to PRA outcomes• Studies done by program due to different outcomes -

aggregate program-level results to better inform a pan-Canadian process

• Further establish screening models• NAC OSCE Discrimination study

• Focus on replicating research with the CFPC’s New Certification Examination in Family Medicine & other specialties

Thank You

30