item response theory in patient-reported outcomes research ron d. hays, ph.d., ucla society of...
TRANSCRIPT
![Page 1: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/1.jpg)
Item Response Theory in Patient-Reported
Outcomes Research
Ron D. Hays, Ph.D., UCLA
Society of General Internal Medicine
California-Hawaii Regional MeetingFebruary 2, 2013 (10:45-12:00 pm)
Covel Commons (Grand Horizon, Salon C)
![Page 2: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/2.jpg)
Computer Adaptive Testing (CAT)
![Page 3: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/3.jpg)
• Patient Reported Outcomes Measurement Information System (PROMIS®)
• Funded by the National Institutes of Health
• One domain captured is “anger”– Mood (irritability, frustration)– Negative social cognitions (interpersonal sensitivity,
envy, disagreeableness) – Needing to control anger
www.nihpromis.org
![Page 4: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/4.jpg)
Reliability (0-1) - 0.70 or above for group comparisons- 0.90 or above for individual assessment
z-scores (mean = 0 and SD = 1):– Reliability = 1 – SE2
– So reliability = 0.90 when SE = 0.32
T-scores (mean = 50 and SD = 10):– Reliability = 1 – (SE/10)2
– So reliability = 0.90 when SE = 3.2
T = 50 + (z * 10)
![Page 5: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/5.jpg)
In the past 7 days …
I was grouchy [1st question]– Never [39]– Rarely [48]– Sometimes [56]– Often [64]– Always [72]
Theta = 56.1 SE = 5.7 (rel. = 0.68)
![Page 6: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/6.jpg)
In the past 7 days …I felt like I was ready to explode [2nd question]
– Never– Rarely– Sometimes– Often– Always
Theta = 51.9 SE = 4.8 (rel. = 0.77)
![Page 7: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/7.jpg)
In the past 7 days …
I felt angry [3rd question]– Never– Rarely– Sometimes– Often– Always
Theta = 50.5 SE = 3.9 (rel. = 0.85)
![Page 8: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/8.jpg)
In the past 7 days …I felt angrier than I thought I should [4th question] - Never
– Rarely– Sometimes– Often– Always
Theta = 48.8 SE = 3.6 (rel. = 0.87)
![Page 9: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/9.jpg)
In the past 7 days …
I felt annoyed [5th question]– Never– Rarely– Sometimes– Often– Always
Theta = 50.1 SE = 3.2 (rel. = 0.90)
![Page 10: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/10.jpg)
In the past 7 days …
I made myself angry about something just by thinking about it. [6th question]
– Never– Rarely– Sometimes– Often– Always
Theta = 50.2 SE = 2.8 (rel = 0.92)
![Page 11: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/11.jpg)
Theta, SEM, and 95% CI
56 and 6 (reliability = .68) W = 2252 and 5 (reliability = .77) W = 1950 and 4 (reliability = .85) W = 1549 and 4 (reliability = .87) W = 1450 and 3 (reliability = .90) W = 1250 and <3 (reliability = .92) W = 11
![Page 12: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/12.jpg)
Response Burden
• Paper and pencil rules of thumb– 3-5 items per minute
• PROMIS computer administration to general population – 8-12 items per minute
• Scleroderma patients at UCLA– 6 items per minute
12
![Page 13: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/13.jpg)
Item Response Theory (IRT)IRT models the relationship between a
person’s response Yi to the question (i) and his or her level of the latent construct :
bik is how difficult it is to answer in category k or higher on item i
ai is the item discrimination or slope parameter.
)exp(1
1)Pr(
ikii bakY
![Page 14: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/14.jpg)
Latent Trait and Item Responses
Latent Trait
Item 1 Response
P(X1=1)P(X1=0)
10
Item 2 Response
P(X2=1)P(X2=0)
10
Item 3 Response
P(X3=0)0
P(X3=2)2
P(X3=1) 1
![Page 15: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/15.jpg)
Item Responses and Trait Levels
Item 1 Item 2 Item 3
Person 1 Person 2Person 3
TraitContinuum
+ z-score- z-score
![Page 16: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/16.jpg)
Person Scale Scores (θ)• Level on attribute measured
• Average items together and compute z-score
• Mean = 0, SD = 1
zX =
SDX
(X - X)
![Page 17: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/17.jpg)
Item difficulty (p = 0.84)
Proportion of people endorsing the item (p) can be expressed as z:
Z = ln (1-p)/p)/1.7 = (ln (1-p) – ln (p))/1.7 = (ln (.16) – ln (.84))/1.7 = (-1.83 + .17)/1.7 = -1.66/1.7 = -1.00-1.00
(-2 -> 2 is typical range)
![Page 18: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/18.jpg)
P-value transformation for an Item (p =.84)
34% 50%
-3 -2 -1 0 +1 +2 +3
![Page 19: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/19.jpg)
Item Discrimination or Slope
Item-scale correlation can be expressed as z:
- z = ½ [ln (1 + r) – ln (1-r) ]
- if r = 0.30, z = 0.31
- if r = 0.80, z = 1.10
- if r = 0.95, z = 1.83
- if r = 0.99, z = 2.65
![Page 20: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/20.jpg)
0
20
40
60
80
100
-.40 -.20 0 .40.20 .60 .80 1.00
28
15
5, 23
2919
13
10 2
4
20
11
1
25
17
8, 9
16
4
263
0 177
63
18
14
22
12
27
Discrimination (Item-Scale Correlation)
Dif
ficu
lty
(Per
cent
Pas
sing
)
![Page 21: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/21.jpg)
IRT Versus CTT • Dichotomous and polytomous items
• Item parameters (difficulty and discrimination) estimated using logistic models instead of proportions and item-scale correlations
• IRT models
• Rasch model/Graded response model
• Difficulty parameter
• Discrimination or slope parameter
![Page 22: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/22.jpg)
2-Parameter Logistic IRT Model
0.00
0.25
0.50
0.75
1.00
-3 -2 -1 0 1 2 3
Fatigue
Pro
babilit
y of
Resp
onse
Severe FatigueEnergetic
b = 0.25
b = 1.33
b = -0.23
a = 2.83
a = 1.11
a = 2.20
ii bai eXP
7.11
11 ai( – bi)
![Page 23: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/23.jpg)
IRT Versus CTT
Reliability (information) is conditional on where one is estimated to be on the underlying attribute
Rather than estimated overall (coefficient alpha)
![Page 24: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/24.jpg)
Information Conditional on Trait Level
• Item information proportional to inverse of standard error of measurement:
• Scale information is the sum over item information:
n
iiII
1
)()(
)(
1)(
ISEM
![Page 25: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/25.jpg)
Item Parameters indicate where items are mostuseful (informative) for distinguishing among
respondents)
0.0
0.5
1.0
1.5
2.0
-3 -2 -1 0 1 2 3Fatigue
Info
rmati
on
• I am too tired to do I am too tired to do errands.errands.
• I am too tired to eatI am too tired to eat• I need to sleep I need to sleep during the dayduring the day..
Severe FatigueEnergetic b = 0.25 b = 1.33b = -0.23
a = 2.83
a = 1.11
a = 2.20
![Page 26: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/26.jpg)
IRT Versus CTT
• Item parameters estimated by:
– Marginal maximum likelihood estimation (MML)
• Level on attribute (person score or θ) estimated by:
– ML (maximum likelihood)– MAP (Maximum a posterior scoring)– EAP (Expected a posterior scoring)
![Page 27: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/27.jpg)
Scoring All Response Patterns Using Sum Score and Different IRT Models
#
Item ResponsePattern
0 = false, 1 = trueSummed
Score
1 PL IRT /Rasch ModelM-L Estimate
2 PL IRT ModelM-L Estimate
1 0 0 0 0 0 -0.84 -0.822 1 0 0 0 1 -0.22 -0.273 0 1 0 0 1 -0.22 -0.214 0 0 1 0 1 -0.22 -0.195 0 0 0 1 1 -0.22 -0.016 1 1 0 0 2 0.22 0.147 1 0 1 0 2 0.22 0.158 0 1 1 0 2 0.22 0.199 1 0 0 1 2 0.22 0.31
10 0 1 0 1 2 0.22 0.3611 0 0 1 1 2 0.22 0.3712 1 1 1 0 3 0.71 0.5213 1 1 0 1 3 0.71 0.7214 1 0 1 1 3 0.71 0.7415 0 1 1 1 3 0.71 0.8016 1 1 1 1 4 1.36 1.35
![Page 28: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/28.jpg)
IRT Assumptions
• Dimensionality– Unidimensionality for typical models
• Local Independence• Monotonicity• Person fit
28
![Page 29: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/29.jpg)
29
Hypothesized One-Factor Model
Physical Function
Climbing a flight of stairs
Climbing a flight of stairs
Running a mile
Running a mile
Feeding myself
Feeding myself
![Page 30: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/30.jpg)
Sufficient Unidimensionality
• One-Factor Categorical Confirmatory Factor Analytic Model (e.g., using Mplus)– Polychoric correlations– Weighted least squares with adjustments for
mean and variance
• Bifactor Model– General factor and group-specific factors
30
![Page 31: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/31.jpg)
Local Independence•After controlling for dominant factor(s), item
pairs should not be associated.
•Evaluated by looking at size of residual correlations from one-factor model
– Look for residual correlations > 0.20
•Avoid asking the same item multiple times.– “I’m generally sad about my life.”– “My life is generally sad.”
![Page 32: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/32.jpg)
Item a b1 b2 b3 b4 Global01 7.37 (na) -1.98 (na) -0.97 (na) 0.03 (na) 1.13 (na) Global03 7.65 (2.31) -1.89 (-2.11) -0.86 (-0.89) 0.15 ( 0.29) 1.20 ( 1.54) Global06 1.86 (2.99) -3.57 (-2.80) -2.24 (-1.78) -1.35 (-1.04) -0.58 (-0.40) Global07 1.13 (1.74) -5.39 (-3.87) -2.45 (-1.81) -0.98 (-0.67) 1.18 ( 1.00) Global08 1.35 (1.90) -4.16 (-3.24) -2.39 (-1.88) -0.54 (-0.36) 1.31 ( 1.17)
Note: Parameter estimates for 5-item scale are shown first, followed by estimates for 4-item scale (in parentheses). na = not applicable
a = discrimination parameter; b1 = 1st threshold; b2 = 2nd threshold; b3 = 3rd threshold; b4 = 4th threshold Global01: In general, would you say your health is …? Global03: In general, how would you rate your physical health?
Global06: To what extent are you able to carry out your everyday physical activities?
Global07: How would you rate your pain on average?
Global08: How would you rate your fatigue on average?
Graded Response Model Parameters for Global Physical Health
Graded Response Model Parameters for Global Physical Health
![Page 33: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/33.jpg)
Monotonicity
•Probability of selecting a response category indicative of better health should increase as underlying health increases.
•Item response function graphs with•y-axis: proportion positive for item step•x-axis: raw scale score minus item score
33
![Page 34: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/34.jpg)
Check of Monotonicity
![Page 35: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/35.jpg)
IRT Model Fit
• Compare observed and expected response frequencies by item and response category
• Items that do not fit and less discriminating items identified and reviewed by content experts
![Page 36: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/36.jpg)
Person Fit• Large negative ZL values indicate misfit.
– ZL has expected value of zero, with variance of one if responses are consistent with IRT model. Large negative ZL values (>= -2.0) indicate misfit.
–
• Person who responded to 14 PROMIS physical functioning items had a ZL = -3.13– For 13 items the person could do the activity
(including running 5 miles) without any difficulty.– But this person reported a little difficulty being
out of bed for most of the day.
![Page 37: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/37.jpg)
Person Fit
Item misfit significantly associated with
– Less than high school education– More chronic conditions – Longer response time
![Page 38: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/38.jpg)
Nice Features of IRT
• Category response curves (CRCs)• Information depending on theta • Assessing differential item
functioning
![Page 39: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/39.jpg)
Samejima’s Graded Response Model (Category Response Curves)
0.0
0.2
0.4
0.6
0.8
1.0
-3 -2 -1 0 1 2 3
Theta ()
P(X
=k|
)
P (X = 3|)P (X = 1|)
P (X = 2|)
P (X = 4|)
![Page 40: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/40.jpg)
Posttraumatic Growth Inventory
Indicate for each of the statements below the degree to which this change occurred in your life as a result of your crisis.
(Appreciating each day)(0) I did not experience this change as result of my crisis
(1) I experienced this change to a very small degree as a result of my crisis
(2) I experienced this change to a small degree as a result of my crisis
(3) I experienced this change to a moderate degree as a result of my crisis
(4) I experienced this change to a great degree as a result of my crisis
(5) I experienced this change to a very great degree as a result of my crisis
![Page 41: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/41.jpg)
0.0
0.2
0.4
0.6
0.8
1.0
-3.00 -2.00 -1.00 0.00 1.00 2.00 3.00
Posttraumatic Growth
Pro
babilit
y o
f R
esponse
Category Response Curves
Great Change
No Change
Very small
change
No chang
e
Small change
Moderate
change
Great change
Very great change
“Appreciating each day.”
![Page 42: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/42.jpg)
Drop Response Options?
Indicate for each of the statements below the degree to which this change occurred in your life as a result of your crisis. (Appreciating each day)
(0) I did not experience this change as result of my crisis
(1) I experienced this change to a moderate degree as a result of my crisis
(2) I experienced this change to a great degree as a result of my crisis
(3) I experienced this change to a very great degree as a result of my crisis
![Page 43: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/43.jpg)
Reword response options?
• Might be challenging to determine what alternative wording to use so that the replacements are more likely to be endorsed.
![Page 44: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/44.jpg)
Keep as is?
• CAHPS global rating items– 0 = worst possible– 10 = best possible
• 11 response categories capture about 3 levels of information.– 10/9/8-0 or 10-9/8/7-0
• Scale is administered as is and then collapsed in analysis
![Page 45: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/45.jpg)
Response Burden vs. Standard Error (SE)
• 3-5 items per minute rule of thumb for paper survey– 8 items per minute for
dichotomous items• Lowering SE means adding or
replacing existing items with more informative ones at the target range of the continuum.
![Page 46: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/46.jpg)
Reliability and Information– Only as much response burden as needed for
target level of reliability
– CATs for patient-reported outcomes yield 0.90 reliability with about 5-6 items
– For z-scores (mean = 0 and SD = 1):• Reliability = 1 – SE2 = 0.90 (when SE = 0.32)• Information = 1/SE2 = 10 (when SE = 0.32)• Reliability = 1 – 1/information
![Page 47: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/47.jpg)
Gastroesophageal Reflux1. How often did you have regurgitation—that is, food or liquid coming back up into your throat or mouth without
vomiting?2. What was the most food or liquid you had come back into your mouth at one time?3. During the time you were awake, how often did you gag on liquid or food coming back up into your throat?4. How often were you awakened from sleep by choking on liquid or food coming back up into your throat?5. How much did regurgitation bother you?6. After eating a meal, how often did food or liquid come back into your throat or mouth without vomiting?7. How often did you re-swallow food that came back into your throat?8. How often did you feel like you were going to burp, but food or liquid came up instead?9. How often did you feel like there was too much saliva in your mouth?10. How frequently did you feel burning in the red area show in the picture—that is, behind the breastbone?11. At its worst, how bad was the pain behind your breastbone?12. How often did you know that you would have pain behind your breastbone before it happened?13. How much did pain behind the breastbone interfere with your day-to-day activities?14. How much did pain behind the breastbone bother you?15. When you had pain behind the breastbone, how long did it usually last?16. How often did you feel burning in your throat?17. How often did you burp?18. How much did burping bother you?19. How often did you have hiccups?20. How much did hiccups bother you?21. How often did you feel like there was a lump in your throat?22. How much did having a lump in your throat bother you?
![Page 48: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/48.jpg)
Test Information for Reflux
0
2
4
6
8
10
12
14
16
18
20
22
24
26
-3 -2 -1 0 1 2 3
Tota
l Info
rmat
ion
Theta
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Standard Error
Group 1, Total Information Curve
Total Information Standard Error
![Page 49: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/49.jpg)
Differential Item Functioning (DIF)
• Probability of choosing each response category should be the same for those who have the same estimated scale score, regardless of other characteristics
• Evaluation of DIF – Different subgroups – Mode differences
![Page 50: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/50.jpg)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4
Trait level
Pro
babi
lity
of "
Yes
" R
espo
nse
Location DIF
Slope DIF
DIF (2-parameter model)
White
AA
AA
White
Location = uniform; Slope = non-uniform
![Page 51: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/51.jpg)
Suggested Readings Cella, D., et al. (2010). Initial item banks and first wave testing of the
Patient-Reported Outcomes Measurement Information System (PROMIS) network: 2005-2008. Journal of Clinical Epidemiology, 63 (11), 1179-1194.
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. New Jersey: Erlbaum.
Gorman, S. Computerized adaptive testing with a military population. http://www.psych.umn.edu/psylabs/catcentral/pdf%20files/go77221.pdf
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff.
Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st Century. Medical Care, 38, II-28-42.
www.nihpromis.org
![Page 52: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/52.jpg)
52
Questions?
Contact Information:[email protected] (310-794-2294)
Powerpoint file available for downloading at: http://gim.med.ucla.edu/FacultyPages/Hays/
![Page 53: Item Response Theory in Patient-Reported Outcomes Research Ron D. Hays, Ph.D., UCLA Society of General Internal Medicine California-Hawaii Regional Meeting](https://reader030.vdocuments.pub/reader030/viewer/2022033104/56649d015503460f949d46ea/html5/thumbnails/53.jpg)
Acknowledgment of Support
University of California, Los Angeles, Resource Center for Minority Aging Research (RCMAR)/ Center for Health Improvement in Minority Elderly (CHIME), NIH/NIA Grant P30-AG021684.