james presentation - holbrook et al
DESCRIPTION
CATCH-IT Journal ClubTRANSCRIPT
CATCH-IT Journal ClubHolbrook et al. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ. 2009 Jul 7;181(1-2):37-44
James MullenOctober 19th, 2009
HAD 5726
Holbrook et al, CMAJ, 2009 2
Outline
• Background and context of the paper• Methodological issues
– The good, the bad, and the ugly!
• Questions for the authors
Holbrook et al, CMAJ, 2009 3
The Paper
• Purpose:“There have been few randomized trials to confirm that computerized decision support systems can reliably improve patient outcomes”
• Interesting to:– eHealth professionals– Healthcare decision makers– Policy makers– Clinicians– Consumers (ie. patients with chronic diseases)
Holbrook et al, CMAJ, 2009 4
Study investigators
• Anne Holbrook (principal investigator), Division of ClinicalPharmacology and Therapeutics, McMaster University, Hamilton, Ont.• Lisa Dolovich, Department of Family Medicine, McMaster University,Hamilton, Ont.• Hui Lee (deceased), Group Health Centre, Sault Ste. Marie, Ont.• Robert Bernstein, Department of Family Medicine, University ofToronto, Toronto, Ont.• David Chan, Department of Family Medicine, McMaster University,Hamilton, Ont.• Hertzel Gerstein, Department of Medicine, McMaster University,Hamilton, Ont.• Dereck Hunt, Department of Medicine, McMaster University, Hamilton,Ont.• Rolf Sebaldt, Department of Medicine, McMaster University, Hamilton,Ont.• Karim Keshavjee, InfoClin, Toronto, Ont.• Lehana Thabane, Department of Clinical Epidemiology
Holbrook et al, CMAJ, 2009 5
Principle author
• Dr. Anne Holbrook MD, PharmD, B.Sc, M.Sc, FRCPC, FISPE
Holbrook AM, Thabane L, Shcherbatykh IY, O'Reilly D. E-health interventions as complex interventions: improving the quality of methods of assessment. AMIA.Annu.Symp.Proc. 2006:952.
Mollon B, Chong J,Jr, Holbrook AM, Sung M, Thabane L, Foster G. Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Med.Inform.Decis.Mak. 2009 Feb 11;9:11.
Mollon B, Holbrook AM, Keshavjee K, Troyan S, Gaebel K, Thabane L, et al. Automated telephone reminder messages can assist electronic diabetes care. J.Telemed.Telecare 2008;14(1):32-36.
Shcherbatykh I, Holbrook A, Thabane L, Dolovich L, COMPETE III investigators. Methodologic issues in health informatics trials: the complexities of complex interventions. J.Am.Med.Inform.Assoc. 2008 Sep-Oct;15(5):575-580.
Holbrook et al, CMAJ, 2009 6
Hypothesis
“The specific hypothesis was that patients in the intervention group, who had electronic and paper access to an individual diabetes tracker (with data related to recent monitoring and results and targets for 13 risk factors) and whose information was shared with their primary care providers, would have improved quality of diabetes care”
Holbrook et al, CMAJ, 2009 7
Patients
• Diagnosed with Diabetes Mellitus• >18 years of age• Fluent in English• Rostered with a community-based primary care
providersn = 1610
n = 511
n = 258 n = 253
Total patients identified
Total patients eligible
ControlIntervention
Holbrook et al, CMAJ, 2009 8
Intervention
Web-based diabetes tracker of the ‘Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness study II (COMPETE II).
– Interfaces with the providers EMR– Interfaces with an automated telephone reminder system
Holbrook et al, CMAJ, 2009 9
Intervention
COMPETE II
EMR
Telephone
5 different types of EMRs
Holbrook et al, CMAJ, 2009 10
Patient Screen
*http://www.cmaj.ca/cgi/data/cmaj.081272/DC2/2
Holbrook et al, CMAJ, 2009 11
Physician Screen
*http://www.cmaj.ca/cgi/data/cmaj.081272/DC2/1
Holbrook et al, CMAJ, 2009 12
Physician Screen
*http://www.cmaj.ca/cgi/data/cmaj.081272/DC2/1
Holbrook et al, CMAJ, 2009 13
Intervention
• 13 variables selected by a clinical subcommittee– Based on practices by CDA, ADA, and internal literature
review
• Mailed out tracker page 4x/year (twice per patient)
• Phone reminders once a month• Tracker updated nightly
– Patients had the option of personal data entry– Trend data was available
Holbrook et al, CMAJ, 2009 14
Context
COMPETE I: focused on successful implementation of EMRs in small, community-based primary care offices
COMPETE II: this study
COMPETE III: broadened to vascular risk: diabetes, hypertension, cholesterol, previous heart attacks or stroke.
Holbrook et al, CMAJ, 2009 15
Context
•COMPETE II considered a Pilot project
*“A deliberate goal of the project was to have patients regularly visit their family physician”
*http://www.hc-sc.gc.ca/hcs-sss/pubs/chipp-ppics/2003-compete/final-eng.php
Holbrook et al, CMAJ, 2009 16
Methods
1. Recruit primary care providers who were already using EMRs in their practices
2. Have providers identify patients3. Selection of patient participants
• Allocation concealment and central computer generation of group assignment
4. Participants and providers complete baseline questionnaires
5. Patients in intervention sent for set of relevant blood tests (pre-determined) then make appt with FP. (NOT CONTROL GROUP)
Holbrook et al, CMAJ, 2009 17
Timelines
• Study conducted from late 2002 to the end of 2003
• Study follow up was 6 months (mean follow up was 5.9 months)
Holbrook et al, CMAJ, 2009 18
Scores
• Process composite score– Sum of the quality monitoring of the 8 variables
• Glycated hemoglobin; BP; LDL cholesterol; Albuminuria; BMI; Foot surveillance; Exercise; Smoking; ABC composite
• Clinical composite score (8 variables)– Clinical targets for the composite scores
• E.g., BP < 130/80 mm Hg
Holbrook et al, CMAJ, 2009 19
Analysis
• Alpha = 0.05• 2-tailed test (1:1 allocation)• Used t tests to assess the difference between
groups in terms of change in the process composite score
Holbrook et al, CMAJ, 2009 20
Analysis
• They analyzed clinical outcomes in 2 ways:– Change in variable (+1 if positive, 0 if no change, -1 if
negative change)– Whether outcome met predefined targets
Holbrook et al, CMAJ, 2009 21
Process Outcomes
• Improvement for intervention group for 7 out of the 9 variables (no improvement in exercise of smoking)
“Number of visits to primary care provider increased significantly more in the intervention group than in the control group (diff of 0.66, 95% CI 0.37 to 1.02. P< 0.001).”
Holbrook et al, CMAJ, 2009 22
Clinical Outcomes
• Intervention had an improvement on a number of clinical composite variables that were on target (though not significantly significant!)
• Statistically significant improvements in BP and glycated hemoglobin
Holbrook et al, CMAJ, 2009 23
Other Outcomes
• Patients were more ‘optimistic’• 75.9% of intervention patients were ‘more
satisfied’ with their care• No statistically significant changes in quality-of-
life measures (SF-12 and Diabetes-39)
Holbrook et al, CMAJ, 2009 24
Interpretations
“Care of complex chronic disease can be improved with electronic tracking and decision support shared by patients and providers”
Holbrook et al, CMAJ, 2009 25
Hypothesis validated?
“The specific hypothesis was that patients in the intervention group, who had electronic and paper access to an individual diabetes tracker (with data related to recent monitoring and results and targets for 13 risk factors) and whose information was shared with their primary care providers, would have improved quality of diabetes care”
Is the result valid?
Holbrook et al, CMAJ, 2009 26
What can:
• Health professionals learn from this?• Consumers learn from this?• Policy makers learn from this?• Researchers learn from this?
Holbrook et al, CMAJ, 2009 27
Thoughts…
• Overall… this was a useful in that it adds to the literature base. But I want to know if the benefits outweigh the investment in time, money, and resources.
Holbrook et al, CMAJ, 2009 28
What others are saying
*Pros:•Successful implementation of an RCT without major flaws such as:
– temporal trends– participatory bias (Hawthorne effect)
•Environment of the study– Not based in a research hospital
*Grant RW, Middleton B. Improving primary care for patients with complex chronic disease: can health information technology play a role? CMAJ. 2009 Jul 7;181(1-2):37-44.
Holbrook et al, CMAJ, 2009 29
What others are saying
*Cons:•“The authors ascribe the positive impact of their intervention to the influence of individualized decision support and to the role (again) of reminders”•Generalizability
*Grant RW, Middleton B. Improving primary care for patients with complex chronic disease: can health information technology play a role? CMAJ. 2009 Jul 7;181(1-2):37-44.
Holbrook et al, CMAJ, 2009 30
Class comments
• More than 50% of participants never used computers or Internet.
• What about the patients’ view?• Usability of the tracker• Did they educate the participants in using
computers and the tracker?• Is there any data on the usage amount on the
tracker by the participants? (interaction)• How was composite score validated?
Holbrook et al, CMAJ, 2009 31
Questions
• If the control group had as many physicians visits as the Intervention group, would there be a difference in care?
• How would this system improve care with at least a neutral effect on time. (follow up time was only 6 months, what about 2 years?)
• Were the results (improved score) from the decision support system or from the increase in physician visits - better coordination of care? In other words, what was the causal agent? Decision support? Reminders to patient?
Holbrook et al, CMAJ, 2009 32
Questions
• What about the difficulties involved with integrating with COMPETE II with the PCP’s EMRs. What about the technical difficulties.
• 51% of the Intervention group NEVER used the internet. Did this have an effect on the results?
• Scoring was +1 if there was improvement, 0 with no change, and -1 without change. – How much improvement was measured? Was this
improvement a result of the decision support (COMPETE II), automated reminders, or coincidence?
Holbrook et al, CMAJ, 2009 33
Questions
• Why didn’t the researchers do a repeated measures design t test.
• Why didn’t the researchers do baseline lab tests on the control group
• Qualitative data was not explained? Was it anecdotal or statistically calculated?
• How did they measure if patients were actually using the system?– Attrition, adoption, acceptance, and user not really
addressed
Holbrook et al, CMAJ, 2009 34
The end
Holbrook et al, CMAJ, 2009 35
Questions
• What about confounders:– Age– Education– Socio-economic status– Region (three regions used in the study)– Morbidity
• What about negative outcomes?• Why did patients dropout