Can evidence-based medicine andclinical quality improvement learn fromeach other?

Paul Glasziou,1 Greg Ogrinc,2 Steve Goodman3

ABSTRACTThe considerable gap between what we know from

research and what is done in clinical practice is well

known. Proposed responses include the

Evidence-Based Medicine (EBM) and Clinical Quality

Improvement. EBM has focused more on ‘doing the

right things’dbased on external research

evidencedwhereas Quality Improvement (QI) has

focused more on ‘doing things right’dbased on local

processes. However, these are complementary and in

combination direct us how to ‘do the right things right’.

This article examines the differences and similarities in

the two approaches and proposes that by integrating

the bedside application, the methodological

development and the training of these complementary

disciplines both would gain.

INTRODUCTION

Those working in healthcare are aware of theconsiderable gap between what we know fromresearch and what is done in clinical practice.1

For example, enforced bed rest is ineffectivein several conditions where it is still used;exercise reduces mortality in heart failure butis rarely used; and brief counselling aftertrauma is fashionable but ineffective. Theresponse to this well-documented problemincludes both Evidence-Based Medicine(EBM)2 3 andClinicalQuality Improvement.4 5

The term EBM was coined by GordonGuyatt in 1992 for the JAMA ‘User Guide’series to describe the bedside use of researchto improve patient care. At the time it wascommon at McMaster teaching hospitals forpatients’ notes to include a key researchpaper relevant to their care, and for this to bediscussed at ward rounds (personal observa-tiondPG). Improved information tech-nology allowed Dave Sackett’s team to use an‘evidence cart’ on ward rounds at the JohnRadcliffe in Oxford; the team asked and

answered two questions for every threepatients, with the searches changing 1/3 ofclinical decisions.6 Different specialties anddifferent individuals have adopted the prin-ciples of EBM to different degrees and invastly different ways (Interviews on www.cebm.net/index.aspx?o¼4648 illustrate thisdiversity).In parallel, the Quality Improvement (QI)

movement emerged to address similar prob-lems, but with a focus on recurrent problemswithin systems of care. The first methodsused in the National Demonstration Projectsin the 1980s were adopted from thoseintroduced by Deming into industry.4 But themethods soon evolved to the different envi-ronment of healthcare, with the founding ofthe Institute for Healthcare Improvement,and the development of the breakthroughseries collaborative, the model for improve-ment, and other modifications and exten-sions of QI methods.EBM and QI have had overall similar goals

but focus on different parts of the problem.EBM has focused more on ‘doing the rightthings’: actions informed by the best availableevidence from our clinical knowledge base(figure 1), whereas QI has focused more on‘doing things right’: making sure that intendedactions are done thoroughly, efficiently andreliably; however, these are complementary(figure 1) and in combination direct us how to‘do the right things right’.7

Before ‘fixing’ an evidenceepractice gap,we would be wise to ask ‘is this the right thingto do?’ Is there really a problem? Is theresomething effective we can do about it? Forexample, many QI initiatives in diabetes hadaimed to achieve low HbA1c levels8 for whichthe evidence was weak, and subsequent largescale randomised trials (ACCORD andADVANCE) have suggested may be unhelpfulor even harmful.9

1Centre for Research intoEvidence-Based Practice,Faculty of Health Sciencesand Medicine, BondUniversity, Queensland,Australia2Community and FamilyMedicine and Medicine,White River Junction VAMedical Center, DartmouthMedical School, Hanover,New Hampshire, USA3Oncology, Pediatrics,Epidemiology andBiostatistics, Johns HopkinsSchools of Medicine andPublic Health, Baltimore,Maryland, USA

Correspondence toPaul Glasziou, Centre forResearch intoEvidence-Based Practice,Faculty of Health Sciencesand Medicine, BondUniversity, Gold Coast,Queensland 4229, Australia;pglaszio@bond.edu.au

Accepted 17 October 2010

This paper is freely availableonline under the BMJJournals unlocked scheme,see http://qualitysafety.bmj.com/site/about/unlocked.xhtml

BMJ Qual Saf 2011;20(Suppl 1):i13ei17. doi:10.1136/bmjqs.2010.046524 i13

The structure of improvement knowledge

group.bmj.com on June 24, 2012 - Published by qualitysafety.bmj.comDownloaded from

The ‘right things right’ process might be illustrated bya familiar clinical example. In the clinic of one of theauthors, the primary care team was considering whetherto try to wean elderly patients from long-term benzodi-azepines being given for insomnia. We and the patientsseemed reluctant. However, the team reviewed theevidence10 and found the risk of falls on benzodiaze-pines was higher than we’d expected and other adverseeffects, such as cognitive loss, added to the problems. Socessation was the ‘right thing’, but how best to achievethis? A review of the controlled trials showed weaningwas possible, but that simple (but very slow) tapering wasas effective as more intensive methods such as cognitivetherapy counselling. Finally, sending a structured letterto the patient (on why and how to withdraw) had alsobeen shown to be effective.11 Without this evidencereview by the clinical team, we might have wasted a lot ofeffort on ineffective means to achieve our goal. Butwithout a clinical improvement process to change ourpractice, this knowledge might not have provokedaction. Of course, QI would also suggest additionalquestions such as how many patients are on whichbenzodiazepines and for how long? How many falls orother adverse events have occurred on the benzodiaze-pines? What will occur with our patient’s anxiety once weremove the medication?This article aims to look at what each of these disci-

plinary areas might learn from the other. The difference

in approach of the two disciplines may be betterunderstood by looking at the problem each perceives itis addressing.

The EBM perspectiveOne cause of the evidenceepractice gap is informationoverload, for example, approximately 8000 refer-encesdincluding around 350 randomised trialsdareadded to MEDLINE each week. But only a small fractionof this is research that is sufficiently valid and relevant tochange practice. So keeping up to date with new devel-opments and information is problematic. One arm ofEBM has been to synthesise and summarise this flood ofresearch, and be able to access evidence wherever andwhenever it is needed. To achieve this requires bothready access (to resources such as MEDLINE and theCochrane Library) and skills (in finding, appraising andapplying evidence) that few healthcare workers currentlyhave. The EBM movement has focused on developingboth the skills and tools to better connect research andclinical practice, with some12 but not universal successes.A particular focus of EBM has been to take a more

sceptical approach to innovation, asking for clearevidence before changing practice. Given that fewinnovations represent a real advance, this cautiousapproach means less disruption arising from unneces-sary changes in practice.13

The QI perspectiveThe problem addressed by the QI approach might becharacterised as the ‘knowingedoing’ gap: we know(individually and collectively) what to do, but fail to do itor fail to do it correctly. The gap has many causes, fromsimple lack of (some individuals) knowledge about whatshould be done, to uncertainties about the ‘how to’. Forexample, we may know that certain groups of patientsshould receive influenza vaccine but fail to do thisbecause the system does not encourage reliable admin-istration of the vaccine.For example, at one institution, the electronic medical

record (EMR) was fully implemented for many years, sothe physicians-in-training, the staff physicians and nursestrusted the EMR. On discharge from the hospitala prompt in the EMR enquired whether a patientneeded to receive the influenza vaccine. No one realisedthat this prompt led to a blind alleydno vaccine wasordered and no vaccine was given. The individuals (andthe EMR) knew that influenza vaccine was the ‘rightthing to do’, but the EMR and the culture of trusting theEMR inhibited ‘doing the right thing’. Fixing thisproblem proved to be a challengedrequiring severalPlaneDoeStudyeAct iterations. The simple change inthe EMR system and the influenza vaccine ordering werelow on the priority list for the IT support group. This

Process/SystemChanges

ClinicalDecisions

GoodPa�ent

Care

Do things right (QI)

Do right things right

Do right things (EBM)

Clinical Knowledge Base

(MEDLINE,Cochrane, GIN, ...)

Process Knowledge Base

(EPOC, IHI, BEME ..)

Local Global

(a) (b)

(c)

Figure 1 Relationships between Quality Improvement (QI)and Evidence-Based Medicine (EBM). (a) sequence of EBMfollowed by QI; (b) EBM uses clinical knowledge to informindividual clinical decisions about patient care; (c) QI focuseson improving recurrent problems in the processes of care(Acronyms: GINdGuidelines International Network;EPOCdEffective Practice and Organisation of Care Group;IHIdInstitute for Healthcare Improvement; BEMEdBestEvidence Medical Education).

i14 BMJ Qual Saf 2011;20(Suppl 1):i13ei17. doi:10.1136/bmjqs.2010.046524

The structure of improvement knowledge

group.bmj.com on June 24, 2012 - Published by qualitysafety.bmj.comDownloaded from

necessitated that the care team create a work-around andlearn how to reliably ‘do the right thing’ in the contextand setting of care.14 Even an intervention as simple asadministering influenza vaccine in a fully integratedEMR environment required a thorough knowledge ofthe system and then creativity in dealing with its limi-tationsdmuch more than just the knowledge of the‘right thing’.The techniques of QI have focused on integrating

what we know and moving it to how we should be doingit. Common techniques to achieve this are remindersystems (paper or computer), simplification of processes(reducing unnecessary steps), structuring the system todo the right thing at the right time and place (puttinghand-washing equipment and/or instructions in theright place) and testing new interventions to determinewhat works in this particular setting. This task is oftena creative and iterative process of identifying barriers,and working out solutions to overcome these.

MARRYING EBM AND QI

As illustrated by the above examples, EBM is involved inthe early stages of checking the validity and applicabilityof the available evidence to the clinical problem. Thisinvolves the traditional ‘four steps’ of EBM illustrated infigure 2. QI processes15 may be triggered at the fourthstep if it seems likely that the clinical problem isa common one for which the current system of practiceis not optimal.16 Similarly, in the planning stage of a QIproject there may be several questions that trigger anEBM cycle to check for evidence.In addition to this merging of EBM and QI processes,

there are deeper organisational and epistemologicalissues in common which we briefly discuss in the nextsection.

THE EVIDENCE FOR EBM AND QI

A common criticism levelled at both EBM and QI is thatthere is only weak evidence that either process makesa difference to patient outcomes. However, neither EBMnor QI are a single ‘thing’ and cannot be evaluated inthe same way as a fixed dose of a single chemical entity.Rather they are disciplines with multiple techniques thatmay be used to address the researchepractice gap. Forexample, we know from large randomised trials thataspirin lowers mortality after myocardial infarction, butis underused; we would therefore want to improveappropriate usage. EBMers might focus on the ‘appro-priate’ part (subgroups, balance of benefits and harms,etc); QIers might focus on the usage part (barriers,prescribing systems, etc). But success here could belargely judged by process measuresdan increased use inaspirindrather than in-hospital mortality. The link tomortality has already been proven in the trials. Hence,rather than enquire, ‘what is the evidence that EBM (orQI) are beneficial’, we should instead ask what are thebest techniques within each discipline for achievingbetter practice? In order to improve, we need to know inwhat circumstances do those techniques work and howwe can we disseminate those techniques?The problems with assessing interventions by process

or outcome measurements is illustrated by a recentsystematic review17 of QI ‘collaboratives’ that focused onincreasing the use of surfactant in premature infants. Acollaborative (sometime called ‘breakthrough collabo-rative’) is a consortium of 20e40 healthcare organisa-tions using QI methods on a specific healthcare qualityproblem. The review found nine eligible studies,including only two randomised trials. The reviewersconcluded that the evidence for collaboratives was‘limited’ because the first trial showed no effect, and thesecond trial showed ‘significant improvement in twospecific processes of care but no significant improve-ment in patient outcomes (mortality and pneumo-thorax)’. However, their conclusions may ask too muchof a trial’s ability to detect real changes in outcomes. Theimprovement in processes of care included a substantialincrease in surfactant use from 18% to 54% (a 36%increase). However, given the pooled trials of surfactant,which included 1500 randomised infants,18 was barelyenough to demonstrate the mortality reduction,expecting to detect a mortality reduction by the collab-orative is unrealistic. With the 36% improvement insurfactant use seen, to reliably detect the predictedmortality reduction, we would require a trial of collabo-ratives about nine times larger (931500), that is at least13 500 infants individually randomised. This may beinfeasible and unnecessary. If both steps (surfactanteffectiveness and the QI process improvement: see

Figure 2 Proposed linkage between EBM and one model forQI.

BMJ Qual Saf 2011;20(Suppl 1):i13ei17. doi:10.1136/bmjqs.2010.046524 i15

The structure of improvement knowledge

group.bmj.com on June 24, 2012 - Published by qualitysafety.bmj.comDownloaded from

figure 1) separately have strong evidence, then thisrepresents a ‘complete causal chain’ whose evidence isequal to the evidence of the weakest link.19 The moreimportant question here then is ‘What elements of theneonatal collaboration were important?’ and ‘How wellwill that transfer to other settings?’By shifting the focus to specific methods we may ask

more focused and answerable questions. For example,how effective are reminder systems to reduce ‘slips’? Asystematic review of trials of reminders is a valuableresource for QI practitioners wanting to know when andhow they do or don’t work?20 Similarly, having to supporta team’s evidence-based practice, a ‘clinical informati-cist’ is intuitively appealing and ‘do-able’,21 but evalua-tive trials, which show a positive impact of clinicalinformaticists on clinical decision making, are relativelyrecent.22 Some techniques, such as a QI collaborative,may be a complex mix of techniques and thereforeintrinsically more difficult to evaluate. Evaluation is stillworthwhile, but may shift focus to understanding whatmethods a particular collaborative used, and whatseemed to work or not and in what circumstances.Finally, the epistemology of both disciplines is

evolving. A better understanding of the science andscientific rules of both areas will be important for theircontinued growth and impact. For example, an early butsimplistic interpretation of EBM was that all interven-tions required randomised trial evidence. While impor-tant, we now recognise that different types of questionsneed different types of evidence,23 and that even fortreatment questions occasional evidence from non-randomised studies, including some case series, can besufficiently compelling.24

A WAY FORWARD

Early QI methods in healthcare incorporated a link toevidence, but this connection seems to have faded overthe years. In the early 1990s, The Hospital Corporationof America (HCA) developed and used FOCUS-PDCAthat explicitly included a detailed analysis of theevidence for proposed changes, the processes and thedata about local performance.25 This methodologydeveloped into the PDSA cycle,15 a common, simple andeffective technique, but one where the connection toevidence is less clear. We propose that re-establishinga clear connection between EBM and QI will benefitboth disciplines and, ultimately, benefit patients andfamilies. For those engaged in either QI or EBM (orhopefully both!) there are several implications, bothepistemological and practical, of the complementaryfocus and methods of the two disciplines.Those working in QI teams, before taking on a change

should routinely check the validity, applicability and

value of the proposed change and should not simplyaccept external recommendations. (Corollary: At leastsome members of a QI team must have high level skillsin EBM.)Those working in EBM should recognise that it is

not sufficient to simply appraise the evidence, but atthe end we should ask ‘what is the next action’16 (andsometimes enter a PDSA cycle) (Corollary: At least somemembers of an EBM team will need high level skillsin QI.)Those working on the methods of QI and EBM should

stop being so concerned about whether the abstractconcepts of EBM or QI ‘work’, and instead focus ondevelopment and evaluation of specific methods of eachthat sheds light on what elements are most effective inwhat circumstances. This evaluation should involve tworelated processes. First, recognise that ‘experientiallearning’ is a cyclic process of doing, noticing, ques-tioning, reflecting, exploring concepts and models(evidence), then doing againdonly doing it better thenext time (PDSA cycles).26 Second, when new potentialgeneralisable techniques are developed, then theseshould be subjected to a more formal evaluation.Recently, several stages of evaluation specific to surgeryhave been proposed,27 which recognise the develop-ment and learning needed before a full evaluation.Related problems have been recognised in applying theMedical Research Council (MRC) complex interventionsframework for health improvement.28 However, somecreative tension between doing, developing and evalu-ating will always exist.Finally, those teaching the next generation of clini-

cians should value both disciplines, which should betaught, integrated and modelled in clinical training.29

Medical curricula, undergraduate and postgraduate, andhealthcare organisations should incorporate both EBMand QI training and these should be taught as an inte-gral whole. Such training requires learning backgroundskills and theory, but also ‘bedside’ teaching andmodelling of how EBM and QI are applied in real clin-ical settings. By integrating the bedside application, themethodological development, the training and theorganisation support of these complementary disci-plines, hopefully, we can ever more frequently do the‘right things right’.

Acknowledgements We would like to thank Paul Batalden for initiating thistopic, and Frank Davidoff for helpful comments.

Funding This material is based on support and use of facilities at the WhiteRiver Junction VA from the VA National Quality Scholars Program; Dr Glasziouis supported by an NHMRC Fellowship.

Competing interests None declared.

Contributors All authors contributed to the development of the ideas, thewriting, and read the final manuscript.

Provenance and peer review Not commissioned; externally peer reviewed.

i16 BMJ Qual Saf 2011;20(Suppl 1):i13ei17. doi:10.1136/bmjqs.2010.046524

The structure of improvement knowledge

group.bmj.com on June 24, 2012 - Published by qualitysafety.bmj.comDownloaded from

REFERENCES1. Asch SM, Kerr EA, Keesey J, et al. Who is at greatest risk for

receiving poor-quality health care? N Engl J Med 2006;354:1147e56.2. Evidence-Based Medicine Working Group. Evidence-based

medicine. A new approach to teaching the practice of medicine. JAMA1992;268:2420e5.

3. Sackett DL, Rosenberg WM, Gray JA, et al. Evidence basedmedicine: what it is and what it isn’t. BMJ 1996;312:71e2.

4. Berwick DM. Continuous improvement as an ideal in health care.N Engl J Med 1989;320:53e6.

5. Batalden PB, Davidoff F. What is “quality improvement” and how canit transform healthcare? Qual Saf Health Care 2007;16:2e3.

6. Sackett DL, Straus SE. Finding and applying evidence during clinicalrounds: the “evidence cart”. JAMA 1998;280:1336e8.

7. Irwig L. An Approach to Evaulating Health Outcomes. NSW PubHealth Bull 1994;4:135e6.

8. Shojania KG, Ranji SR, McDonald KM, et al. Effects of qualityimprovement strategies for type 2 diabetes on glycemic control:a meta-regression analysis. JAMA 2006;296:427e40.

9. Lehman R, Krumholz HM. Glycated haemoglobin below 7%. No toQOF target of less than 7%, again. BMJ 2010;340:c985.

10. Glass J, Lanctot KL, Herrmann N, et al. Sedative hypnotics in olderpeople with insomnia: meta-analysis of risks and benefits. BMJ2005;331:1169.

11. Heather N, Bowie A, AshtonH, et al. RandomizedControlled trial of twobrief interventions against long-term benzodiazepine use: outcome ofinterventions. Addiction Research and Theory 2004:141e54.

12. Straus SE, Ball C, Balcombe N, et al. Teaching evidence-basedmedicine skills can change practice in a community hospital. J GenIntern Med 2005;20:340e3.

13. Dixon-Woods M, Amalberti R, Goodman S, et al. Problems andpromises of innovation: why healthcare needs to rethink its love/haterelationship with the new. BMJ Qual Saf 2011;20(S1):i47ei51.

14. Ogrinc G, Batalden PB. Realist evaluation as a framework for theassessment of teaching about the improvement of care. J Nurs Educ2009;48:661e7.

15. Langley GL, Nolan KM, Nolan TW, et al. The Improvement Guide: APractical Approach to Enhancing Organizational Performance. SanFrancisco: Jossey-Bass, 1996.

16. Glasziou P. Applying evidence: what’s the next action? Evid BasedMed 2008;13:164e5.

17. Schouten LM, Hulscher ME, van Everdingen JJ, et al. Evidence forthe impact of quality improvement collaboratives: systematic review.BMJ 2008;336:1491e4.

18. Soll R, Ozek E. Prophylactic protein free synthetic surfactant forpreventing morbidity and mortality in preterm infants. CochraneDatabase Syst Rev 2010;(1):CD001079.

19. Howick J, Aronson J, Glasziou P. Evidence-based mechanisticreasoning. JRSM 2010;103:433e41 (accepted).

20. Shojania KG, Jennings A, Mayhew A, et al. Effect of point-of-carecomputer reminders on physician behaviour: a systematic review.CMAJ 2010;182:E216e25.

21. Scura G, Davidoff F. Case-related use of the medical literature.Clinical librarian services for improving patient care. JAMA1981;245:50e2.

22. Mulvaney SA, Bickman L, Giuse NB, et al. A randomizedeffectiveness trial of a clinical informatics consult service: impact onevidence-based decision-making and knowledge implementation.J Am Med Inform Assoc 2008;15:203e11. Epub 2007 Dec 20.

23. Glasziou P, Vandenbroucke JP, Chalmers I. Assessing the quality ofresearch. BMJ 2004;328:39e41.

24. Glasziou P, Chalmers I, Rawlins M, et al. When are randomised trialsunnecessary? Picking signal from noise. BMJ 2007;334:349e51.

25. Batalden PB. ‘Building Knowledge for Improvement - An IntroductoryGuide to the Use of FOCUS-PDCA,’ Quality Resource Group,Hospital Corporation of America. Nashville, TN, 1992.

26. Kolb D. Experiential Learning: Experience as the Source ofLearning and Development. Upper Saddle River, NJ: Prentic Hall,1984.

27. McCulloch P, Altman DG, Campbell WB, et al; Nicholl J for the BalliolCollaboration. No surgical innovation without evaluation: the IDEALrecommendations. Lancet 2009;374:1105e12.

28. Mackenzie M, O’Donnell C, Halliday E, et al. Do health improvementprogrammes fit with MRC guidance on evaluating complexinterventions? BMJ 2010;340:c185.

29. Coomarasamy A, Khan KS. What is the evidence that postgraduateteaching in evidence based medicine changes anything? Asystematic review. BMJ 2004;329:1017.

BMJ Qual Saf 2011;20(Suppl 1):i13ei17. doi:10.1136/bmjqs.2010.046524 i17

The structure of improvement knowledge

group.bmj.com on June 24, 2012 - Published by qualitysafety.bmj.comDownloaded from

doi: 10.1136/bmjqs.2010.046524 2011 20: i13-i17BMJ Qual Saf

 Paul Glasziou, Greg Ogrinc and Steve Goodman quality improvement learn from each other?Can evidence-based medicine and clinical

http://qualitysafety.bmj.com/content/20/Suppl_1/i13.full.htmlUpdated information and services can be found at:

These include:

References

http://qualitysafety.bmj.com/content/20/Suppl_1/i13.full.html#related-urlsArticle cited in:  

http://qualitysafety.bmj.com/content/20/Suppl_1/i13.full.html#ref-list-1This article cites 24 articles, 13 of which can be accessed free at:

Open Access

http://creativecommons.org/licenses/by-nc/2.0/legalcode.http://creativecommons.org/licenses/by-nc/2.0/ and compliance with the license. See:work is properly cited, the use is non commercial and is otherwise in use, distribution, and reproduction in any medium, provided the originalCreative Commons Attribution Non-commercial License, which permits This is an open-access article distributed under the terms of the

serviceEmail alerting

the box at the top right corner of the online article.Receive free email alerts when new articles cite this article. Sign up in

CollectionsTopic

(72 articles)Unlocked   � Articles on similar topics can be found in the following collections

Notes

http://group.bmj.com/group/rights-licensing/permissionsTo request permissions go to:

http://journals.bmj.com/cgi/reprintformTo order reprints go to:

http://group.bmj.com/subscribe/To subscribe to BMJ go to:

group.bmj.com on June 24, 2012 - Published by qualitysafety.bmj.comDownloaded from