The Evaluation of RCTs & Observational studies in HTAs on Selected Medical Device Therapies:

WHAT IS THE REAL STATUS OF THIS RELATIONSHIP AND WHAT IMPLICATIONS MAY IT HAVE?

Liesl Birinyi-Strachan: Medtronic Australia

Kathy Cargill: Medtronic International

Gillian Barnett: G Barnett & Associates

Background to medical devices

• Increasing focus of HTA on medical devices internationally

• There are some important differences between devices and drugs

Device development is shorter & incremental in nature (~2yrs)

Sometimes difficulty in conducting RCTs of medical devices

- Issues with blinding (sham often unethical); learning curve for implantable devices (outcomes improve over time); small sample sizes (most severe patients); appropriate comparators; short length (rapid technology turnover)

Research Questions

1.Is all available clinical evidence - both non-randomised /observational AND RCTs studies - being looked for & included in HTAs on selected medical devices?

2.Is the quality of non-randomised /observational studies & RCTs in HTAs on selected medical devices being assessed?

Objectives

To collate, compare and contrast published international HTAs on selected medical device therapies relevant to Medtronic

• Literature searches conducted through to Nov 2011 Medical databases (Medline, Embase, Cochrane and CRD York HTA databases)

HTA agency websites searched Inclusion criteria: • Publically available HTAs from selected medical device therapies (relevant to Medtronic) that treat a range of diseases

• Full HTAs (excluded mini-reviews or HZ scanning reports)

• No language limitations (internal translation assistance)

Methods

• Characteristics extracted according to a pre-defined protocol

• Internal quality checks of extracted data

Included HTAs by therapy

5

5

5

5

8

9

9

10

0 1 2 3 4 5 6 7 8 9 10

Thoracic Endovascular Abdominal Aortic Repair(TEVAR)

Trancatheter aortic valve implant (TAVI)

Intrathecal Baclofen (ITB)

Pacemakers (IPGs)

Cardiac Resynchronisation Therapy (CRT)

Deep Brain Stimulation (DBS)

Sacral Nerve Stimulation (SNS)

Implantable Cardiac Defibrllators (ICD)

56 HTAs

in total

8 therapies

Included HTAs by country

11

9

8

7

7

5

3

2

2

1

1

0 2 4 6 8 10 12

CanadaUSASpain

AustraliaUK

FranceSwedenAustria

ArgentinaBelgiumPoland 11

countries24 agencies

MSAC

AHRQ, BCBS, ICSI, ECRI, WSHCA

CADTH, OHTAC, MUHC

CA HTA, SESCS, AETSA, ANAES, AVALIA, UETS

NICE, BTAG

HAS

SBU, VG

KCE

ICES

IALLC

LBI-HTA

Research Question 1:

Were both non-randomised /observational AND RCTs studies - being looked for & included in HTAs on selected medical devices?

How many HTAs included observational studies/non-RCTs compared to RCTs?

0

5

10

15

20

25

30

35

40

45

INCLUDED NOT INCLUDED

73%

27%

RCTsObs/Non-RCTs

RCTs

32%

Obs/ NonRCT

41/56

15/56

38/56

18/56

No RCTsavailable

68%

73% vs

68%

Were observational studies & non-RCTs explicitly excluded?

0%

20%

40%

60%

80%

Observationalstudies explicitly

excluded

Search strategynot provided

Observationalstudies includedbut none found

72%

YES

17% 11%

13/18Why?

3/182/18

Did the HTAs that explicitly excluded observational studies seek to evaluate efficacy?

NoIn fact, 54% (7/13) actually sought to

examine “effectiveness” of the therapy under evaluation No pragmatic RCTs were searched for &/or

identified in the 7 HTAs

Only 1/13 HTAs sought to evaluate efficacy

In the HTAs that sought to examine “effectiveness” but excluded observational data…what did they conclude?

6 out of 7 of the HTAs had favourable

conclusions on therapy “effectiveness” ….. based on

RCT evidence only large numbers of

Is the quality/ risk of bias of observational studies & RCTs in HTAs on selected medical devices being assessed?

Research question 2

Quality Evaluation

0%

10%

20%

30%

40%

50%

60%

No qualityassessment

Assigned levelsor a score ONLY

Detailed qualityassessment

RCTs Observational studies

24% vs 26%

52% vs 45%

24% vs 29%

Did the type of QA performed affect the conclusions of the HTA?

No QA

RCTs Observational

Levels of evidence only

RCTs

Comprehensive QA

RCTs

FavourableUnfavourable

More studies required

Observational

Observational

In Summary

Of the HTAs that explicitly excluded observational data the majority aimed to evaluate therapy effectiveness

Observational/ non-randomised studies ARE being included in HTAs of selected medical devices

68%

“Distinguishing efficacy from effectiveness and emphasizing its importance to decision making dates back to at least 1978 (Office of Technology Assessment 1978), so it is far from a new concept. ………..But it is somewhat disconcerting that confusion still exists.”

Luce et al 2010: EBM, HTA, and CER: Clearing the Confusion. The Milbank Quarterly, 88(2);256–276

“efficacy trials are all too commonly misspecified as leading to evidence of effectiveness, which can result in over-interpreting their value and relevance to routine practice settings and the resulting decisions.”

In Summary

25% of the HTAs studied failed to assess quality of the studies altogether

……..and a further 45-50% only assigned a simple level as a proxy quality measure

When a comprehensive quality assessment was performed on selected HTAs of medical device therapies, those HTAs that included non-randomised & observational studies were less likely to result in a favourable conclusion (36%) as compared to HTAs that included assessment of RCTs (71%)…

Main Limitations of this analysis

Generalisability: Represents an analysis of published HTAs on selected medical device therapies of relevance to Medtronic

Only HTAs that were publically available in full-text were included

This analysis

Thank you