basic concepts of research
DESCRIPTION
Some basic concepts of research. A lecture given to postgraduate students of emergency medicine Master of Medicine in Universiti Sains Malaysia. Contents include: how to appraise an article using the RAMMBO, asking PICO question, test for heterogeneity in meta-analyses, confidence interval, forest-plot concept, Odds vs ratio, numbers needed to treat (NNT), absolute risk reductionTRANSCRIPT
-
Basic Components
Objectives Search process QUORUM diagram Test of heterogeneity and model used for
meta-analysis Types of data and effect size Risk of biases assessment Forest Plot
-
Objectives
A statement of questions being addressed with reference to
[P = participants/populations] [I=interventions] [C=comparisons] [O=outcomes], and [S = study design]
-
PICO Questions
P = Population/Problem I = Intervention/indicator/
index test C = Comparator O = Outcome
-
P = Population
Population What are the important characteristics
that define the participants? What is the setting? (hospital,
community, etc) How is this disease/condition defined? Who make the diagnosis?
-
P = Population
Population Are there any co-morbidities to be excluded? Are there any other types of people that
should be excluded or considered in the review (because of the intervention in a different way)?
How will studies involving only a subset of relevant participants be handled?
-
I = Intervention
Intervention Does the intervention have variations (e.g.
dosage, components, mode of delivery, personnel, frequency, duration or timing)?
Are all variations to be included (e.g. is there a minimum dose or components without which intervention may not be expected to work in the same way)?
-
C = Comparison
Comparison What are you interested in comparing the
intervention to (e.g. an active intervention, no intervention or placebo, any available comparison)?
Refer back to primary question of review
-
O = Outcomes
Outcomes What are the important outcomes that you
plan to measure in your review? What are your primary outcomes (max 3)? What are your secondary outcomes? Have you included possible adverse effects?
-
O = Outcomes
Outcomes How should the outcomes be measured (e.g.
validated tools)? Are there important time points at which
outcomes should be measured (e.g. long enough to expect an observable effect)?
Have you included outcomes relevant to all potential decision-makers?
-
CRITICAL APPRAISALS
STUDY QUESTION
CRITICAL APPRAISAL
P= Population R = Recruitment I= Intervention A = Allocation
C = Comparison M = Maintenance O = Outcome M = Measurement
B = Blinding O = Objective outcomes
-
CRITICAL APPRAISAL
Study question: Population Critical appraisal: recruitment Were the subjects representative of the target
population? Larger studies are preferred.
For continuous outcomes (e.g. BP, weight, etc) ! 50 100 might be sufficient
For binary outcomes (e.g. AMI) ! numbers depend on how common it is.
Common events ! hundreds would be enough. Rare events ! may require thousands to capture the events
-
CRITICAL APPRAISAL
Study question: Intervention Critical appraisal: Allocation
It is vital that the groups are matched as closely as possible in every way except for the intervention (or exposure or other indicator).
-
CRITICAL APPRAISAL
Study question: Intervention Critical appraisal: Allocation
If the groups are not comparable to begin with, then any difference in outcomes may be due to one of the non-matched characteristics (or confounding factors), rather than due to the intervention (or exposure or other indicator) under consideration.
-
CRITICAL APPRAISAL
Study question: Intervention Critical appraisal: Allocation For example, ways in which groups could differ:
Age Gender Smoker/non-smoker Disease severity Other risk factors, etc The most important matching factors are those that could
predict the outcomes of the condition, which are often most related to the severity of the illness
-
CRITICAL APPRAISAL
Study question: Intervention Critical appraisal: Allocation Blinding process: must ensure that neither
the trial subjects nor the investigators can influence the group each person ends up in (allocation concealment)
Otherwise, this could introduce selective allocations, skewing the groups, or bias in reporting of outcomes
-
CRITICAL APPRAISAL
Study question: Comparison Critical appraisal: Maintenance Once comparable groups have been set up,
it is important that they stay that way. The study groups should be managed so
that the only difference between the groups is the factor being tested (e.g. Drug A vs no drug A, smoking vs no smoking).
-
CRITICAL APPRAISAL
Study question: Comparison Critical appraisal: Maintenance Groups should be treated exactly the same
as the experimental group in every respect except for the factor being tested.
Also, ensure identical measurement strategy equipment, methods, assessors
-
CRITICAL APPRAISAL
Study question: Comparison Inevitably, some subjects do drop out or lost
during follow-up. This can be serious when the remaining groups
may no longer be comparable. Therefore, check that:
Subjects at the start = subjects at the end (majority has been accounted for)
Subjects are analyzed in the groups that they started out in (the intention-to-treat principle)
-
Intention-To-Treat
The Intention-to-Treat principle Once a subject is randomized, he or she
should be analyzed in the group they are randomized to, even if they never receive the treatment, discontinue the trial or cross over to the other groups.
-
CRITICAL APPRAISAL
Study question: Comparison Critical appraisal: Maintenance The results can also be biased if the subjects
are not followed up for long enough for relevant outcomes to be revealed in both groups.
It is therefore important that the subjects are followed up until the relevant outcome occurs or until death (cohort studies).
-
CRITICAL APPRAISAL
Study question: Outcome Critical appraisal: Measurement Were the outcomes measured with: Blinded subjects and assessors and/or Objective outcomes? Measurement bias can be reduced if the
outcomes measured are objective (e.g. BP) rather than subjective (e.g. feeling better)
-
The EQUATOR Network
http://www.equator-network.org/
-
The EQUATOR Network
-
PRISMA Statement
-
PRISMA Statement
-
Search process
Where do the information sources come from?
For each database searched, authors should report the database, platform, or provider (such as Ovid, Dialog, PubMed) and the start and end dates for the search of each database
What are the search terms?
-
Example
Studies were identified by searching electronic databases, scanning reference lists of articles and consultation with experts in the field..No limits were applied for language and foreign papers were translated. This search was applied to Medline (1966 Present), CancerLit (1975 Present), and adapted for Embase (1980 Present)..and Pre-Medline electronic databases. Cochrane and DARE (Database of Abstracts of Reviews of Effectiveness) databases were reviewedThe last search was run on 19 June 2001. In addition, we handsearched contents pages of Journal of Clinical Oncology 2001, together with abstracts printed in these journals 1999 - 2001.
-
QUORUM diagram
The Quality of Reporting of Meta-Analyses (QUORUM) guidelines contain a listing of essential steps and items that must be included in a well-conducted meta-analysis
-
QUORUM diagram
-
Test of Heterogeneity
Heterogeneity is a term used to describe variability among studies
Two sources of heterogeneity in a meta-analysis A. Within-study variability: Due to sampling variability. Always present because every study uses different samples
-
Test of Heterogeneity
B. Between-studies variability Is the true heterogeneity Influenced by numerous characteristics that
vary among the studies (e.g., variations in study design, variations in treatment, etc)
-
Test of Heterogeneity
When the studies results only differ by the sampling error (homogeneous case) ! a fixed-effects model is used
When the study results differ by more than the sampling error, which means, both within- and between-studies variability (heterogeneous case), ! random-effects model is used
-
Types of Data and Effect Size
Type of data Type of effect size Continuous data Mean difference
Standardized mean difference
Dichotomous data Odds ratio Risk ratio/relative risk
-
Classifications of biases
Type of bias Description
Selection bias Systematic differences between baseline characteristics of the groups that are compared
Performance bias Systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest.
Detection bias Systematic differences between groups in how outcomes are determined.
Attrition bias Systematic differences between groups in withdrawals from a study.
Reporting bias Systematic differences between reported and unreported findings.
http://www.cochrane.org/handbook
-
No matter how carefully a sample is selected to be a fair and unbiased representation of the population,
relying on information from a sample will always lead to some level of uncertainty. CI expresses this
uncertainty.
Why Confidence Interval?
-
Narrow vs Wide CI
Generally when a confidence interval is very wide, it is an indication of an inadequate sample size
A narrow confidence interval (CI) implies high precision; we can specify plausible values to within a tiny range.
A wide confidence interval implies poor precision; we can only specify plausible values to a broad and uninformative range.
-
Why Confidence Interval?
Important Question:
Does the interval contain a value that implies no change
or no effect?
-
When CI contains Null Value
With a confidence interval for a difference look to see whether that interval includes zero.
With a confidence interval for a ratio, look to see whether that interval contains one. Null Value
Red line : CI that contains the null value implying no statistically significant change. Blue and Black lines: statistical significant effects
-
Forest Plot
A forest plot displays effect estimates and confidence intervals for both individual studies and meta-analyses. Each individual study is represented by a square at the point estimate of intervention effect with a horizontal line extending either side of the block.
-
Forest Plot
Depending on the outcome in the studies, the CI can be (a) the relative likelihood of a given outcome for the intervention group vs the control group (e.g., live/die, success/failure) is measured by OR), or (b) the difference between treatment & control means (e.g. BP)
-
Forest Plot A vertical line is typically displayed to indicate no effect or null value, e.g. an OR = 1 or a mean difference of 0. Where ratios are presented (e.g. OR), null value is 1 and the scales are logarithmic. When the difference between means are presented, null value is zero (0) and the scales are normal.
-
Forest Plot
-
Effect of probio;cs on the risk of an;bio;c associated diarrhoea
Akobeng AK. Understanding systematic reviews and meta-analysis. Archives of disease in childhood. 2005;90(8):845-8.
Forest Plot
-
Understanding Risk vs Odds
-
Analyzing Dichotomous Outcomes
Risk vs Odds: Dichotomous Outcomes = when the outcome for every par@cipant is either one of two possibili@es or events; e.g., alive or dead, pain or no pain, pregnant or non-pregnant Expressing chance in terms of
RISK and ODDS
-
Risk vs Odds
Example: 12 people drank coffee
3 developed tachycardia
RISK of developing tachycardia?
-
Risk vs Odds
Risk:
3 developed tachy 12 who could have one
= = 25%
Risk = Number of events of interest Total number of observations
-
Risk vs Odds
Example: 12 people drank coffee
3 developed tachycardia
Odds of developing tachycardia?
-
Risk vs Odds
Odds (Total: 12)
3 developed tachy 9 no tachy
= 1/3 = 33.3%
Odds = Number of event of interest Number of observations WITHOUT the event
-
Risk vs Odds
Expressing in words Risk: the chance of tachycardia is one in four or 25%
Odds: the chance of tachycardia is one third (1/3) of the chance of no headache or
Odds: for every one person who develops tachycardia, three persons do not.
-
Do Risk and Odds Differ Much?
Scenario 1 N = 100; developed tachycardia: 5 Risk: 5/100 (5.00%), Odds: 5/95 (5.26%) Scenario 2 N = 100, developed tachycardia: 60 Risk: 60/100 (60%), Odds 60/40 (150%)
If number of observed events is BIG, risk and odds will differ a lot
If number of observed events is SMALL, risk and odds do not differ much
-
Example:
In patients with ACS (population), does the use of LMWH (treatment) decrease mortality and morbidity (outcomes) compared to UFH (control)?
-
Ref: Cohen M, Demers C, Gurfinkel EP, Turpie AG, Fromell GJ, Goodman S, Langer A, Califf RM, Fox KA, Premmereur J, Bigonzi F. A
comparison of low-molecular-weight heparin with unfractionated heparin for unstable coronary artery disease. Efficacy and Safety of
Subcutaneous Enoxaparin in Non-Q-Wave Coronary Events Study Group. N Engl J Med. 1997 Aug 14;337(7):447-52.
EFFICACY AND SAFETY OF SUBCUTANEOUS ENOXAPARIN IN
NONQ-WAVE CORONARY EVENTS (ESSENCE) STUDY GROUP 1997
-
Death at 48 hours
No death at 48 hours
Risk
Enoxaparin (n = 1607)
8 1599 0.005
UFH (n = 1564)
7 1557 0.004
Ref: Cohen M, Demers C, Gurfinkel EP, Turpie AG, Fromell GJ, Goodman S, Langer A, Califf RM, Fox KA, Premmereur J, Bigonzi F. A
comparison of low-molecular-weight heparin with unfractionated heparin for unstable coronary artery disease. Efficacy and Safety of
Subcutaneous Enoxaparin in Non-Q-Wave Coronary Events Study Group. N Engl J Med. 1997 Aug 14;337(7):447-52.
RISK RATIO (RELATIVE RISK) = 0.005/0.004 = 1.13
-
Death at 48 hours
No death at 48 hours
Odds
Enoxaparin (n = 1607)
8 1599 0.005
UFH (n = 1564)
7 1557 0.004
Ref: Cohen M, Demers C, Gurfinkel EP, Turpie AG, Fromell GJ, Goodman S, Langer A, Califf RM, Fox KA, Premmereur J, Bigonzi F. A
comparison of low-molecular-weight heparin with unfractionated heparin for unstable coronary artery disease. Efficacy and Safety of
Subcutaneous Enoxaparin in Non-Q-Wave Coronary Events Study Group. N Engl J Med. 1997 Aug 14;337(7):447-52.
ODDS RATIO = 0.005/0.004 = 1.11
-
Risk Ra@o:
INTERVENTION RISK CONTROL RISK
Odds Ra@o:
INTERVENTION ODDS CONTROL ODDS
-
MI at 48 hours
No MI at 48 hours
Risk
Enoxaparin (n = 1607)
11 1596 0.007
UFH (n = 1564)
14 1550 0.009
Ref: Cohen M, Demers C, Gurfinkel EP, Turpie AG, Fromell GJ, Goodman S, Langer A, Califf RM, Fox KA, Premmereur J, Bigonzi F. A
comparison of low-molecular-weight heparin with unfractionated heparin for unstable coronary artery disease. Efficacy and Safety of
Subcutaneous Enoxaparin in Non-Q-Wave Coronary Events Study Group. N Engl J Med. 1997 Aug 14;337(7):447-52.
RISK RATIO (RELATIVE RISK) = 0.78
-
MI at 48 hours
No MI at 48 hours
Odds
Enoxaparin (n = 1607)
11 1596 0.007
UFH (n = 1564)
14 1550 0.009
Ref: Cohen M, Demers C, Gurfinkel EP, Turpie AG, Fromell GJ, Goodman S, Langer A, Califf RM, Fox KA, Premmereur J, Bigonzi F. A
comparison of low-molecular-weight heparin with unfractionated heparin for unstable coronary artery disease. Efficacy and Safety of
Subcutaneous Enoxaparin in Non-Q-Wave Coronary Events Study Group. N Engl J Med. 1997 Aug 14;337(7):447-52.
ODDS RATIO = 0.78
-
MI at 48 hours
No MI at 48 hours
Risk
Enoxaparin (n = 1607)
11 1596 0.007
UFH (n = 1564)
14 1550 0.009
Ref: Cohen M, Demers C, Gurfinkel EP, Turpie AG, Fromell GJ, Goodman S, Langer A, Califf RM, Fox KA, Premmereur J, Bigonzi F. A
comparison of low-molecular-weight heparin with unfractionated heparin for unstable coronary artery disease. Efficacy and Safety of
Subcutaneous Enoxaparin in Non-Q-Wave Coronary Events Study Group. N Engl J Med. 1997 Aug 14;337(7):447-52.
ABSOLUTE RISK REDUCTION: 0.009 0.007 = 0.002
-
Number Needed To Treat (NNT)
is the number of patients you need to treat to prevent one additional bad outcome (death, stroke, etc.).
For example, if a drug has an NNT of 5, it means you have to treat 5 people with the drug to prevent one additional bad outcome.
The ideal NNT is 1. The higher the NNT, the less effective is the
treatment
-
Number Needed To Treat (NNT)
Calculation To calculate the NNT, you need to know the
Absolute Risk Reduction (ARR); the NNT is the inverse of the ARR:
NNT = 1/ARR
Where ARR = CER (Control Event Risk) - EER (Experimental Event Risk)
NNTs are always rounded up to the nearest whole number
-
MI at 48 hours
No MI at 48 hours
Risk
Enoxaparin (n = 1607)
11 1596 0.007
UFH (n = 1564)
14 1550 0.009
Ref: Cohen M, Demers C, Gurfinkel EP, Turpie AG, Fromell GJ, Goodman S, Langer A, Califf RM, Fox KA, Premmereur J, Bigonzi F. A
comparison of low-molecular-weight heparin with unfractionated heparin for unstable coronary artery disease. Efficacy and Safety of
Subcutaneous Enoxaparin in Non-Q-Wave Coronary Events Study Group. N Engl J Med. 1997 Aug 14;337(7):447-52.
NNT = 1/(ABSOLUTE RISK REDUCTION) = 1/0.002 = 5000
-
Number Needed To Treat (NNT)
Example: The ARR is therefore the amount by which
your therapy reduces the risk of the bad outcome. For example, if your drug reduces the risk of a bad outcome from 50 per cent to 30 per cent, the ARR is: ARR = CER - EER = 0.5 - 0.3 = 0.2 (20%)
therefore NNT = 1/ARR = 1/0.2 = 5
-
Choosing a Statistical Test
Type of data
Quantitative analysis
Normal Distribution
T-test
>2 samples: One-way ANOVA
Skewed/kurtotic/not non-
parametric
Wilcoxon signed test/
Mann-Whitney
> 2 samples: Kruskal-Wallis
Qualitative analysis
Chi-square/Fisher exact (if
sample size small)
-
Extra Slides
-
Test of Heterogeneity
Methods to test for heterogeneity: Q test: If Q test cannot reject the homogeneity
hypothesis ! use fixed-effects model because it is assumed that the estimated effect sizes only differ by sampling error
If reject homogeneity hypothesis ! use random-effects model that includes both within- and between-studies variability.
-
Test of Heterogeneity
A problem with Q test is that it has poor power to detect true heterogeneity among studies when the meta-analysis includes a small number of studies and excessive power to detect negligible variability with a high number of studies
-
Test of Heterogeneity
The other method is to use I2 index A meta-analysis with I2 = 0 means that all
variability in effect size estimates is due to sampling error within studies.
A meta-analysis with I2 = 50 means that half of the total variability among effect sizes is caused not by sampling error, but by true heterogeneity between studies.
-
Test of Heterogeneity
If the confidence interval around I2 contains the 0% value, then homogeneity hypothesis is assumed.