Caspiano
Ivanaaaa
systematic_review.pdf
Systematic review and meta analysis
Master of Public Health Systematic Review Workshop
Learning objectives
Distinguish systematic reviews from narrative reviews
Design a systematic review in concordance with the PRISMA
guideline.
Identify the additional elements in meta-analysis (forest plots,
funnel plots, test of heterogeneity etc)
Detect common problems that may affect meta-analysis
Evaluate the strengths and limitations of systematic reviews
and meta analyses
Introduction
What is a systematic review?
The application of strategies that limits bias in the assembly,
critical appraisal, and synthesis of all relevant studies on a
specific topic (Porta, 2008)
Porta MS, International Epidemiological Association. A dictionary of epidemiology. 5th ed. Oxford ; New York: Oxford University Press; 2008.
Introduction
Why do we need systematic reviews?
Introduction
Why do we need systematic review?
Lau J et
al.,
NEJM.
1992;
327:248-
254
Introduction
Why do we need systematic review?
Antman EM et al.,
JAMA. 1992; 268:
240-248
Narrative vs. Systematic
Systematic review is more than just an essay
Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376-380.
Introduction
Value of Systematic Reviews
in healthcare
Evidence-based practice
Better evidence for decision
making in healthcare compared
to a particular study such as
Effectiveness of
interventions (drugs,
screening)
Lifestyle factors on disease
progression
Centre for Evidence Based Medicine OU. Level of Evidence. 2009. (www.cebm.net). (Accessed 25/03 2011).
Introduction
~2,500 systematic reviews produced annually
Quality of reporting inconsistent (Moher, 2007)
(Juni, 2009)
Juni P, Egger M. PRISMAtic reporting of systematic reviews and meta-analyses. Lancet. 2009;374:1221-1223. Moher D, Tetzlaff J, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.
Introduction
Are there any guidelines to follow?
Introduction
Which guideline should I follow?
Depending on
Your research question
Type of studies to be included
Although they are more or less the same…
Some common guidelines include
PRISMA (Preferred Reporting Items for Systematic Reviews
and Meta-Analyses)
MOOSE (Meta-analysis of Observational Studies in
Epidemiology)
Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 2009;6(7):e1000100. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000;283(15):2008-12.
PRISMA (checklist) for reviews
Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for
reporting systematic reviews and meta-analyses of studies that
evaluate health care interventions: explanation and elaboration. PLoS
Med 2009;6(7):e1000100.
1. Title
Identify the report as a systematic review
Mortality in randomized trials of antioxidant supplements for
primary and secondary prevention: systematic review and meta-
analysis
2. Structured summary (Abstract)
Introduction
Context
Methods
Data sources; study selection
Results
Summary of results (i.e. number of studies retrieved etc)
Conclusions
Implications
2. Structured summary (Abstract)
An example from an international journal
2. Structured summary (Abstract)
An example from an international journal
Risnes KR, Vatten LJ, Baker JL, et al. Birthweight and mortality in adulthood: a systematic review and meta-analysis. Int J Epidemiol 2011;40(3):647-61.
3. Rationale (Introduction)
Describe the rationale for the review in the context of what
is already known (i.e. literature review)
Things to be included:
Explain the importance of the review question
Current state and knowledge
Are there any conflicts between studies?
If so, why?
What this review aims to add
4. Objective (Introduction)
What is the objective of the review
To examine whether topical or intralumnial antibiotics reduce
catheter-related bloodstream infection, we reviewed
randomized, controlled trials that assessed the efficacy of these
antibiotics for primary prophylaxis against catheter-related
bloodstream infection and mortality compared with no
antibiotic therapy in adults undergoing hemodialysis
5. Eligibility Criteria of the study
(Methods)
Specify the study characteristics and report characteristics to
be included in the review
Things to be considered as eligibility criteria:
PICOS approach
Types of Participants (age, ethnicity, sex)
Types of Interventions (lifestyle, intervention, exposure)
Types of Comparators (control group)
Types of Outcome (how the outcome was measured, categorical or
continuous)
Types of Studies (observational/experimental)
Language of the studies (English, Chinese, German…)
6. Study selection (Methods)
State the process for selecting studies (i.e. how you identify
studies to be included in the review)
Things to be considered:
Which database to be used (PubMed, ISI Web of knowledge,
EMBASE, reference lists of retrieved studies)
Considering other literatures
Unpublished studies
Trial registration
Dissemination reports
Time frame (identifying studies from when to when)
**If you find a lot of new studies from the reference list, that
mean indicate a problem with the research and the review
might not be very systematic.
6. Study selection (Methods)
6. Study selection (Methods)
Things to be considered:
Make use of the limits to screen out irrelevant references
http://www.ncbi.nlm.nih.gov/pubmed/?term=statin+AND+cardiovas
cular+diseases
How to search the selected databases?
Keyword search
Medical Subject Heading (MeSH)
o http://www.nlm.nih.gov/mesh/meshhome.html
Other related words
* (drink* can refer to any word that has drink at the beginning, such as
drinking, drinker…)
Boolean operators
o “AND” “OR” “NOT”
o (marijuana OR cannabis) AND (therapeutic use OR medicinal use)
NOT (Smoking)
AND vs. OR vs. NOT vs. ( )
PubMed search: (2 January, 2014)
- marijuana OR cannabis
19,411 results
- marijuana AND cannabis
12,513 results
- marijuana OR cannabis AND therapeutic use
5,122 results
- therapeutic use AND marijuana OR cannabis
13,903 results
- therapeutic use AND (marijuana OR cannabis)
5,122 results
Search Fields
statin AND clinical trials AND (Lancet [TA] OR NEJM [TA]
OR BMJ [TA] OR JAMA [TA])
6. Study selection (Methods)
List out all combinations in detail
(alcohol use OR drinking) AND (cardiovascular disease OR heart disease)
Ronksley PE, Brien SE, Turner BJ, et al. Association of alcohol consumption with selected cardiovascular disease outcomes: a systematic review and meta-analysis. BMJ 2011;342:d671.
6. Study selection (Methods)
Other ways of reporting on the search terms…
Listing out all the words
Search strategy available from authors, but why not put them in the Methods if they are not complicated
Risnes KR, Vatten LJ, Baker JL, et al. Birthweight and mortality in adulthood: a systematic review and meta-analysis. Int J Epidemiol 2011;40(3):647-61.
Briel M, Ferreira-Gonzalez I, You JJ, et al. Association between change in high density lipoprotein cholesterol and cardiovascular disease morbidity and mortality: systematic review and meta-regression analysis. BMJ 2009;338.
7. Study assessment (Methods)
Assessments of selected studies
Quality
How good the authors have tried to do. (example: confounder
adjustments, presence of control outcome, control exposure for
specificity cross-checking)
Risk of bias
How the results have been affected due to methodological issues, such as
absence of blinding, heterogeneity of certain categories (example:
questions did not separate abstainers and ex-drinkers)
STROBE (observational studies)
CONSORT
(RCTs)
8. Number of reviewers
Number of reviewers
Reviews are often conducted by 2 independent reviewers to
improve objectiveness. Discrepancies will be resolved by
consensus.
9. Report study selection process
(Results)
Things to report
Number of studies
Screened and assessed for eligibility
Included in the review
With reasons for the above items
Briel M, Ferreira-Gonzalez I, You JJ, et al. Association between change in high density lipoprotein cholesterol and cardiovascular disease morbidity and mortality: systematic review and meta-regression analysis. BMJ 2009;338.
9. Report study selection process
(Results)
A flow chart can help a reader to understand the selection
process easily
9. Report study selection process
(Results)
Ronksley PE, Brien SE, Turner BJ, et al. Association of alcohol consumption with selected cardiovascular disease outcomes: a systematic review and meta-analysis. BMJ 2011;342:d671.
10. Study characteristics (Results)
Report details of each study in the form of a table (either as
tables or appendices)
Citation
Time and place
PICOS
Types of Participants (age, ethnicity, sex)
Types of Interventions (lifestyle, intervention, exposure)
Types of Comparators (control group)
Types of Outcome (how the outcome was measured, categorical or
continuous)
Types of Studies (observational/experimental)
Do not mix up the results from observational studies and randomized
controlled trials
10. Study characteristics (Results) Estimates
Follow up
Confounder adjustments, if any
Quality of the study
10. Study characteristics (Results)
Whincup PH, Kaye SJ, Owen CG, et al. Birth weight and risk of type 2 diabetes: a systematic review. JAMA 2008;300(24):2886-97.
10. Study characteristics (Results)
Summarize the characteristics of which data were extracted
Grouped under some characteristics such as types of studies, or
settings:
Study size (such as the range: 50-20,000)
PICOS
Follow up period (for prospective studies and experimental
studies)
What studies show concerning the association between
exposure and outcome
10. Study characteristics (Results)
Cowling BJ, Zhou Y, Ip DKM, Leung GM, and Aiello AE. Face masks to prevent transmission of influenza virus: a systematic review. 2010. Epidemiol. Infect. 138:449-456
11. Summary of evidence
(Discussion)
Summarize the main findings in general
What does your review tell?
Eg. Majority of the studies included in this review suggested a positive
association between exposure and outcome
Strength of evidence
Critically appraise the evidence rather than regurgitating the
results
Whether they are methodologically sound
Consistent? If not, why?
Eg 1. There were discrepancies between observational studies and
randomized controlled trials, possibly because of the…….
Eg. 2. Inconsistent findings among studies could be a reflection of
unstandardized definition of the exposure. For example….
11. Summary of evidence
(Discussion)
Relevance to key groups
Health care providers
Users
Policy makers
12. Limitations (Discussion)
Limitations of the studies included
Study designs
Measurements of exposures and outcomes
Risk of bias and confounding
Quality
Doshi P, Jones M, Jefferson T. Rethinking credible evidence synthesis. BMJ 2012;344:d7898.
13. Conclusions
General interpretation of the results in the context of other
evidence
Implications for future research
Harder T, Roepke K, Diller N, et al. Birth weight, early weight gain, and subsequent risk of type 1 diabetes: systematic review and meta-analysis. Am J Epidemiol 2009;169(12):1428-36.
Meta analysis
Meta-analyses
A sub-set of systematic reviews
Systematic reviews with a statistical component
Analysis of results from separate studies (i.e. pooling of results)
Requiring studies to have common exposures/interventions and
outcomes
Examples of meta analysis
Meta analysis of birth weight and risk of Type 1 diabetes
Harder T et al., Am J Epidemiol 2009. 169:1428-1436
Examples of meta analysis
Meta analysis of chocolate consumption and cardiovascular
diseases
Buitrago-Lopez
et al., BMJ.
2011. 343:d4488
Forest plot
A plot that summarizes
different estimates of
the same quantity
However, are these
variation in the
estimates due to
chance or other
factors?
Ronksley PE, Brien SE, Turner BJ, et al. Association of alcohol consumption with selected cardiovascular disease outcomes: a systematic review and meta-analysis. BMJ 2011;342:d671.
Heterogeneity
Studies included in a systematic review can be different due
to
Clinical diversity (Variability in terms of PICO)
Methodological diversity (The “S” in PICOS)
Depending on heterogeneity, different models will be used to
pool the study estimates together
Example of heterogeneity
Higgins JPT and Thompson SG. Statist Med. 2002; 21:1539-1558
Homogeneous Moderate heterogeneity
Example of heterogeneity
Higgins JPT and Thompson SG. Statist Med. 2002; 21:1539-1558
Heterogeneous Presence of outlier
Test for heterogeneity
Cochran’s Q test
P value<0.05 means
heterogeneity
Poor at detecting true
heterogeneity among
studies as significant when
number of studies is small
Excessive power when
there are many studies,
especially when those
studies are large
I2
Higher % means higher
heterogeneity
Can be accompanied by an
uncertainty interval
Does not depend on the
number of the studies in
the meta analysis
Higgins JPT et al., BMJ 2003;327:557-60
Meta-regression analysis
Examine heterogeneity using regression approach
Each study becomes a “subject”
Exposure variable would be the elements of each study, such as
follow up time in each cohort study, or study design (case
control/ prospective)
Analysis is weighted by the standard error of the coefficient
estimate
Outcome would be the coefficient estimate in each study
Variables associated with the outcome will be stratified in the
meta analysis
One issue: False positive due to post-hoc multiple
comparisons
Fixed effect vs. Random effect
Fixed effect model. We
assume
all studies are functionally
identical, and hence having
a common effect size
In practice, this is rarely
plausible because of clinical
and methodological
variation in different
studies
Mathematically, this is a
special case of random
effect model
Random effect model. We
assume
A distribution of true
effect sizes because of the
differences across studies
More plausible than fixed
effect models and will
yield the identical results as
the fixed effect model in
the absence of
heterogeneity
Could be problematic if
the studies are extremely
heterogeneous
Fixed effect vs. Random effect
Fixed effect Random effect
Borenstein et al., Res Syn
Meth 2010; 1:97-111
Fixed effect vs. Random effect
Borenstein et al., Res Syn
Meth 2010; 1:97-111
Funnel plot
A scatter plot of the effect estimates from individual studies
against some measure of each study’s size or precision
If there is absence of bias and between study heterogeneity,
the scatter will be due to sampling variation alone and the
plot will resemble a symmetrical inverted funnel
Sterne JAC et al., BMJ 2011; 342:d4002
Heterogeneity as a cause of
asymmetry
Heterogeneity, defined as the differences between study
results beyond those attributable to chance, could be driven
by
Clinical differences between studies (settings, types of
participants, implementation of the intervention)
Correlation between study sizes and intervention effects
Differences in methodological quality
Reporting bias as a cause of
asymmetry
Dissemination of research findings is influenced by the nature
and direction of results because
Fails to locate an eligible study because all information about it
is suppressed or hard to find (publication bias)
A located study may not provide usable data for the outcome of
interest because the results are considered not interesting by the
authors (selective outcome reporting)
A located study may provide biased results for some outcome
(selective analysis reporting)
Chance as a cause of asymmetry
Role of chance is important especially when the number of
studies included are small, and the test for heterogeneity may
subject to false positive findings
Statistical tests for heterogeneity
Examines whether the association between estimated
intervention effects and a measure of study size is greater
than might be expected to occur by chance
Example: Egger’ test
Limitations
Low statistical power leading to false positive, especially when
the number of studies included is less than 10
Example of funnel plot
Sterne JAC et al., BMJ 2011; 342:d4002
Some methods to avoid
Qualitative tally (Vote counting)
Example: Of 10 studies to date, 6 have found a positive
association, 3 have found a negative association, and 1 found
null association; hence preponderance of evidence favors
positive association
Problem: No association can be driven by a lack of sample size;
Studies showing positive associations can be biased in the same
way.
Quality scoring
Example: Weighting studies on a quality score, which the score
is based on some subjectivity based on features of studies
Problem: Submerges important information by combining
disparate study features into a single score
Strengths of systematic review and
meta analysis
More objective assembly of existing evidence concerning a
research question compared to narrative review
Increased power to detect important differences which might
not be detected in small individual studies
Limitations of systematic review
and meta analysis
False impression of consistency across study results even
though the individual study results are too imprecise to
reveal inconsistencies (Heterogeneity)
Meta analysis cannot compensate for the limits of non-
experimental data for making inferences about causal effects
Failure to include all data due to systematic failure to publish
or report certain types of results
Some examples of controversial
findings about meta analyses
Reporting bias on meta analyses
Hart B, Lundh A and
Bero L. BMJ
2011;344:d7202
Some examples of controversial
findings about meta analyses
Funding sources and its relation with the results
Ridker and Torres. JAMA 2006;295:2270-74
Take home messages
Systematic reviews and meta analyses allows assessment of
the exposure-outcome relation based on an objective search
of the existing evidence
Guidelines are available for proper conduct of systematic
reviews and meta analyses
Systematic reviews and meta analyses cannot compensate for
the limits of non- experimental data for making inferences
about causal effects
