One of the principles of evidence based practice is the aggregation of all available research on a topic in order to make evidence based decisions on the effectiveness of interventions, diagnostic accuracy, or quality improvement. Systematic reviews (SRs) and meta-analyses are powerful tools to summarize evidence and are preferably conducted according to the PRISMA guidelines to ensure high quality.
- Moher D.
- Liberati A.
- Teztlaff J.
- Altman D.G.
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
Before starting a SR, find at least one co-investigator, because conducting a SR is a structured and transparent process, and all steps of the SR must be done in duplicate to minimize the risk of mistakes and bias. Consider registering your SR in the PROSPERO database of ongoing reviews to avoid unplanned duplication. Conducting a SR starts with formulating a clinical question, for example according to the PICO format (Patient, Intervention, Control, Outcome). As an example, one could investigate whether a Patient with a ruptured abdominal aortic aneurysm (RAAA) treated with endovascular aneurysm repair (EVAR) (Intervention) compared with open repair (Control) has better 30 day survival (Outcome). The PICO defines the search strategy that should be reported in detail, enabling other investigators to reproduce the search. It is mandatory to query multiple databases, such as Medline, Embase, and the Cochrane databases, since none of these are fully comprehensive. If possible, collaborate with a clinical librarian to increase the efficiency and yield of the search. Two independent investigators should formulate strict inclusion and exclusion criteria and select studies based on title and abstract. Studies may be excluded after reading the full text, whereas reference lists may identify additional studies. Present the selection process in a flow diagram.
Data from the included studies must be extracted independently by two investigators on a data extraction form. A very important part of a SR is the assessment of methodological quality of the included studies and identification of sources of bias. Bear in mind that not every study designed as a randomized controlled trial (RCT) is of high quality. The free Cochrane Collaboration's Review Manager software (http://tech.cochrane.org/revman/download
) includes a risk of bias assessment tool to identify strengths and weaknesses of RCTs. One could use the Newcastle–Ottawa scale for observational studies, and the QUADAS-2 for diagnostic research.
- Whiting P.F.
- Rutjes A.W.
- Westwood M.E.
- Mallet S.
- Deeks J.J.
- Reitsma J.B.
- et al.
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.
Next, independently extract baseline characteristics of included studies, for example of populations, interventions, and outcomes, to get an impression of the presence of clinical heterogeneity, and present these in a table. If according to your clinical expertise there are marked differences between studies regarding baseline characteristics or interventions (clinical heterogeneity), you could decide that pooling of data is inappropriate, or only appropriate for a subset of studies. In the absence of clinical heterogeneity the results of the individual studies can be combined to determine a summary estimate of effect (meta-analysis). For dichotomous outcomes, effect size may be expressed as a risk ratio (RR) or an odds ratio (OR) with 95% confidence interval (CI), and for continuous outcomes as a weighted mean difference with 95% CI. Review Manager and other meta-analysis software have a tool to identify statistical heterogeneity between study results, which can be expressed as the I2
(Inconsistency) index. As a rule of the thumb, an I2
< 30%, 30–60%, and >60% corresponds with low, moderate, and substantial heterogeneity. The debate on which model to pool data is best is still open. One could decide to always use a random effects model because this takes into account between study variation and yields a more conservative summary estimate of effect with a wider 95% CI than a fixed effects model. Others prefer a fixed effects model in case of low statistical heterogeneity, and use a random effects model when heterogeneity is moderate. Pooling data from observational studies is increasingly applied. It is important to realize that although the forest plots look exactly the same as those from RCTs, they do not show the lower level of evidence and the higher risk of bias. The forest plot from a SR comparing mortality after EVAR and open repair for RAAA clearly shows the effect of selection in observational studies and administrative registries, and the unbiased comparison in RCTs.
- Van Beek S.C.
- Conijn A.P.
- Koelemay M.J.
- Balm R.
Editor's choice – endovascular aneurysm repair versus open repair for patients with a ruptured abdominal aortic aneurysm: a systematic review and meta-analysis of short-term survival.
Incorporating the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system in a SR allows a better rating of quality of evidence and strength of recommendations per outcome.
- Guyatt G.H.
- Oxman A.D.
- Vist G.E.
- Kunz R.
- Falck-Ytter Y.
- Alonso-Coello P.
- et al.
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
Using this tool grades evidence as high, moderate, low, or very low. When evidence is graded as high for an outcome, further research is unlikely to change confidence in the estimate of effect. When evidence is rated as very low you have found a knowledge gap that can be filled by new research.
- Bosanquet D.C.
- Glasbey J.C.
- Williams I.M.
- Twine C.P.
Systematic review and meta-analysis of direct versus indirect angiosomal revascularization of infrapopliteal arteries.
Published online: December 19, 2015
© 2015 European Society for Vascular Surgery. Published by Elsevier Inc.