Advertisement

A Brief Guide to Performing Review and Meta-analysis of Rare Diseases, Procedures, and Other Low Frequency Pathology

Published:August 04, 2022DOI:https://doi.org/10.1016/j.ejvs.2022.07.045
      Meta-analysis publications have proliferated in the last 20 years. Publicly available, open access, easy to use software has allowed anyone to perform a systematic review and meta-analysis. Decision makers are increasingly demanding systematic review and/or meta-analysis as high quality evidence for the basis of policy and guidelines.
      • Antoniou G.A.
      • Bastos Goncalves F.
      • Bjorck M.
      • Chakfe N.
      • Coscas R.
      • Dias N.V.
      • et al.
      Editor's Choice – European Society for Vascular Surgery clinical practice guideline development scheme: an overview of evidence quality assessment methods, evidence to decision frameworks, and reporting standards in guideline development.
      As part of this process there is a widespread acceptance that meta-analysis of randomised trials is the apex of evidence. But from a data quality point of view this is not entirely true. Most clinicians understand that meta-analysis of non-randomised data is of inherently lower quality. The problem is that meta-analyses of all data qualities are often reported in the same way, which may lead to the results and conclusions misleading the reader. Even “higher quality” meta-analyses in higher impact journals fall into this trap and do not have consistent reporting quality.
      • Hameed I.
      • Demetres M.
      • Tam D.Y.
      • Rahouma M.
      • Khan F.M.
      • Wright D.N.
      • et al.
      An assessment of the quality of current clinical meta-analyses.
      As these studies are used by decision makers, it is extremely important to understand and communicate the limitations of a systematic review and meta-analysis within the study.
      Meta-analysis of the non-powered endpoints from randomised trials is a good example of lower quality data that may be misinterpreted as the apex of evidence. The powered outcome data from the randomised trial (usually the primary outcome) is the most valid and robust data, with non-powered outcomes becoming less useful as the event rate decreases. An example is the 2018 meta-analysis of mortality data following paclitaxel balloon angioplasty that changed policy internationally.
      • Katsanos K.
      • Spiliopoulos S.
      • Kitrou P.
      • Krokidis M.
      • Karnabatidis D.
      Risk of death following application of paclitaxel-coated balloons and stents in the femoropopliteal artery of the leg: a systematic review and meta-analysis of randomized controlled trials.
      Even though this was a “meta-analysis of randomised trials”, the trials were small in terms of patient numbers and were powered to detect patency loss of the treated artery, not death, which happened far more infrequently. Meta-analysing the non-powered, low event rate mortality outcome is reasonable, but should be clearly communicated with the caveats of low quality data that may be prone to a number of biases. In this study the authors’ assessment of bias using the Cochrane risk of bias 1 tool for randomised trials was favourable; however, that assessment only really applies to the powered outcome of the trial, not the mortality data.
      • Higgins J.P.
      • Altman D.G.
      • Gøtzsche P.C.
      • Jüni P.
      • Moher D.
      • Oxman A.D.
      • et al.
      The Cochrane Collaboration's tool for assessing risk of bias in randomised trials.
      This flaw is corrected in the risk of bias 2 tool, which came out after the paclitaxel study. This limitation can also be mitigated by performing a certainty analysis for each meta-analysed outcome using a framework such as GRADE (Grading of Recommendation, Assessment, Development and Evaluation).
      • Gagnier J.J.
      • Moher D.
      • Boon H.
      • Beyene J.
      • Bombardier C.
      Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature.
      GRADE allows authors to focus directly on the result extracted for an individual meta-analysis calculation (e.g., here mortality from a randomised trial powered for patency) and then state the quality, or certainty, of that result for context. This would have been very helpful for the paclitaxel meta-analysis. “There is low certainty evidence for an association between the application of paclitaxel coated balloons and a subsequent risk of death.” reads very differently from the published conclusion, “There is an increased risk of death following application of paclitaxel coated balloons and stents...”. Using the word “association” to acknowledge the lower quality result also changes the tone. The GRADE result would be different if the same randomised trials were used for a pooled patency result where the certainty would be higher.
      Meta-analysis of cohort or case series data comes with the same problems as non-powered data from randomised trials, as well as being more prone to publication and selection bias. The European Journal of Vascular and Endovascular Surgery is about to publish a systematic review and meta-analysis on the treatment of median arcuate ligament syndrome (MALS).
      • Metz F.
      • Blauw J.T.M.
      • Brusse-Keizer M.
      • Kolkman J.J.
      • Bruno M.J.
      • Geelkerken R.H.
      A scoping review on the efficacy of treatment of the median arcuate ligament syndrome.
      As a rare disease, this topic has a large number of low volume case series in the literature. Again, we are back to unpowered data with low event rates. In addition, studies describing rare diseases or procedures are highly likely to have publication bias from authors with either very good or very bad results, and selection bias because of individual clinical practice. In this situation it is important to look for clinical heterogeneity before making the decision to pool the data. Are the study results reasonable to pool, or is the clinical practice so different that you are including apples with oranges? There are ways to assess for common biases before an analysis begins,
      • Gagnier J.J.
      • Moher D.
      • Boon H.
      • Beyene J.
      • Bombardier C.
      Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature.
      commonly used examples are ROBINS-I and QUADAS-2.
      • Sterne J.A.
      • Hernán M.A.
      • Reeves B.C.
      • Savović J.
      • Berkman N.D.
      • Viswanathan M.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      ,
      • Whiting P.F.
      • Rutjes A.W.
      • Westwood M.E.
      • Mallett S.
      • Deeks J.J.
      • Reitsma J.B.
      • et al.
      QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.
      These are to non-randomised studies what the Cochrane risk of bias tool is for randomised, with QUADAS-2 being specific for primary diagnostic accuracy studies. If the decision is made to pool the data, these assessments feed into the GRADE framework to allow any bias detected to downgrade the certainty recommendation for the meta-analysis. In the current review on MALS, appraisal of the literature identified on systematic review with the QUADAS-2 tool suggested that none of the articles were of sufficient quality to be regarded as “low risk” for bias. This was due to ill defined and heterogeneous outcome measures in the studies, and the risk of confounding and bias. As a result, meta-analysis of the data was not performed by the authors. As no synthesis of data was done, the authors could not perform a GRADE evaluation of the findings.
      Sometimes when there is a large volume of low quality data in the literature a systematic review with meta-analysis is the wrong choice of study design. Scoping reviews may be used in this situation. While there remains no standardised definition, scoping reviews are broadly a means to identify the types, characteristics, concepts, or definitions of the evidence available for a topic.
      • Powell J.T.
      • Koelemay M.J.W.
      Systematic reviews of the literature are not always either useful or the best way to add to science.
      A good example was when the authors of the European Society for Vascular Surgery acute limb ischaemia guidelines wanted to provide an update in light of COVID 19.
      • Bjorck M.
      • Earnshaw J.J.
      • Acosta S.
      • Bastos Goncalves F.
      • Cochennec F.
      • Debus E.S.
      • et al.
      Editor's Choice – European Society for Vascular Surgery (ESVS) 2020 clinical practice guidelines on the management of acute limb ischaemia.
      They recognised there was a huge amount of low quality data published in a very short space of time on limb outcomes during COVID.
      • Jongkind V.
      • Earnshaw J.J.
      • Bastos Goncalves F.
      • Cochennec F.
      • Debus E.S.
      • Hinchliffe R.
      • et al.
      Editor's Choice – Update of the European Society for Vascular Surgery (ESVS) 2020 clinical practice guidelines on the management of acute limb ischaemia in light of the COVID-19 pandemic, based on a scoping review of the literature.
      A systematic review would have produced a vast number of studies too heterogeneous and biased for meaningful meta-analysis and difficult to summarise. The scoping review allowed them to understand the key concepts of the COVID limb outcomes literature and summarise it in a way that facilitated updated recommendations.
      • Jongkind V.
      • Earnshaw J.J.
      • Bastos Goncalves F.
      • Cochennec F.
      • Debus E.S.
      • Hinchliffe R.
      • et al.
      Editor's Choice – Update of the European Society for Vascular Surgery (ESVS) 2020 clinical practice guidelines on the management of acute limb ischaemia in light of the COVID-19 pandemic, based on a scoping review of the literature.
      So if a research group is considering summarising the literature for a rare disease or procedure the first question to ask is whether it should be a systematic or scoping review.
      • Powell J.T.
      • Koelemay M.J.W.
      Systematic reviews of the literature are not always either useful or the best way to add to science.
      If systematic, are the studies too heterogeneous for meta-analysis?
      • Gagnier J.J.
      • Moher D.
      • Boon H.
      • Beyene J.
      • Bombardier C.
      Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature.
      Then, finally, if meta-analysis is performed, the results must be presented clearly, communicating the limits of the inputted data, including a bias assessment for each relevant study outcome
      • Sterne J.A.
      • Hernán M.A.
      • Reeves B.C.
      • Savović J.
      • Berkman N.D.
      • Viswanathan M.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      ,
      • Whiting P.F.
      • Rutjes A.W.
      • Westwood M.E.
      • Mallett S.
      • Deeks J.J.
      • Reitsma J.B.
      • et al.
      QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.
      and a quality or certainty assessment for each meta-analysis.
      • Gagnier J.J.
      • Moher D.
      • Boon H.
      • Beyene J.
      • Bombardier C.
      Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature.
      Accurate, informed decisions can then be made by policy makers who will look to these important studies for guidance.

      Conflict of interest statement and funding

      None.

      References

        • Antoniou G.A.
        • Bastos Goncalves F.
        • Bjorck M.
        • Chakfe N.
        • Coscas R.
        • Dias N.V.
        • et al.
        Editor's Choice – European Society for Vascular Surgery clinical practice guideline development scheme: an overview of evidence quality assessment methods, evidence to decision frameworks, and reporting standards in guideline development.
        Eur J Vasc Endovasc Surg. 2022; 63: 791-799
        • Hameed I.
        • Demetres M.
        • Tam D.Y.
        • Rahouma M.
        • Khan F.M.
        • Wright D.N.
        • et al.
        An assessment of the quality of current clinical meta-analyses.
        BMC Med Res Methodol. 2020; 20: 105
        • Katsanos K.
        • Spiliopoulos S.
        • Kitrou P.
        • Krokidis M.
        • Karnabatidis D.
        Risk of death following application of paclitaxel-coated balloons and stents in the femoropopliteal artery of the leg: a systematic review and meta-analysis of randomized controlled trials.
        J Am Heart Assoc. 2018; 7e011245
        • Higgins J.P.
        • Altman D.G.
        • Gøtzsche P.C.
        • Jüni P.
        • Moher D.
        • Oxman A.D.
        • et al.
        The Cochrane Collaboration's tool for assessing risk of bias in randomised trials.
        BMJ. 2011; 343: d5928
        • Gagnier J.J.
        • Moher D.
        • Boon H.
        • Beyene J.
        • Bombardier C.
        Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature.
        BMC Med Res Methodol. 2012; 12: 111
        • Metz F.
        • Blauw J.T.M.
        • Brusse-Keizer M.
        • Kolkman J.J.
        • Bruno M.J.
        • Geelkerken R.H.
        A scoping review on the efficacy of treatment of the median arcuate ligament syndrome.
        Eur J Vasc Endovasc Surg. 2022 Sep 5; S1078-5884(22)00539-1 (Online ahead of print)https://doi.org/10.1016/j.ejvs.2022.08.033
        • Sterne J.A.
        • Hernán M.A.
        • Reeves B.C.
        • Savović J.
        • Berkman N.D.
        • Viswanathan M.
        • et al.
        ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
        BMJ. 2016; 355: i4919
        • Whiting P.F.
        • Rutjes A.W.
        • Westwood M.E.
        • Mallett S.
        • Deeks J.J.
        • Reitsma J.B.
        • et al.
        QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.
        Ann Intern Med. 2011; 155: 529-536
        • Powell J.T.
        • Koelemay M.J.W.
        Systematic reviews of the literature are not always either useful or the best way to add to science.
        EJVES Vasc Forum. 2022; 54: 2-6
        • Bjorck M.
        • Earnshaw J.J.
        • Acosta S.
        • Bastos Goncalves F.
        • Cochennec F.
        • Debus E.S.
        • et al.
        Editor's Choice – European Society for Vascular Surgery (ESVS) 2020 clinical practice guidelines on the management of acute limb ischaemia.
        Eur J Vasc Endovasc Surg. 2020; 59: 173-218
        • Jongkind V.
        • Earnshaw J.J.
        • Bastos Goncalves F.
        • Cochennec F.
        • Debus E.S.
        • Hinchliffe R.
        • et al.
        Editor's Choice – Update of the European Society for Vascular Surgery (ESVS) 2020 clinical practice guidelines on the management of acute limb ischaemia in light of the COVID-19 pandemic, based on a scoping review of the literature.
        Eur J Vasc Endovasc Surg. 2022; 63: 80-89

      Linked Article

      Comments

      Commenting Guidelines

      To submit a comment for a journal article, please use the space above and note the following:

      • We will review submitted comments as soon as possible, striving for within two business days.
      • This forum is intended for constructive dialogue. Comments that are commercial or promotional in nature, pertain to specific medical cases, are not relevant to the article for which they have been submitted, or are otherwise inappropriate will not be posted.
      • We require that commenters identify themselves with names and affiliations.
      • Comments must be in compliance with our Terms & Conditions.
      • Comments are not peer-reviewed.