Advertisement

The Swinging Pendulum of Evidence: Is There a Reality Behind Results from Randomised Trials and Real World Data? Lessons Learned from the Paclitaxel Debate

  • Christian-Alexander Behrendt
    Correspondence
    Corresponding author. Research Group GermanVasc, Department of Vascular Medicine, University Heart and Vascular Centre Hamburg, University Medical Centre Hamburg-Eppendorf, Hamburg, Germany
    Affiliations
    Department of Vascular Medicine, Research Group GermanVasc University Heart and Vascular Centre Hamburg, University Medical Centre Hamburg-Eppendorf, Hamburg, Germany
    Search for articles by this author
  • Frederik Peters
    Affiliations
    Department of Vascular Medicine, Research Group GermanVasc University Heart and Vascular Centre Hamburg, University Medical Centre Hamburg-Eppendorf, Hamburg, Germany
    Search for articles by this author
  • Kevin Mani
    Affiliations
    Department of Surgical Sciences, Uppsala University, Uppsala, Sweden
    Search for articles by this author
Open ArchivePublished:February 19, 2020DOI:https://doi.org/10.1016/j.ejvs.2020.01.029
      For many years, researchers have discussed the complementary value of evidence from randomised controlled trials (RCT) and so called real world data.
      • Mani K.
      • Bjorck M.
      Alternatives to randomised controlled trials for the poor, the impatient, and when evaluating emerging technologies.
      While RCTs still present the gold standard for determining causal relationships in clearly specified patient groups, real world data from clinical practice provide the observed outcomes of new interventions in the actual patient population. The IDEAL statement for innovation in surgery (Idea Development, Exploration, Assessment, Long term Follow up) highlights the role of routinely collected data for surveillance of rare and long term outcomes. Residual confounding, missing or imprecise information, and coding inaccuracy have been raised as major limitations of real world data by their opponents. Meanwhile, the limited generalisability, insufficient statistical power, attrition bias, and lack of independence from competing interests are often named as potential drawbacks of poorly planned RCTs. Additionally, well performed RCTs, adequately designed to assess a primary outcome, are often analysed for secondary outcomes (e.g., the paclitaxel example), and the value of such analyses can be questioned. Another point of criticism with impacts on both RCT and real world data addresses the reliability and validity of study data. Although there remains no clear consensus in this discussion, an increasing number of pragmatic trials have been started trying the advantages of both approaches.
      The paclitaxel controversy may serve as an example par excellence demonstrating that neither RCT nor real world data can claim to determine “reality” beyond any doubt.
      • Katsanos K.
      • Spiliopoulos S.
      • Kitrou P.
      • Krokidis M.
      • Karnabatidis D.
      Risk of death following application of paclitaxel-coated balloons and stents in the femoropopliteal artery of the leg: a systematic review and meta-analysis of randomized controlled trials.
      ,
      • Freisinger E.
      • Koeppe J.
      • Gerss J.
      • Goerlich D.
      • Malyar N.R.
      • Marschall U.
      • et al.
      Mortality after use of paclitaxel-based devices in peripheral arteries: a real-world safety analysis.
      In fact, the results from RCTs vs. observational studies in this field remain diametrically opposed. There is a signal towards higher mortality for the group exposed to paclitaxel vs. controls in summary level data from RCTs, and a contrasting signal towards lower mortality among the paclitaxel exposed in real world data. This opposition of results emphasises the need for further research to reflect the question of what is “real”. Starting another well powered RCT involving long term follow up is certainly advisable, but it will take a long time until the results will become available. Meanwhile, evidence from real world data is available but still not commonly accepted as high level evidence.
      Certainly, there is a new challenge entering the stage. With the increasing digitalisation of health care, the establishment of large scale registries, and possibilities for cross linkage of data sources, more groups will take advantage of analysing real world data. In highly complex data such as in peripheral arterial occlusive disease treatment, it is likely that competing study groups will achieve different results and reach different conclusions. A possible publication bias due to false positive signals needs to be considered. The methodological study design and applied statistical methods are of utmost importance to limit the imminent selection bias and residual confounding. Minimum standards are necessary to keep the high quality of real world evidence in the future. International collaborations such as the VASCUNET can help by reaching a broad consensus among real world data experts on minimum requirements.
      Two recently published papers on the effect of paclitaxel on survival using long term longitudinal data from a large German health insurance fund, BARMER, underline the major methodological challenges involved in this field.
      • Behrendt C.A.
      • Sedrakyan A.
      • Peters F.
      • Kreutzburg T.
      • Schermerhorn M.
      • Bertges D.J.
      • et al.
      Long term survival after femoropopliteal artery revascularizations with paclitaxel coated devices: a propensity score matched cohort analysis.
      ,
      • Columbo J.A.
      • Martinez-Camblor P.
      • MacKenzie T.A.
      • Staiger D.O.
      • Kang R.
      • Goodney P.P.
      • et al.
      Comparing long-term mortality after carotid endarterectomy vs carotid stenting using a novel instrumental variable method for risk adjustment in observational time-to-event data.
      The first study used a flexible Cox regression analysis on patients treated between 2007 and 2015 to conclude that drug eluting devices are safe for lower limb endovascular therapy.
      • Behrendt C.A.
      • Sedrakyan A.
      • Peters F.
      • Kreutzburg T.
      • Schermerhorn M.
      • Bertges D.J.
      • et al.
      Long term survival after femoropopliteal artery revascularizations with paclitaxel coated devices: a propensity score matched cohort analysis.
      Supporting this claim and adding more details, the second study applied a stratified propensity score matching on patients with symptomatic peripheral arterial occlusive disease having femoropopliteal artery treatment between 2010 and 2018 to prove that drug eluting devices were associated with lower mortality, amputation free survival, and major cardiovascular events, especially pronounced in patients with chronic limb threatening ischaemia.
      • Columbo J.A.
      • Martinez-Camblor P.
      • MacKenzie T.A.
      • Staiger D.O.
      • Kang R.
      • Goodney P.P.
      • et al.
      Comparing long-term mortality after carotid endarterectomy vs carotid stenting using a novel instrumental variable method for risk adjustment in observational time-to-event data.
      Both methods have advantages and limitations.
      As crude comparisons are well known to be of inferior value due to non-random assignment of treatment, traditionally, observational studies apply regression methods to adjust for most important confounders. Thereby, adjustment for confounding faces the challenge of selecting appropriate variables from the abundance of data with many thousands of candidate variables available in health insurance claims. This entails an assessment of the reliability and validity of each variable and the trade off between selecting too many variables vs. too few with a consequent risk of residual confounding. Various ways exist to adjust for observed confounding in statistical models, starting from merely picking the covariates regarding their clinical relevance, through to automatic selection procedures such as least absolute shrinkage and selection operator with cross validation. Importantly, the main analyses of real world data should be stratified by fundamentally different subgroups. As an example, in an assessment of the role of paclitaxel in lower limb revascularisation, patients treated for intermittent claudication should be analysed separately from those treated for chronic limb threatening ischaemia. In RCTs, this is less of an issue as samples are much more homogeneous owing to stricter inclusion and exclusion criteria.
      Another major element of study design concerns the tailoring of the data. Simply using all available patients over all calendar years at hand might result in an impressive sample size, long follow up, and high statistical power. Yet, this strategy might result in a highly heterogeneous sample with limited stability (susceptible to the influence of atypical/exotic patients) and limited validity. A sound study design involves trimming the data set to a homogeneous subset of patients by using clear and clinically meaningful inclusion and exclusion criteria. For example, including very early years, comprising only few highly selected patients being treated with first generation devices, might bias results. Including more up to date patients, however, lowers the median follow up duration.
      While randomisation and blinding are commonly accepted to be the gold standard in causative study design, in observational studies no such standard exists. In general, propensity score matching approaches provide a robust and feasible solution for a wide range of applications. Above all, propensity score based methods are more robust than pure multivariable regression methods merely at the price of a reduced sample size. Indeed, various ways exist to build matching pairs, and the stratification on the propensity score introduces further challenges so that a transparent reporting of the modelling steps is of importance. While analysis of data based on regression methods suggested a survival benefit for patients treated with paclitaxel, this benefit diminished when the same data were analysed using stratified propensity score matching. In another study concerning differences in long term mortality after carotid revascularisation, the authors demonstrated how estimates in observational studies step by step were increasingly similar to results from RCT after more proper adjustments had been applied.
      • Columbo J.A.
      • Martinez-Camblor P.
      • MacKenzie T.A.
      • Staiger D.O.
      • Kang R.
      • Goodney P.P.
      • et al.
      Comparing long-term mortality after carotid endarterectomy vs carotid stenting using a novel instrumental variable method for risk adjustment in observational time-to-event data.
      Nevertheless, even the best designs cannot sufficiently solve the problem of residual confounding in observational studies due to the absence of blinding and randomisation.
      In conclusion, it is important to further develop ways to ensure high quality in observational studies. Pre-registration of observational study designs may help to prevent p hacking and publication bias. Marginal differences in the selection of the target population, the matching and adjustment, and the definition of variables probably cause marked differences in results, influencing the conclusions. Thus, it is wise to implement and report multiple modelling strategies and test respective assumptions in comprehensive sensitivity analyses. Experienced statisticians should work hand in hand with health services researchers to keep the high quality of real world evidence. As routine data were collected for other purposes than research, extensive and thorough data preparation, data cleaning and internal validation are particularly important. Perhaps even more important than in other areas, analyses greatly benefit from international collaborations to identify and discuss peculiarities of country specific real world data from a broader context. It is an open yet urgent question for the vascular community to discuss, in which cases evidence from real world data may complement evidence from the sometimes inadequate RCTs.

      References

        • Mani K.
        • Bjorck M.
        Alternatives to randomised controlled trials for the poor, the impatient, and when evaluating emerging technologies.
        Eur J Vasc Endovasc Surg. 2019; 57: 598-599
        • Katsanos K.
        • Spiliopoulos S.
        • Kitrou P.
        • Krokidis M.
        • Karnabatidis D.
        Risk of death following application of paclitaxel-coated balloons and stents in the femoropopliteal artery of the leg: a systematic review and meta-analysis of randomized controlled trials.
        J Am Heart Assoc. 2018; 7e011245
        • Freisinger E.
        • Koeppe J.
        • Gerss J.
        • Goerlich D.
        • Malyar N.R.
        • Marschall U.
        • et al.
        Mortality after use of paclitaxel-based devices in peripheral arteries: a real-world safety analysis.
        Eur Heart J. 2019; https://doi.org/10.1093/eurheartj/ehz698
        • Behrendt C.A.
        • Sedrakyan A.
        • Peters F.
        • Kreutzburg T.
        • Schermerhorn M.
        • Bertges D.J.
        • et al.
        Long term survival after femoropopliteal artery revascularizations with paclitaxel coated devices: a propensity score matched cohort analysis.
        Eur J Vasc Endovasc Surg. 2020; 59: 587-596
        • Columbo J.A.
        • Martinez-Camblor P.
        • MacKenzie T.A.
        • Staiger D.O.
        • Kang R.
        • Goodney P.P.
        • et al.
        Comparing long-term mortality after carotid endarterectomy vs carotid stenting using a novel instrumental variable method for risk adjustment in observational time-to-event data.
        JAMA Netw Open. 2018; 1e181676