Advertisement

Editor's Choice – The GermanVasc Score: A Pragmatic Risk Score Predicts Five Year Amputation Free Survival in Patients with Peripheral Arterial Occlusive Disease

Open AccessPublished:December 14, 2020DOI:https://doi.org/10.1016/j.ejvs.2020.11.013

      Objective

      Patients with peripheral arterial occlusive disease (PAOD) face an increased risk of both lower limb amputation and death. To date, it has been challenging to predict the long term outcomes for PAOD. The aim was to develop a risk score to predict worse five year amputation free survival (AFS).

      Methods

      In this retrospective analysis of claims data, symptomatic PAOD patients were split into training and validation sets. Variables in the model were patient age and sex, Elixhauser comorbidities, and the 190 most common secondary diagnoses. Penalised Cox regression (least absolute shrinkage and selection operator [LASSO]) with tenfold cross validation for variable selection was performed and patients were categorised into five risk groups using the ten most important variables. All analyses were stratified by intermittent claudication (IC) and chronic limb threatening ischaemia (CLTI).

      Results

      In total, 87 293 patients with PAOD (female 45.3%, mean age 71.4 ± 11.1 years) were included in the analysis. The most important variable predicting worse five year AFS was patient age >80 years. The GermanVasc score exhibited good predictive accuracy both for IC (c statistic = 0.70, 95% confidence interval [CI] 0.69–0.71) and CLTI (c statistic = 0.69, 95% CI 0.68–0.70) with adequate calibration due largely to alignment of observed and expected risk. Depending on the cumulative point score, the five year risk of amputation or death ranged from 9% (low risk) to 48% (high risk) for IC, and from 25% to 88% for CLTI.

      Conclusion

      The GermanVasc score predicts worse five year AFS stratified for inpatients suffering from IC and CLTI, with good predictive accuracy. By separating low from high risk patients, the GermanVasc score may support patient centred consent.

      Keywords

      The first prediction score has been developed using machine learning techniques for long term outcomes in patients with a symptomatic peripheral arterial occlusive disease. The model exhibited high accuracy and adequate discrimination. The five year risk of amputation or death varied between 9% and 48% in patients with intermittent claudication, and between 25% and 88% in patients with chronic limb threatening ischaemia. In the routine clinical setting, the pragmatic score presented can help identify patients in need of intensified medical care and support decision making on invasive revascularisation opportunities.

      Introduction

      Patient centred health care should respect patients’ individual needs without incurring delays. High quality medical evidence develops through research over decades, primarily by performing randomised clinical trials. Yet, in peripheral vascular medicine, trial data are frequently not available and therefore many guidelines are limited to consensus recommendations with a low level of evidence.
      • Conte M.S.
      • Bradbury A.W.
      • Kolh P.
      • White J.V.
      • Dick F.
      • Fitridge R.
      • et al.
      Global vascular guidelines on the management of chronic limb-threatening ischemia.
      • Aboyans V.
      • Ricco J.B.
      • Bartelink M.E.L.
      • Björck M.
      • Brodmann M.
      • Cohnert T.
      • et al.
      2017 ESC guidelines on the diagnosis and treatment of peripheral arterial diseases, in collaboration with the European society for vascular surgery (ESVS).
      • Gerhard-Herman M.D.
      • Gornik H.L.
      • Barrett C.
      • Barshes N.R.
      • Corriere M.A.
      • Drachman D.E.
      • et al.
      2016 AHA/ACC guideline on the management of patients with lower extremity peripheral artery disease: a report of the American College of Cardiology/American heart association task force on clinical practice guidelines.
      For patients with peripheral arterial occlusive disease (PAOD), quality of life is seriously impaired by the risk of amputation and death.
      • Conte M.S.
      • Bradbury A.W.
      • Kolh P.
      • White J.V.
      • Dick F.
      • Fitridge R.
      • et al.
      Global vascular guidelines on the management of chronic limb-threatening ischemia.
      Knowledge about the individual probability of long term outcomes after hospitalisation for PAOD is sparse, therefore, it can be challenging to make the best choice from a wide range of possible invasive and best medical treatments.
      Routinely collected data from registries or health insurance claims can help to quantify the risks for specific long term outcomes. With the FINNVASC registry study, a linear sum score for post-operative mortality and/or major lower limb amputation was developed. However, this score was developed for short term outcomes only.
      • Biancari F.
      • Salenius J.P.
      • Heikkinen M.
      • Luther M.
      • Ylönen K.
      • Lepäntalo M.
      Risk-scoring method for prediction of 30-day postoperative outcome after infrainguinal surgical revascularization for critical lower-limb ischemia: a Finnvasc registry study.
      ,
      • Arvela E.
      • Söderström M.
      • Korhonen M.
      • Halmesmäki K.
      • Albäck A.
      • Lepäntalo M.
      • et al.
      Finnvasc score and modified Prevent III score predict long-term outcome after infrainguinal surgical and endovascular revascularization for critical limb ischemia.
      Recruiting multicentre registry trials such as SWEDEPAD (Swedish Drug-elution Trial in Peripheral Arterial Disease),
      • Nordanstig J.
      • Falkenberg M.
      The Swedish drug-elution trials in peripheral arterial disease (swedepad) – an update halfway through the overall inclusion phase.
      BASIL-3 (BAlloon versus Stenting in severe Ischaemia of the Leg-3),
      • Hunt B.D.
      • Popplewell M.A.
      • Davies H.
      • Meecham L.
      • Jarrett H.
      • Bate G.
      • et al.
      BAlloon versus Stenting in severe Ischaemia of the Leg-3 (BASIL-3): study protocol for a randomised controlled trial.
      BEST-CLI (Best Endovascular vs. Best Surgical Therapy in Patients with Critical Limb Ischemia),
      • Menard M.T.
      • Farber A.
      • Assmann S.F.
      • Choudhry N.K.
      • Conte M.S.
      • Creager M.A.
      • et al.
      Design and rationale of the best endovascular versus best surgical Therapy for patients with critical limb ischemia (BEST-CLI) trial.
      and the GermanVasc registry trial
      • Debus E.S.
      • Kriston L.
      • Schwaneberg T.
      • Hischke S.
      • Riess H.C.
      • Harter M.
      • et al.
      Rationale and methods of the IDOMENEO health outcomes of the peripheral arterial disease revascularisation study in the GermanVasc registry.
      are currently collecting primary research data, including a follow up beyond 30 days. While those clinical registries usually need a certain time to collect enough data for valid prediction models, health insurance claims provide a large sample size suitable to use for data driven methods and predictive modelling. Although machine learning approaches have been used frequently in stroke and cardiac risk prediction, its use remains extremely rare in patients with PAOD.
      • Chen-Ying H.
      • Wei-Chen C.
      • Po-Tsun L.
      • Ching-Heng L.
      • Chi-Chun L.
      Comparing deep neural network and other machine learning algorithms for stroke prediction in a large-scale population-based electronic medical claims database.
      ,
      • Weng S.F.
      • Reps J.
      • Kai J.
      • Garibaldi J.M.
      • Qureshi N.
      Can machine-learning improve cardiovascular risk prediction using routine clinical data?.
      This study aimed to develop an easy to use score to estimate the five year probability of amputation free survival (AFS) of patients with PAOD, based on routinely collected health insurance claims data in Germany.

      Materials and Methods

      Data source

      Health insurance claims data from Germany's second largest insurance fund, BARMER, cover approximately nine million German citizens (10.8% of Germany's population). The German Modification of the World Health Organization's International Classification of Diseases 10th Revision (ICD-10-GM), Operations and Procedures Codes, and the German version of the international Anatomical Therapeutic Chemical classification for pharmacological treatment codes were used.

      Study population

      Adult patients aged 40 years and above treated in legally endorsed German hospitals and presenting with a primary diagnosis of symptomatic PAOD (stages II to IV, according to the Fontaine classification) from 1 January 2008 to 31 December 2016 were included. To identify an incident diagnosis of PAOD, a three year lookback was used (i.e., a disease free interval of three years). Patients with prior major amputation or death within 30 days after discharge were excluded (see Fig. 1).
      Figure 1
      Figure 1Flowchart for training and validation dataset of hospitalised patients with peripheral arterial occlusive disease (PAOD) stratified into intermittent claudication (IC) and chronic limb threatening ischaemia (CLTI) groups. CV = cardiovascular event, including myocardial infarction or stroke.

      Variables

      Variables included age, sex (male vs. female), smoking, 30 different Elixhauser comorbidity groups (three year lookback),
      • Elixhauser A.
      • Steiner C.
      • Harris D.R.
      • Coffey R.M.
      Comorbidity measures for use with administrative data.
      ,
      • Quan H.
      • Sundararajan V.
      • Halfon P.
      • Fong A.
      • Burnand B.
      • Luthi J.C.
      • et al.
      Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.
      year of discharge, history of prior myocardial infarction (MI) or prior stroke, atrial fibrillation (AF), dialysis, gangrene, discharge to rehabilitation or nursing home or hospice (three year lookback), and the 190 most common inpatient secondary diagnoses on admission. The patients were grouped according to age, to account for non-linear effects: 40–60 years; 61–70 years; 71–80 years; and ≥80 years. Missing data (0.5%) were deleted in a listwise manner (complete case analysis). All codes for the included variables are listed in Appendix S1 (see Supplementary Material).

      Outcome

      The primary outcome was death or major amputation above ankle level, measured as the composite endpoint of five year AFS after the index hospitalisation. Secondary outcomes were cardiovascular events, including MI or stroke after discharge.

      Statistical analysis

      To account for right censoring, a time to event framework using Cox proportional hazard models was used. The survival for five different risk groups from low to high risk was illustrated with Kaplan–Meier curves. The steps are outlined below. The implemented R code is given in the Appendix Table A5 (Supplementary Material), to encourage other research groups to use this risk score for better comparability.

      Step 1: stratification

      All analyses were stratified by Fontaine stages (according to the corresponding ICD-10 code) during the index stay: stage II for intermittent claudication (IC) vs. stage III/IV for chronic limb threatening ischaemia (CLTI).

      Step 2: training and validation data set

      The original dataset was separated into a training set (60%) and a validation set (40%). The predictive models and point score were developed in the training set, and model performance was assessed in the validation set.

      Step 3: variable selection

      Using the least absolute shrinkage and selection operator (LASSO) method, a parsimonious Cox survival model was estimated using a penalty term λ shrinking coefficients of irrelevant variables to zero (lambda).
      • Tibshirani R.
      Regression shrinkage and selection via the Lasso.
      The optimal λ was selected by tenfold cross validation. Using the non-zero variables only, the Cox survival model was re-estimated without a penalty term.

      Step 4: top 10

      To identify predictors contributing most to the generalisation power of the model from the previous step, variables were ranked based on the Breiman permutation method within the validation set.
      • Breiman L.
      Random forests.
      The ten variables with c statistics (general term for area under the curve for a Cox regression; range 0–1) and highest Breiman importance were selected for use in the GermanVasc score.

      Step 5: GermanVasc score

      A final Cox model was fitted to the training set using only the 10 selected variables from step 4. The beta coefficients of this model were transformed to points (integer values), which, in sum, represent the GermanVasc score for individual risk prediction. The beta values were multiplied by ten and rounded to integers, following Austin et al.
      • Austin P.C.
      • Lee D.S.
      • D'Agostino R.B.
      • Fine J.P.
      Developing points-based risk-scoring systems in the presence of competing risks.
      This resulted in a pragmatic sum point score of the ten most important variables for usage in clinical routine care.

      Step 6: model performance (discrimination)

      The discrimination of the variables was assessed in the validation set using the concordance statistics of c, a rank correlation coefficient accounting for censoring in the data. Further out of sample statistics were calculated for a subgroup of the validation set excluding high volume hospitals (>1000 interventions during the study period), a subgroup of the validation set excluding patients with an index stay after 2012, and cardiovascular event free survival as an outcome.

      Step 7: model performance (calibration)

      Calibration of the prediction model was assessed by comparing the observed risk with the expected risk for each GermanVasc point value. The observed risk was measured by the Kaplan–Meier survival function fitted to the validation set and the expected risk by the Kaplan–Meier survival function fitted to the training set.

      Step 8: risk groups

      To categorise patients in five different risk groups, quantiles of the GermanVasc score points (x0.2, x0.4, x0.6, and x0.8) were used to find equal sized risk groups with low risk (≤x0.2) to high risk (>x0.8). Risks for each group were estimated and plotted using Kaplan–Meier functions and hazard ratios (HRs) for each group using Cox regression models.

      Step 9: summary sheet

      To facilitate the application of the GermanVasc score, a summary sheet was created. Therein, points for each variable are displayed along with the risk associated with summarised points for each of the five risk groups. A tutorial and examples are provided in Fig. 2.
      Figure 2
      Figure 2Summary sheet for the GermanVasc risk score stratified for intermittent claudication (IC) and chronic limb threatening ischaemia (CLTI), including the five risk groups: low; low–moderate; moderate (medium); moderate–high; and high risk. The numbers before the risk factors are the point scores to be summarised for the presence of this risk factor. COPD = chronic obstructive pulmonary disease.

      Sensitivity analyses

      An elastic net approach was used as a sensitivity analysis (option alpha). The elastic net approach is known to perform better in feature selection if variables are highly correlated and more variables are used.
      • Zou H.
      • Hastie T.
      Regularization and variable selection via the elastic net.
      The results of these analyses are given in the Appendix (Supplementary Material).

      Software

      Data processing was performed with SAS version 9.04 (SAS Institute, Cary, NC, USA). Descriptive analyses, Cox models, LASSO, elastic net, model diagnostics, and illustrations were performed with R version 3.3 (survival and glmnet packages; R Foundation for Statistical Computing, Vienna, Austria). Visualisation was performed with Adobe Illustrator version 24.1.2 (Adobe, San Jose, CA, USA).

      Reporting guidelines

      Results were reported using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.
      • Moons K.G.
      • Altman D.G.
      • Reitsma J.B.
      • Ioannidis J.P.
      • Macaskill P.
      • Steyerberg E.W.
      • et al.
      Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration.

      Results

      Baseline characteristics

      In total, 87 293 patients with an incident symptomatic PAOD diagnosis between 2008 and 2016 (∼10 000 per year; see Fig. 1) were included. Of these, 46 703 were diagnosed with IC (43% female) and 40 590 with CLTI (48% female). Patients with IC differed substantially from those with CLTI (Table 1). Patients with IC were younger (mean age 69 vs. 75 years); had fewer comorbidities such as congestive heart failure (14% vs. 33%), cardiac dysrhythmias (16% vs. 34%), or renal failure (19% vs. 37%); and were prescribed fewer medications (median 8 vs. 11). Within five years of index discharge, 19% of the patients with IC died and 1% had a major amputation, so that the composite endpoint was about 20%. For patients with CLTI, 50% died and 9% had a major amputation; the composite endpoint was 52%. The characteristics of the corresponding training and validation sets were largely similar (Table 1).
      Table 1Patient characteristics of the training and validation dataset for patients with intermittent claudication (IC) and chronic limb-threatening ischaemia (CLTI)
      Total (n = 87 293)IC training (n = 28 021)IC validation (n = 18 682)CLTI training (n = 24 354)CLTI validation (n = 16 236)
      Age – y71.4 ± 11.168.8 ± 10.169.0 ± 10.274.5 ± 11.374.4 ± 11.3
      Female sex39 545 (45.3)12 046 (43.0)8 050 (43.1)11 684 (48.0)7 765 (47.8)
      Elixhauser groups (3 y)
       Congestive heart failure20 399 (23.4)4 081 (14.6)2 698 (14.4)8 200 (33.7)5 420 (33.4)
       Cardiac dysrhythmias21 351 (24.5)4 530 (16.2)3 081 (16.5)8 205 (33.7)5 535 (34.1)
       Hypertension68 384 (78.3)21 437 (76.5)14 316 (76.6)19 587 (80.4)13 044 (80.3)
       Diabetes, complicated24 004 (27.5)4 116 (14.7)2 792 (14.9)10 268 (42.2)6 828 (42.1)
       Renal failure23 728 (27.2)5 045 (18.0)3 533 (18.9)9 074 (37.3)6 076 (37.4)
       COPD12 634 (14.5)3 655 (13.0)2 507 (13.4)3 915 (16.1)2 557 (15.7)
       Obesity11 656 (13.4)3 237 (11.6)2 073 (11.1)3 813 (15.7)2 533 (15.6)
       Smoking15 330 (17.6)5 919 (21.1)3 880 (20.8)3 385 (13.9)2 146 (13.2)
       Prior myocardial infarction7 465 (8.6)2 218 (7.9)1 499 (8.0)2 274 (9.3)1 474 (9.1)
       Prior stroke7 307 (8.4)1 545 (5.5)1 031 (5.5)2 846 (11.7)1 885 (11.6)
       Atrial fibrillation12 801 (14.7)2 379 (8.5)1 580 (8.5)5 303 (21.8)3 539 (21.8)
       Nursing care, discharge reason2 076 (2.4)119 (0.4)86 (0.5)1 141 (4.7)730 (4.5)
       Rehabilitation, discharge reason5 367 (6.1)1 147 (4.1)761 (4.1)2 109 (8.7)1 350 (8.3)
       Dialysis1 812 (2.1)180 (0.6)122 (0.7)880 (3.6)630 (3.9)
      Diagnosis (index stay)
       Dyslipidaemia, E7828 580 (32.7)10 812 (38.6)7 111 (38.1)6 399 (26.3)4 258 (26.2)
       Dementia, F031 711 (2.0)107 (0.4)73 (0.4)918 (3.8)613 (3.8)
      Polypharmacy9 (5–13)8 (5–11)8 (5–12)11 (7–16)11 (7–16)
      Follow up time – d1 503 (839–1825)1 825 (1134–1825)1 804 (1127–1825)1 131 (481–1825)1 144 (475–1825)
      Antithrombotics40 437 (46.3)11 760 (42.0)8 018 (42.9)12 353 (50.7)8 306 (51.2)
      Lipid lowering drugs39 431 (45.2)14 271 (50.9)9 546 (51.1)9 280 (38.1)6 334 (39.0)
      Antihypertensive72 602 (83.2)22 557 (80.5)15 041 (80.5)20 971 (86.1)14 033 (86.4)
      Event rate within five years
       Composite endpoint: all cause death or major amputation30 635 (35.1)5 514 (19.7)3 728 (20.0)12 852 (52.8)8 541 (52.6)
       All cause death29 129 (33.4)5 323 (19.0)3 592 (19.2)12 118 (49.8)8 096 (49.9)
       Major amputation4 256 (4.9)401 (1.4)255 (1.4)2 169 (8.9)1 431 (8.8)
       Myocardial infarction10 180 (11.7)3 073 (11.0)2 142 (11.5)3007 (12.3)1958 (12.1)
       Stroke11 874 (13.6)3 541 (12.6)2 295 (12.3)3 596 (14.8)2 442 (15.0)
      Data are presented as n (%), mean ± standard deviation, or median (interquartile range). COPD = chronic obstructive pulmonary disorder.

      GermanVasc score

      For the subgroup of patients with IC, the GermanVasc score exhibited good discrimination (model top 10 variables, c = 0.70, 95% confidence interval [CI] 0.69–0.71). The summarised point score ranged from 0 to 49. Patients with IC with lower point scores represented younger and healthier patients, and higher point scores older or sicker patients (Fig. A1a; see Supplementary Material). For the high risk group, the hazard ratio (HR) was 7.48 (95% CI 6.63–8.45), compared with the low risk group. The Kaplan–Meier survival curve (event rate for five year amputation or death in the low risk group was 8.7% and for the high risk group 47.6%) is given in Fig. 3(A). The 10 most important predictors of worse AFS were older age, male sex, cancer, absence of dyslipidaemia, alcohol abuse, chronic pulmonary disease, prior hospital stay, diabetes, dialysis, and fluid and electrolyte disorders (Table 2, Table 3). The point score also achieved good discrimination for the subgroups excluding high volume hospitals (c = 0.69, 95% CI 0.67–0.71 [Table A3; see Supplementary Material]) and excluding patients with an index stay after 2012 (c = 0.71, 95% CI 0.70–0.72) and for cardiovascular event free survival as the outcome (c = 0.63, 95% CI 0.62–0.64).
      Figure 3
      Figure 3Cumulative Kaplan–Meier estimate of five year amputation free survival for GermanVasc score with the top 10 variables stratified for (A) intermittent claudication (IC) and (B) chronic limb threatening ischaemia (CLTI) for five risk groups: low; low–moderate; moderate; moderate–high; and high risk.
      Table 2Cox regression model prediction for five year amputation free survival for intermittent claudication in 46 703 patients: least absolute shrinkage and selection operator method with hazard ratios (HR) and 95% confidence intervals (CIs); the top 10 variables; and points for risk score based on highest Breiman importance
      VariableHR (95% CI)Importance (Breiman)
      Breiman importance multiplied by 10 000.
      GermanVasc score
      Age – y
       >80 (ref. 40–60)4.80 (4.35–5.30)1 16816
       71–80 (ref. 40–60)2.21 (2.02–2.43)5528
       61–70 (ref. 40–60)1.50 (1.36–1.66)1444
      Male sex1.33 (1.26–1.41)733
      Cancer1.62 (1.48–1.77)695
      Absence of dyslipidaemia (E78)1.21 (1.14–1.27)442
      Alcohol abuse1.75 (1.56–1.97)336
      Chronic obstructive pulmonary disease1.56 (1.46–1.67)314
      Prior hospital stay1.25 (1.18–1.32)272
      Diabetes1.46 (1.38–1.55)264
      Dialysis3.31 (2.74–3.98)1712
      Fluid and electrolyte disorders1.61 (1.51–1.71)165
      Breiman importance multiplied by 10 000.
      Table 3Cox regression model prediction for five year amputation free survival for intermittent claudication in 46 703 patients: hazard ratios (HR) and 95% confidence intervals (CIs) and five year risk of amputation or death in five groups ranked from “low risk” to “high risk”
      5 y risk of amputation or death
      c = 0.70 (95% CI 0.69–0.71).
      HR (95% CI)Amputation or death after 5 y – %Range 0–49
      Low riskReference8.70–7
      Low–moderate1.54 (1.32–1.79)13.08–10
      Moderate2.37 (2.07–2.70)19.111–14
      Moderate–high3.82 (3.37–4.34)28.915–19
      High risk7.48 (6.63–8.45)47.620–49
      c = 0.70 (95% CI 0.69–0.71).
      Likewise, for the subgroup of patients with CLTI, the GermanVasc score exhibited good discrimination (model top 10 variables, c = 0.69, 95% CI 0.68–0.70). The point score ranged from 0 to 41 (Table 4, Table 5), the distribution of the symmetric point score distribution is illustrated in Fig. A1b (Supplementary Material). For the high risk group, the HR was 7.83 (95% CI 7.24–8.48). The Kaplan–Meier survival curve (low risk event rate 25.2%; high risk five year rate of amputation or death 88.2%) is given in Fig. 3(B). The 10 most important predictors of worse AFS were older age, gangrene, unspecified dementia, dialysis, congestive heart failure, vascular dementia, cancer, fluid and electrolyte disorders, renal failure, and cardiac dysrhythmias (Table 4, Table 5). The point score also achieved good discrimination for the subgroups excluding high volume hospitals (c = 0.71, 95% CI 0.69–0.73 [Table A3; Supplementary Material]) and excluding patients with an index stay after 2012 (c = 0.70, 95% CI 0.69–0.71) and for cardiovascular event free survival as the outcome (c = 0.61, 95% CI 0.60–0.62).
      Table 4Cox regression model prediction for five year amputation free survival for chronic limb threatening ischaemia in 40 590 patients: least absolute shrinkage and selection operator method with hazard ratios (HR) and 95% confidence intervals (CIs), the top 10 variables, and points for the risk score based on highest Breiman importance
      Top 10 variablesHR (95% CI)Importance (Breiman)
      Breiman importance multiplied by 10 000.
      GermanVasc score
      Age – y
       >80 (ref. 40–60)2.94 (2.72–3.17)1 33911
       71–80 (ref. 40–60)1.84 (1.70–1.99)3416
       61–70 (ref. 40–60)1.39 (1.28–1.51)763
      Gangrene1.55 (1.48–1.62)1484
      Unspecified dementia (F03)1.87 (1.74–2.01)466
      Dialysis1.74 (1.61–1.88)416
      Congestive heart failure1.37 (1.32–1.43)383
      Vascular dementia (F01)2.02 (1.84–2.21)317
      Cancer1.47 (1.39–1.57)314
      Fluid and electrolyte disorders1.34 (1.29–1.39)293
      Renal failure1.26 (1.21–1.31)202
      Cardiac dysrhythmias1.23 (1.18–1.28)172
      Breiman importance multiplied by 10 000.
      Table 5Cox regression model prediction for five year amputation free survival for chronic limb threatening ischaemia in 40 590 patients: hazard ratios (HR) and 95% confidence intervals (CIs) and five year risk of amputation or death in five groups ranked from “low risk” to “high risk”
      5 y risk of amputation or death∗HR (95% CI)Amputation or death after 5 y – %Range 0–41
      Low riskReference25.20–7
      Low–moderate2.06 (1.89–2.24)45.58–12
      Moderate3.06 (2.81–3.33)59.113–15
      Moderate–high4.61 (4.26–4.99)73.716–20
      High risk7.83 (7.24–8.48)88.221–41
      ∗c = 0.69 (95% CI 0.68–0.70).

      Calibration

      In the validation set, observed and expected risk showed a high degree of agreement for both IC and CLTI patients, except for high point score values, where only a few cases occurred. This is not only true for the full data, but also subgroups excluding high volume hospitals and excluding patients with index stay after 2012 (Fig. A2 – A4; Supplementary Material).

      Sensitivity analysis

      Elastic net

      All analyses for five year AFS were performed with elastic net, which provided similar results for the top ten variables but with a slightly different ranking by Breiman's permutation method and the same range for the risk groups (from low risk to high risk). For the high risk IC group the HR was 7.18 (95% CI 6.35–8.11; c = 0.70 [95% CI 0.69–0.71]), and for patients in the high risk CLTI group the HR was 7.83 (95% CI 7.24–8.48; c = 0.69 [95% CI 0.68–0.70]). The results are given in full in Tables A1 and A2 (Supplementary Material).

      Summary sheet

      All relevant information is provided in the summary sheet (Fig. A5; see Supplementary Material).

      Discussion

      In this large scale health insurance claims data analysis, two pragmatic point scores were developed exhibiting high predictive accuracy and adequate discrimination between risk groups to predict worse five year AFS in patients with IC and CLTI. The scores were based on a data driven machine learning approach on longitudinal data. Owing to the advantages of the study sample, as compared with other data sources, there was only a small amount of missing data and no losses to follow up.
      • Ohlmeier C.
      • Langner I.
      • Hillebrand K.
      • Schmedt N.
      • Mikolajczyk R.
      • Riedel O.
      • et al.
      Mortality in the German Pharmacoepidemiological Research Database (GePaRD) compared to national data in Germany: results from a validation study.
      Previous risk scores have been developed to predict outcomes in comparable target populations such as FINNVASC (3 925 patients with CLTI, 1991–1999, 30 day follow up),
      • Biancari F.
      • Salenius J.P.
      • Heikkinen M.
      • Luther M.
      • Ylönen K.
      • Lepäntalo M.
      Risk-scoring method for prediction of 30-day postoperative outcome after infrainguinal surgical revascularization for critical lower-limb ischemia: a Finnvasc registry study.
      ,
      • Arvela E.
      • Söderström M.
      • Korhonen M.
      • Halmesmäki K.
      • Albäck A.
      • Lepäntalo M.
      • et al.
      Finnvasc score and modified Prevent III score predict long-term outcome after infrainguinal surgical and endovascular revascularization for critical limb ischemia.
      the PREVENT-III risk score (1 166 patients with CLTI, 2003–2007, one year follow up),
      • Schanzer A.
      • Goodney P.P.
      • Li Y.
      • Eslami M.
      • Cronenwett J.
      • Messina L.
      • et al.
      Validation of the PIII CLI risk score for the prediction of amputation-free survival in patients undergoing infrainguinal autogenous vein bypass for critical limb ischemia.
      the ERICVA simplified score (672 patients with CLTI, 2005–2010, one year follow up),
      • Brizuela Sanz J.A.
      • González Fajardo J.A.
      • Taylor J.H.
      • Río Solá L.
      • Muñoz Moreno M.F.
      • Vaquero Puerta C.
      Design of a new risk score in critical limb ischaemia: the ERICVA model.
      the COPART (COhorte de Patients ARTériopathes) risk score (184 patients with IC, 2002–2004, five year follow up),
      • Hackl G.
      • Jud P.
      • Avian A.
      • Gary T.
      • Deutschmann H.
      • Seinost G.
      • et al.
      COPART risk score, endothelial dysfunction, and arterial hypertension are independent risk factors for mortality in claudicants.
      Arruda-Olsen et al. (1 676 patients with PAOD, 1998–2011, five year follow up),
      • Arruda-Olson A.M.
      • Afzal N.
      • Priya Mallipeddi V.
      • Said A.
      • Moussa Pacha H.
      • Moon S.
      • et al.
      Leveraging the electronic health record to create an automated real-time prognostic tool for peripheral arterial disease.
      or the SMART (Second Manifestations of Arterial Disease) risk score (5 788 patients with PAOD, 1996–2010, five year follow up).
      • Dorresteijn J.A.
      • Visseren F.L.
      • Wassink A.M.
      • Gondrie M.J.
      • Steyerberg E.W.
      • Ridker P.M.
      • et al.
      Development and validation of a prediction rule for recurrent vascular events based on a cohort study of patients with arterial disease: the SMART risk score.
      and van Walraven et al.
      • van Walraven C.
      • Austin P.C.
      • Jennings A.
      • Quan H.
      • Forster A.J.
      A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data.
      who developed a comorbidity score using a large administrative database from Canada to predict in hospital mortality (345 795 patients, 1996–2008, in hospital follow up). Ambler et al.
      • Ambler G.K.
      • Brooks D.E.
      • Al Zuhir N.
      • Ali A.
      • Gohel M.S.
      • Hayes P.D.
      • et al.
      Effect of frailty on short- and mid-term outcomes in vascular surgical patients.
      used logistic regression methods to highlight the impact of frailty on outcomes (413 vascular surgery patients, one year follow up). On closer consideration, these previous risk scores were partially limited by several methodological issues. They either represented only a small subset of the entire target population (e.g., patients with CLTI or IC) with 184 to 5 788 patients or predicted short and mid term outcomes only, although most events occur in the longer term. The largest prediction model included a rather non-specific inpatient population, while the risk scores were most likely context specific. Furthermore, the variable selection was frequently subject to an investigator bias and therefore not objectifiable for external evaluation. For instance, the FINNVASC and PREVENT-III scores both relied on only four variables. The current study confirmed previous findings while adding an objectifiable variable selection using data driven approaches on a large disease specific sample.
      A suitable risk score predicting long term outcomes may support the choice of optimal treatment and informed consent.
      • Lee D.S.
      • Austin P.C.
      • Rouleau J.L.
      • Liu P.P.
      • Naimark D.
      • Tu J.V.
      Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model.
      As an example, the American College of Cardiology and American Heart Association guidelines recommend initiation of statin treatment if the risk of stroke or MI exceeds 7.5%.
      • Goff Jr., D.C.
      • Lloyd-Jones D.M.
      • Bennett G.
      • Coady S.
      • D'Agostino R.B.
      • Gibbons R.
      • et al.
      2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American heart association task force on practice guidelines.
      Those risk adjusted recommendations certainly demand an evidence based risk stratification. Another central advantage of the GermanVasc risk score may be a likely better comparability of studies using administrative data. With the increasing importance of real world data in health services research, harmonisation of methods appears crucial. By using the GermanVasc score, a single metric variable can be used to adjust multivariable models for major composite endpoints.
      Interestingly, age was by far the most relevant predictor of long term outcomes for both IC and CLTI patients in the current study, while some prior studies deemed age to be less relevant for outcomes in PAOD.
      • Biancari F.
      • Salenius J.P.
      • Heikkinen M.
      • Luther M.
      • Ylönen K.
      • Lepäntalo M.
      Risk-scoring method for prediction of 30-day postoperative outcome after infrainguinal surgical revascularization for critical lower-limb ischemia: a Finnvasc registry study.
      Rather, the current study results are in line with the extensive literature on biomarkers predicting mortality, where even complex models rarely perform better than age alone.
      • Levine M.E.
      Modeling the rate of senescence: can estimated biological age predict mortality more accurately than chronological age?.
      Besides age, only three comorbidities (dialysis, fluid and electrolyte disorders, and cancer) were equally included both in the IC and CLTI models. The notable differences between the models for IC and CLTI underscore the need for stratified risk estimation in fundamentally different groups. In line with this, guidelines separate PAOD patients with IC and CLTI,
      • Conte M.S.
      • Bradbury A.W.
      • Kolh P.
      • White J.V.
      • Dick F.
      • Fitridge R.
      • et al.
      Global vascular guidelines on the management of chronic limb-threatening ischemia.
      ,
      • Aboyans V.
      • Ricco J.B.
      • Bartelink M.E.L.
      • Björck M.
      • Brodmann M.
      • Cohnert T.
      • et al.
      2017 ESC guidelines on the diagnosis and treatment of peripheral arterial diseases, in collaboration with the European society for vascular surgery (ESVS).
      and most risk scores had been developed for patients with CLTI only, for example, the PREVENT-III score.
      • Schanzer A.
      • Goodney P.P.
      • Li Y.
      • Eslami M.
      • Cronenwett J.
      • Messina L.
      • et al.
      Validation of the PIII CLI risk score for the prediction of amputation-free survival in patients undergoing infrainguinal autogenous vein bypass for critical limb ischemia.
      Recent findings on interventions, treatment patterns, and outcomes also confirmed a stratified approach by disease severity.
      • Secemsky E.A.
      • Kundi H.
      • Weinberg I.
      • Jaff M.R.
      • Krawisz A.
      • Parikh S.A.
      • et al.
      Association of survival with femoropopliteal artery revascularization with drug-coated devices.
      ,
      • Peters F.
      • Kreutzburg T.
      • Rieß H.C.
      • Heidemann F.
      • Marschall U.
      • L'Hoest H.
      • et al.
      Optimal pharmacological treatment of symptomatic peripheral arterial occlusive disease and evidence of female patient disadvantage: an analysis of health insurance claims data.
      This is the era of large datasets and rapid development of “big data” methods in medicine, especially in clinical predictions. However, most of the cardiovascular disease prediction models lack external validation.
      • Damen J.A.A.G.
      • Hooft L.
      • Schuit E.
      • Debray T.P.A.
      • Collins G.S.
      • Tzoulaki I.
      • et al.
      Prediction models for cardiovascular disease risk in the general population: systematic review.
      Differences between the included target populations, the contextual validity, and marked varieties between healthcare systems must be considered. A risk score validated for a heterogeneous cohort in a certain country needs to be validated before applying to another cohort in a different country.
      • Kriston L.
      Machine learning's feet of clay.
      To date, the available risk scores have not been validated for the German population, and there are reports that even among European populations outcomes in vascular medicine are significantly different.
      • Beck A.W.
      • Sedrakyan A.
      • Mao J.
      • Venermo M.
      • Faizer R.
      • Debus S.
      • et al.
      Variations in abdominal aortic aneurysm care: a report from the International Consortium of Vascular Registries.
      Besides validation, the clinical relevance of the research question should be more focused than big datasets and be easy to achieve statistical significance.
      • Steyerberg E.W.
      • Harrell Jr., F.E.
      Prediction models need appropriate internal, internal-external, and external validation.

      Limitations

      Firstly, for most of the study variables, only inpatient data were used. There was a possibility that a small population were undergoing invasive revascularisation at outpatient facilities. Unlike the USA system, the German reimbursement system thus far motivates physicians to perform revascularisations as inpatients. The coding of health care in outpatient facilities differs from that in hospitals. Hence, very few publications are available to shed light on this aspect. Secondly, there is an ongoing discussion about the advantages and limitations of health insurance claims data. Some variables require a composite of codes to approximate to the relevant risk factor (e.g., smoking). No information about certain clinical parameters, for example, ankle brachial index, body mass index, cholesterol, or further laboratory tests, is available in routinely collected administrative data. Encouragingly, owing to regular external cross validation and harsh penalties for the submission of false claims, major comorbidities and endpoints are known to be of high internal validity. This is especially true for observations that must be reported to different institutions and authorities such as major surgery, blood transfusion, or mortality. Lastly, as with many other models, the current score lacks independent external validation. Differences between the included target populations, the contextual validity, and marked variations between healthcare systems must be considered. Prospectively collected registry data, clinical parameters, and patient reported outcomes can be used to complement and extend the risk score. External validation of the GermanVasc score will be performed according to appropriate model approaches with a hierarchical structure (clustered data) and subgroup comparisons presented elsewhere.

      Future directions

      The GermanVasc score will be continuously updated with new claims data and registry data collected in the future. The most recent version and an online calculator are available at https://riskscore.germanvasc.de.
      The GermanVasc score developed in the current analysis may help patients and their physicians to predict individual five year AFS to support patient centred consent and treatment decision making. Patients at high risk may benefit significantly from being directed to an intensified treatment in line with demand.

      Conclusion

      The GermanVasc score predicts worse five year AFS stratified for inpatients suffering from IC and CLTI with good predictive accuracy. By separating low from high risk patients, the GermanVasc score may support patient centred consent.

      Conflicts of interest

      None.

      Funding

      The RABATT project is funded by the innovations fund of the German Federal Joint Committee from 2019 to 2022 (grant number 01VSF18035 ; PI: CAB).

      Appendix A. Supplementary data

      The following is the Supplementary data to this article:

      References

        • Conte M.S.
        • Bradbury A.W.
        • Kolh P.
        • White J.V.
        • Dick F.
        • Fitridge R.
        • et al.
        Global vascular guidelines on the management of chronic limb-threatening ischemia.
        Eur J Vasc Endovasc Surg. 2019; 58: S1-S109
        • Aboyans V.
        • Ricco J.B.
        • Bartelink M.E.L.
        • Björck M.
        • Brodmann M.
        • Cohnert T.
        • et al.
        2017 ESC guidelines on the diagnosis and treatment of peripheral arterial diseases, in collaboration with the European society for vascular surgery (ESVS).
        Eur J Vasc Endovasc Surg. 2018; 55: 305-368
        • Gerhard-Herman M.D.
        • Gornik H.L.
        • Barrett C.
        • Barshes N.R.
        • Corriere M.A.
        • Drachman D.E.
        • et al.
        2016 AHA/ACC guideline on the management of patients with lower extremity peripheral artery disease: a report of the American College of Cardiology/American heart association task force on clinical practice guidelines.
        J Am Coll Cardiol. 2017; 69: e71-126
        • Biancari F.
        • Salenius J.P.
        • Heikkinen M.
        • Luther M.
        • Ylönen K.
        • Lepäntalo M.
        Risk-scoring method for prediction of 30-day postoperative outcome after infrainguinal surgical revascularization for critical lower-limb ischemia: a Finnvasc registry study.
        World J Surg. 2007; 31: 217-225
        • Arvela E.
        • Söderström M.
        • Korhonen M.
        • Halmesmäki K.
        • Albäck A.
        • Lepäntalo M.
        • et al.
        Finnvasc score and modified Prevent III score predict long-term outcome after infrainguinal surgical and endovascular revascularization for critical limb ischemia.
        J Vasc Surg. 2010; 52: 1218-1225
        • Nordanstig J.
        • Falkenberg M.
        The Swedish drug-elution trials in peripheral arterial disease (swedepad) – an update halfway through the overall inclusion phase.
        Eur J Vasc Endovasc Surg. 2019; 58: e367-e368
        • Hunt B.D.
        • Popplewell M.A.
        • Davies H.
        • Meecham L.
        • Jarrett H.
        • Bate G.
        • et al.
        BAlloon versus Stenting in severe Ischaemia of the Leg-3 (BASIL-3): study protocol for a randomised controlled trial.
        Trials. 2017; 18: 224
        • Menard M.T.
        • Farber A.
        • Assmann S.F.
        • Choudhry N.K.
        • Conte M.S.
        • Creager M.A.
        • et al.
        Design and rationale of the best endovascular versus best surgical Therapy for patients with critical limb ischemia (BEST-CLI) trial.
        J Am Heart Assoc. 2016; 5e003219
        • Debus E.S.
        • Kriston L.
        • Schwaneberg T.
        • Hischke S.
        • Riess H.C.
        • Harter M.
        • et al.
        Rationale and methods of the IDOMENEO health outcomes of the peripheral arterial disease revascularisation study in the GermanVasc registry.
        Vasa. 2018; 47: 499-505
        • Chen-Ying H.
        • Wei-Chen C.
        • Po-Tsun L.
        • Ching-Heng L.
        • Chi-Chun L.
        Comparing deep neural network and other machine learning algorithms for stroke prediction in a large-scale population-based electronic medical claims database.
        Annu Int Conf IEEE Eng Med Biol Soc. 2017; 2017: 3110-3113
        • Weng S.F.
        • Reps J.
        • Kai J.
        • Garibaldi J.M.
        • Qureshi N.
        Can machine-learning improve cardiovascular risk prediction using routine clinical data?.
        PLoS One. 2017; 12e0174944
        • Elixhauser A.
        • Steiner C.
        • Harris D.R.
        • Coffey R.M.
        Comorbidity measures for use with administrative data.
        Med Care. 1998; 36: 8-27
        • Quan H.
        • Sundararajan V.
        • Halfon P.
        • Fong A.
        • Burnand B.
        • Luthi J.C.
        • et al.
        Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.
        Med Care. 2005; 43: 1130-1139
        • Tibshirani R.
        Regression shrinkage and selection via the Lasso.
        J R Stat Soc Ser B (Methodological). 1996; 58: 267-288
        • Breiman L.
        Random forests.
        Machine Learn. 2001; 45: 5-32
        • Austin P.C.
        • Lee D.S.
        • D'Agostino R.B.
        • Fine J.P.
        Developing points-based risk-scoring systems in the presence of competing risks.
        Stat Med. 2016; 35: 4056-4072
        • Zou H.
        • Hastie T.
        Regularization and variable selection via the elastic net.
        J R Stat Soc. 2005; 67: 301-320
        • Moons K.G.
        • Altman D.G.
        • Reitsma J.B.
        • Ioannidis J.P.
        • Macaskill P.
        • Steyerberg E.W.
        • et al.
        Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration.
        Ann Intern Med. 2015; 162 (W1–73)
        • Ohlmeier C.
        • Langner I.
        • Hillebrand K.
        • Schmedt N.
        • Mikolajczyk R.
        • Riedel O.
        • et al.
        Mortality in the German Pharmacoepidemiological Research Database (GePaRD) compared to national data in Germany: results from a validation study.
        BMC Public Health. 2015; 15: 570
        • Schanzer A.
        • Goodney P.P.
        • Li Y.
        • Eslami M.
        • Cronenwett J.
        • Messina L.
        • et al.
        Validation of the PIII CLI risk score for the prediction of amputation-free survival in patients undergoing infrainguinal autogenous vein bypass for critical limb ischemia.
        J Vasc Surg. 2009; 50: 769-775
        • Brizuela Sanz J.A.
        • González Fajardo J.A.
        • Taylor J.H.
        • Río Solá L.
        • Muñoz Moreno M.F.
        • Vaquero Puerta C.
        Design of a new risk score in critical limb ischaemia: the ERICVA model.
        Eur J Vasc Endovasc Surg. 2016; 51: 90-99
        • Hackl G.
        • Jud P.
        • Avian A.
        • Gary T.
        • Deutschmann H.
        • Seinost G.
        • et al.
        COPART risk score, endothelial dysfunction, and arterial hypertension are independent risk factors for mortality in claudicants.
        Eur J Vasc Endovasc Surg. 2016; 52: 211-217
        • Arruda-Olson A.M.
        • Afzal N.
        • Priya Mallipeddi V.
        • Said A.
        • Moussa Pacha H.
        • Moon S.
        • et al.
        Leveraging the electronic health record to create an automated real-time prognostic tool for peripheral arterial disease.
        J Am Heart Assoc. 2018; 7e009680
        • Dorresteijn J.A.
        • Visseren F.L.
        • Wassink A.M.
        • Gondrie M.J.
        • Steyerberg E.W.
        • Ridker P.M.
        • et al.
        Development and validation of a prediction rule for recurrent vascular events based on a cohort study of patients with arterial disease: the SMART risk score.
        Heart. 2013; 99: 866-872
        • van Walraven C.
        • Austin P.C.
        • Jennings A.
        • Quan H.
        • Forster A.J.
        A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data.
        Med Care. 2009; 47: 626-633
        • Ambler G.K.
        • Brooks D.E.
        • Al Zuhir N.
        • Ali A.
        • Gohel M.S.
        • Hayes P.D.
        • et al.
        Effect of frailty on short- and mid-term outcomes in vascular surgical patients.
        Br J Surg. 2015; 102: 638-645
        • Lee D.S.
        • Austin P.C.
        • Rouleau J.L.
        • Liu P.P.
        • Naimark D.
        • Tu J.V.
        Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model.
        JAMA. 2003; 290: 2581-2587
        • Goff Jr., D.C.
        • Lloyd-Jones D.M.
        • Bennett G.
        • Coady S.
        • D'Agostino R.B.
        • Gibbons R.
        • et al.
        2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American heart association task force on practice guidelines.
        Circulation. 2014; 129: S49-S73
        • Levine M.E.
        Modeling the rate of senescence: can estimated biological age predict mortality more accurately than chronological age?.
        J Gerontol A Biol Sci Med Sci. 2013; 68: 667-674
        • Secemsky E.A.
        • Kundi H.
        • Weinberg I.
        • Jaff M.R.
        • Krawisz A.
        • Parikh S.A.
        • et al.
        Association of survival with femoropopliteal artery revascularization with drug-coated devices.
        JAMA Cardiol. 2019; 4: 332-340
        • Peters F.
        • Kreutzburg T.
        • Rieß H.C.
        • Heidemann F.
        • Marschall U.
        • L'Hoest H.
        • et al.
        Optimal pharmacological treatment of symptomatic peripheral arterial occlusive disease and evidence of female patient disadvantage: an analysis of health insurance claims data.
        Eur J Vasc Endovasc Surg. 2020; 60: P421-P429
        • Damen J.A.A.G.
        • Hooft L.
        • Schuit E.
        • Debray T.P.A.
        • Collins G.S.
        • Tzoulaki I.
        • et al.
        Prediction models for cardiovascular disease risk in the general population: systematic review.
        BMJ. 2016; 353: i2416
        • Kriston L.
        Machine learning's feet of clay.
        J Eval Clin Pract. 2020; 26: 373-375
        • Beck A.W.
        • Sedrakyan A.
        • Mao J.
        • Venermo M.
        • Faizer R.
        • Debus S.
        • et al.
        Variations in abdominal aortic aneurysm care: a report from the International Consortium of Vascular Registries.
        Circulation. 2016; 134: 1948-1958
        • Steyerberg E.W.
        • Harrell Jr., F.E.
        Prediction models need appropriate internal, internal-external, and external validation.
        J Clin Epidemiol. 2016; 69: 245-247

      Linked Article

      Comments

      Commenting Guidelines

      To submit a comment for a journal article, please use the space above and note the following:

      • We will review submitted comments as soon as possible, striving for within two business days.
      • This forum is intended for constructive dialogue. Comments that are commercial or promotional in nature, pertain to specific medical cases, are not relevant to the article for which they have been submitted, or are otherwise inappropriate will not be posted.
      • We require that commenters identify themselves with names and affiliations.
      • Comments must be in compliance with our Terms & Conditions.
      • Comments are not peer-reviewed.