Advertisement

Editor's Choice – Development and Validation of a Multivariable Prediction Model of Peri-operative Mortality in Vascular Surgery: The New Zealand Vascular Surgical Risk Tool (NZRISK-VASC)

Open ArchivePublished:January 07, 2021DOI:https://doi.org/10.1016/j.ejvs.2020.12.008

      Objective

      Risk calculators and prediction models are available to assist clinicians and patients with peri-operative decision making to optimise outcomes. In a vascular surgical setting, the majority of these models is based on open AAA repair outcomes, and in general their clinical use is limited. The objective of this study was to develop and validate a simple and accurate vascular surgical risk prediction model.

      Methods

      A national administrative database was accessed to collect information on all adult patients undergoing vascular surgery between 1 July 2011 and 30 June 2016 in New Zealand. The primary outcomes were mortality at 30 days, one year, and two years. Previously established covariables including American Society of Anaesthesiologists (ASA) physical status score, sex, surgical urgency, cancer status and ethnicity were tested, and other covariables such as smoking status, presence of renal failure, diabetes, anatomical site of operation, structure operated, and type of procedures (open or endovascular) were explored. LASSO regression was used to select variables for inclusion in the model.

      Results

      A total of 21 597 cases formed the final risk prediction models, with covariables including ASA score, gender, surgical urgency, cancer status, presence of renal failure, diabetes, anatomical site, structure operated, and endovascular procedure. The area under the receiver operating curve (AUROC) for 30 day, one year, and two year mortality using L-min model was 0.869, 0.833, and 0.824, respectively, demonstrating very good discrimination. Calibration with the validation dataset was also excellent, with slopes of 0.971, 1.129, and 1.011, respectively, and McFadden’s pseudo-R2 statistics of 0.250, 0.227, and 0.227, respectively.

      Conclusion

      A simple and accurate multivariable risk calculator for vascular surgical patients was developed and validated using the New Zealand national dataset, with excellent discrimination and calibration for 30 day, one year, and two year mortality.

      Keywords

      Pre-operative decision making is a crucial part of a patient’s management plan, and risk calculators and prediction models are available to assist clinicians and patients during this process to optimise the outcome. There are numerous surgery specific risk prediction tools for vascular surgical patients; however, their general clinical utility is limited. A multivariable risk prediction model was developed that can predict 30 day, one year, and two year mortality in vascular surgical patients. It requires only 10 easily obtainable covariables in the pre-operative setting, making it potentially practical and easy to use.

      Introduction

      Pre-operative decision making is becoming more challenging with the increase in elderly, co-morbid patients undergoing major operations. Peri-operative risk prediction models can be used to provide a more accurate estimate of an individual patient’s risk than clinicians’ estimates,
      • Moonesinghe S.R.
      • Mythen G.M.
      • Das M.P.
      • Rowan W.K.
      • Grocott M.P.
      Risk stratification tools for predicting morbidity and mortality in adult patients undergoing major surgery: qualitative systematic review.
      ,
      • Wijeysundera D.N.
      Predicting outcomes: is there utility in risk scores?.
      and they can be used in peri-operative planning. Vascular surgery carries a high patient mortality burden,
      • Stewart B.
      • Khanduri P.
      • McCord C.
      • Ohene-Yeboah M.
      • Uranues S.
      • Vega Rivera F.
      • et al.
      Global disease burden of conditions requiring emergency surgery.
      and such risk prediction tools may be useful in shared clinical decision making.
      There are a number of prediction models currently available for use in vascular surgery. The majority of instruments is based on open abdominal aortic aneurysm (AAA) repair outcomes to predict in hospital deaths,
      • van Beek S.C.
      • Blankensteijn J.D.
      • Balm R.
      Validation of three models predicting in-hospital death in patients with an abdominal aortic aneurysm eligible for both endovascular and open repair.
      ,
      • Ambler G.K.
      • Gohel M.S.
      • Mitchell D.C.
      • Loftus I.M.
      • Boyle J.R.
      The Abdominal Aortic Aneurysm Statistically Corrected Operative Risk Evaluation (AAA SCORE) for predicting mortality after open and endovascular interventions.
      or longer term survival.
      • Khashram M.
      • Kvizhinadze G.
      • Khashram Z.
      • Williman J.A.
      • Jones G.T.
      • Roake J.A.
      Development and validation of a predictive model to aid in the management of intact abdominal aortic aneurysms.
      A risk prediction tool for carotid endarterectomy among patients with asymptomatic carotid stenosis has also been described,
      • Keyhani S.
      • Madden E.
      • Cheng E.M.
      • Bravata D.M.
      • Halm E.
      • Austin P.C.
      • et al.
      Risk prediction tools to improve patient selection for carotid endarterectomy among patients with asymptomatic carotid stenosis.
      as well as one for an elective infra-inguinal bypass surgery to predict 30 day mortality in elective patients.
      • Gupta P.K.
      • Ramanan B.
      • Lynch T.G.
      • Sundaram A.
      • Mactaggart J.N.
      • Gupta H.
      • et al.
      Development and validation of a risk calculator for prediction of mortality after infrainguinal bypass surgery.
      However, the routine use of such clinical tools in vascular surgical practice is limited compared with more widely incorporated models such as the European System for Cardiac Operative Risk Evaluation (EuroSCORE) for cardiac surgery,
      • Nashef S.A.M.
      • Roques F.
      • Michel P.
      • Gauducheau E.
      • Lemeshow S.
      • Salamon R.
      European system for cardiac operative risk evaluation (EuroSCORE).
      and other non-specialty specific tools such as Surgical Outcome Risk Tool (SORT).
      • Protopapa K.L.
      • Simpson J.C.
      • Smith N.C.
      • Moonesinghe S.R.
      Development and validation of the Surgical Outcome Risk Tool (SORT).
      Other general risk calculators have been used to estimate the risk of adverse cardiac events after vascular surgery including The American College of Surgeons National Surgical Quality Improvement Program (NSQIP), the Revised Cardiac Risk Index (RCRI), and the Vascular Study Group of New England (VSGNE) Cardiac Risk Index.
      • Moses D.A.
      • Johnston L.E.
      • Tracci M.C.
      • Robinson W.P.
      • Cherry K.J.
      • Kern J.A.
      • et al.
      Estimating risk of adverse cardiac event after vascular surgery using currently available online calculators.
      However, they have shown significant variability in their predictability at procedural and institutional levels, generally underestimating the cardiac risk.
      • Moses D.A.
      • Johnston L.E.
      • Tracci M.C.
      • Robinson W.P.
      • Cherry K.J.
      • Kern J.A.
      • et al.
      Estimating risk of adverse cardiac event after vascular surgery using currently available online calculators.
      A multivariable prediction model NZRISK has been developed recently, and validated for patients undergoing non-cardiac surgical procedures in New Zealand.
      • Campbell D.
      • Boyle L.
      • Soakell-Ho M.
      • Hider P.
      • Wilson L.
      • Koea J.
      • et al.
      National risk prediction model for perioperative mortality in non-cardiac surgery.
      Derived from a large, full coverage, national dataset of more than 360 000 national cases, it has demonstrated excellent calibration and discrimination in predicting 30 day, one year, and two year mortality.
      • Campbell D.
      • Boyle L.
      • Soakell-Ho M.
      • Hider P.
      • Wilson L.
      • Koea J.
      • et al.
      National risk prediction model for perioperative mortality in non-cardiac surgery.
      However, NZRISK was designed as a universal calculator and its use may not be optimal in the vascular surgical setting where the nature of some vascular procedures and the higher patient risk profile may differ from other surgical specialties.
      The aim of this study was to develop and validate a specific vascular surgical risk prediction model, using readily available pre-operative variables from a national administrative dataset. The goal was to implement a simple and accessible vascular surgical risk calculator to assist clinicians and patients in their pre-operative shared decision making process.

      Methods

      This manuscript was prepared adhering to the TRIPOD (Transparent Reporting of a multivariable Prediction model for Individual Prognosis Or Diagnosis) guidelines.
      • Collins G.S.
      • Reitsma J.B.
      • Altman D.G.
      • Moons K.G.M.
      Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD).

      TRIPOD. The TRIPOD Statement. Available at: https://www.tripod-statement.org. [Accessed 1 August 2017].

      • Moons K.G.
      • Altman D.G.
      • Reitsma J.B.
      • Ioannidis J.P.
      • Macaskill P.
      • Steyerberg E.W.
      • et al.
      Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration.
      Ethical approval for this study was obtained from the New Zealand National Health and Disability Ethics Committees (HDEC reference 16/CEN/200).

      Sources of data

      Data from the National Minimum Dataset (NMDS) were accessed for patients who had a hospital admission between 1 July 2011 and 30 June 2016 inclusive. The NMDS is a national registry of all inpatient events reported to the New Zealand Ministry of Health, capturing 99% of all surgical procedures performed in New Zealand. Follow up was completed by 30 June 2018, to allow calculation of two year mortality in all participants.

      Participants

      All patients aged ≥18 years in the dataset who underwent a vascular surgical procedure, as defined by the Australian Classification of Health Intervention (ACHI) standardised codes were included. The procedures were identified as non-cardiac vascular operations commonly performed by vascular surgeons in New Zealand. Included vascular surgical procedures are listed in the Supplementary material, Appendix 1. Participants who did not have an associated anaesthetic code were excluded. Anaesthetic codes included general anaesthesia, regional anaesthesia, local anaesthesia, and sedation, and indicated the presence of an anaesthetist for the case. If a patient had multiple procedures in a single admission during the study period, only the first vascular surgical procedure was analysed as the “index procedure”, being the first procedure matching the criteria in each admission. If a patient was re-admitted for a revision operation, this was counted as a new operation.

      Outcomes

      The primary outcomes were mortality at 30 days, one year, and two years, where mortality was defined as the presence of a date of death recorded in the NMDS. There may be a three month delay in the death reporting from New Zealand Births and Deaths Registry to the NMDS,

      New Zealand Government. Births, deaths and marriages. Available at: https://www.govt.nz/organisations/births-deaths-and-marriages/ [Accessed 1 August 2017].

      hence the analysis was performed once full reporting was assured.

      Predictors

      The recently created NZRISK model used eight predictors including patient’s age, sex, urgency of surgery (elective or acute), American Society of Anesthesiologists (ASA) score, surgical grade, surgical specialty, presence of cancer, and ethnicity.
      • Campbell D.
      • Boyle L.
      • Soakell-Ho M.
      • Hider P.
      • Wilson L.
      • Koea J.
      • et al.
      National risk prediction model for perioperative mortality in non-cardiac surgery.
      Six of these confirmatory variables were analysed, excluding the surgical grade and surgical specialty as vascular surgery was found to comprise higher risk groups. Acute surgery included all non-elective operations. ASA score was handled as a categorical predictor with five categories. Cancer status was a binary variable, defined in the NMDS as the presence or absence of a cancer with the potential to affect mortality. Ethnicity was defined by level 1 ethnicity codes reported to the NMDS.
      Additional exploratory predictors relevant to vascular surgery were investigated, including anatomical site of operation, structure operated on, and whether it was an endovascular procedure. Two authors, J.Y.K and M.K., reviewed the procedure codes independently and derived specific categories, which were further reviewed by D.C. The anatomical sites were categorised into abdominal, limb unspecified, supra-inguinal lower limbs, infra-inguinal lower limbs, upper limbs, neck, thorax, and other (Supplementary Appendix 1). The three most prevalent procedures in each anatomical category are provided in Supplementary Appendix 2 (Table S1). Operated structures included major amputation, minor amputation, artery, vein, and arteriovenous (AV).
      Four further exploratory covariables were tested in model development; socioeconomic status, smoking status, presence of renal failure, and diabetes. Socioeconomic status was categorised as 1 to 10 using a national scoring system, NZDep13.
      • Atkinson J.
      • Salmond C.
      • Crampton P.
      NZDep2013 Index of Deprivation.
      Renal failure was categorised into acute, chronic, dialysis dependent and unspecified, using the International Statistical Classification of Diseases and Related Health Problems codes (ICD diagnosis codes, provided in Supplementary Appendix 3). All predictors were coded while in hospital and were reported to the Ministry of Health in New Zealand within 21 days of hospital discharge aggregated in the NMDS.

      Sample size

      A minimum of 10 endpoints per confirmatory covariable and a minimum of 100 endpoints per exploratory covariable are required to construct a robust model.
      • Steyerberg E.W.
      • Vickers A.J.
      • Cook N.R.
      • Gerds T.
      • Gonen M.
      • Obuchowski N.
      • et al.
      Assessing the performance of prediction models: a framework for traditional and novel measures.
      ,
      • Steyerberg E.W.
      • Vergouwe Y.
      Towards better clinical prediction models: seven steps for development and an ABCD for validation.
      The model in the present study had six confirmatory covariables and seven exploratory covariables. Therefore, a conservative estimate of 760 events was required in the dataset for a stable model. The 30 day mortality in the present dataset was 3.5%, requiring a sample size of 21 714 to ensure sufficient events. A five year dataset was used, with 21 597 eligible participants, which ensured model stability.

      Missing data

      It was assumed that all mortality outcomes were recorded because it is mandatory for a date of death to be reported in New Zealand. Participants were excluded if they had missing data. The baseline characteristics of the included and excluded participants were similar in the original modelling,
      • Campbell D.
      • Boyle L.
      • Soakell-Ho M.
      • Hider P.
      • Wilson L.
      • Koea J.
      • et al.
      National risk prediction model for perioperative mortality in non-cardiac surgery.
      so it was assumed data were missing at random and a complete case analysis was undertaken.

      Statistical analysis

      Least absolute shrinkage and selection operator (LASSO) regression analysis helped select the variables for inclusion to construct 30 day, one year, and two year mortality prediction models. The prediction model was derived from a random 75% split of the dataset and validated with the remaining 25%.
      The area under the receiver operating characteristic curve (AUROC) was used to assess model accuracy by estimating discrimination, and the calibration plots and McFadden’s pseudo-R2 statistic were used to assess calibration and the goodness of fit. An AUROC of over 0.8 is taken to represent very good discrimination and McFadden’s pseudo-R2 values of between 0.2 and 0.4 to indicate excellent calibration.
      • Moonesinghe S.R.
      • Mythen G.M.
      • Das M.P.
      • Rowan W.K.
      • Grocott M.P.
      Risk stratification tools for predicting morbidity and mortality in adult patients undergoing major surgery: qualitative systematic review.
      ,
      • Wijeysundera D.N.
      Predicting outcomes: is there utility in risk scores?.
      ,
      • Steyerberg E.W.
      • Vickers A.J.
      • Cook N.R.
      • Gerds T.
      • Gonen M.
      • Obuchowski N.
      • et al.
      Assessing the performance of prediction models: a framework for traditional and novel measures.
      ,
      • Steyerberg E.W.
      • Vergouwe Y.
      Towards better clinical prediction models: seven steps for development and an ABCD for validation.
      The statistical analyses were repeated for optimal performance of the prediction models, with adjustments in the covariables included and the way each covariable was categorised. The lambda-minimum (L-min) model was used to construct the final prediction models for simplicity, statistical accuracy, and clinical plausibility. Model coefficients for each covariable were summed with the intercept constant from the regression model and converted to a predicted mortality risk using the formula:
      Risk score = exp(mortality risk)/(1 + exp(mortality risk))


      Results

      The final analysis dataset comprised 21 597 adult patients who underwent vascular surgical procedures in New Zealand between 1 July 2011 and 30 June 2016 (Fig. 1). Ten of the initially tested 13 covariables were selected for inclusion in the final models: age, sex, urgency of surgery (elective or acute), ASA score, cancer status, anatomical site, structure, endovascular procedure, and presence of renal failure and diabetes. The descriptive data of the final analysis dataset are presented in Table 1. Ethnicity, smoking status, and socioeconomic status were removed from the final models as they did not show statistically significant effect on mortality, thus did not contribute to improvement in the models’ predictive performance. A final sample size of 16 197 for the derivation cohort and 5 400 for the validation cohort was used in NZRISK-VASC. Descriptive data for these cohorts are presented in Supplementary Appendix 2 (Table S2).
      Figure 1
      Figure 1Flow diagram showing how the final participants for analysis were selected for the development and validation of a multivariable prediction model of peri-operative mortality in vascular surgery: New Zealand vascular surgical risk tool (NZRISK-VASC).
      Table 1Descriptive data for the analysis dataset for New Zealand vascular surgical risk tool (NZRISK-VASC)
      Analysis dataset (n=21 597)
      Mortality
       30 days756 (3.5)
       One year2 959 (13.7)
       Two years7 883 (36.5)
      Age – y65 (52–75)
      Sex
       Male13 632 (63.1)
       Female7 965 (36.9)
      Urgency of surgery
       Elective12 879 (59.6)
       Acute8 718 (40.4)
      ASA score
       12 273 (10.5)
       25 215 (24.1)
       310 892 (50.4)
       43 063 (14.2)
       5154 (0.7)
      Cancer present
       Yes850 (3.9)
       No20 747 (96.1)
      Anatomical site
       Abdominal2 728 (12.6)
       Limbs unspecified985 (4.6)
       Supra-inguinal lower limbs1 121 (5.2)
       Infra-inguinal lower limbs7 677 (35.5)
       Upper limbs3 269 (15.1)
       Neck1 810 (8.4)
       Thorax117 (0.5)
       Other3 890 (18.0)
      Structures
       Major amputation810 (3.8)
       Minor amputation3 079 (14.3)
       Artery11 391 (52.7)
       Veins2 586 (12.0)
       Arteriovenous3 731 (17.3)
      Endovascular procedure
       Yes1 984 (9.2)
       No19 613 (90.8)
      Diabetes
       Yes5 939 (27.5)
       No15 658 (72.5)
      Renal failure
       Acute1 287 (6.0)
       Chronic898 (4.2)
       Dialysis dependent2 643 (12.2)
       No16 769 (77.6)
      Data are presented as n (%) or median (interquartile range).
      From this dataset, a multivariable risk calculation model was developed for each of the primary outcomes: 30 day, one year, and two year mortality (Table 2).
      Table 2New Zealand vascular surgical risk tool (NZRISK-VASC) multivariable prediction models of mortality at 30 days, one year, and two years
      Model coefficients
      30 daysOne yearTwo years
      Intercept – k−6.475−6.360−6.246
      Age per year0.0210.0390.044
      Male sex−0.459−0.122−0.037
      Emergency surgery0.8740.6040.534
      ASA score
       2
       30.6050.9000.902
       41.7331.8391.813
       53.3183.1022.782
      Cancer present0.8271.5701.705
      Anatomical site
       Abdominal0.663-0.098
       Limbs unspecified−0.157
       Supra-inguinal lower limbs0.1670.3440.168
       Upper limbs−0.091−0.013
       Neck−0.023−0.516−0.611
       Thorax0.906−0.212
       Other0.4610.073
      Structures
       Major amputation1.0600.9570.951
       Minor amputation0.2170.250
       Artery0.214
       Arteriovenous−1.004−0.189
      Endovascular procedure−0.1290.0540.116
      Diabetes present0.0770.3060.289
      Renal failure
       Acute1.2620.8500.667
       Chronic0.3210.3870.383
       Dialysis dependent0.4160.6800.776
      The strongest 30 day mortality predictor was higher ASA score. Patients with ASA scores four and five had odds ratios (OR) of 5.66 and 27.61, respectively, for 30 day mortality, when compared with patients with an ASA score of one (Table 3). Other significant covariables associated with 30 day mortality included urgency of the surgery, anatomical site of the operation, structure operated on, and presence of renal failure. Procedures on abdomen and thorax conferred higher risk (OR 1.94 and 2.47, respectively) when compared with infra-inguinal lower limb operations. Major amputation was also associated with higher risk when compared with operations on venous structures with OR of 2.89. Final one and two year prediction model results are provided in Supplementary Appendix 2 (Tables S3 and S4).
      Table 3New Zealand vascular surgical risk tool (NZRISK-VASC) final 30 day mortality prediction model results with model coefficients and odds ratio
      Co-variablesL-min
      L-min = Lambda-minimum; the model selected for use.
      Odds ratio
      Odds ratio has been rounded to two decimal points.
      Age per year0.0211.02
      Male sex−0.4590.63
      Acute urgency of surgery0.8742.40
      ASA score
       201
       30.6051.83
       41.7335.66
       53.31827.61
      Cancer presence0.8272.29
      Anatomical site
       Abdominal0.6631.94
       Limbs unspecified−0.1570.85
       Supra-inguinal lower limbs0.1671.18
       Upper limbs−0.0910.91
       Neck−0.0230.98
       Thorax0.9062.47
       Other0.4611.59
      Structures
       Major amputation1.0602.89
       Minor amputation01
       Artery0.2141.24
       Arteriovenous (AV)−1.0040.37
      Endovascular procedure−0.1290.88
      Diabetes0.0771.08
      Renal failure
       Acute1.2623.53
       Chronic0.3211.38
       Dialysis dependent0.4161.52
      L-min = Lambda-minimum; the model selected for use.
      Odds ratio has been rounded to two decimal points.
      The three final risk prediction models were assessed for performance and generalisability with the validation dataset. The AUROC for 30 day, one year, and two year mortality using the L-min model were 0.869, 0.833, and 0.824, respectively, representing very good discrimination across risk groups. Calibration with the validation dataset was excellent, shown by McFadden’s pseudo-R2 statistics of 0.250, 0.227, and 0.227, respectively, and calibration slopes of 0.971, 1.129, and 1.011, respectively (Fig. 2).
      Figure 2
      Figure 2Calibration plots of New Zealand vascular surgical risk tool (NZRISK-VASC) for mortality prediction models at (A) 30 days, (B) one year, and (C) two years. Intercept −0.0032, −0.015, −0.004; slope 0.971, 1.129, 1.011; area under the receiver operating characteristics (AUROC) 0.869, 0.833, 0.824; McFadden’s pseudo-R2 0.250, 0.227, 0.227 for the three models, respectively.

      Discussion

      In this study, one of the first multivariable risk prediction tools specifically designed for the vascular surgical population has been created, based on a large national dataset of more than 21 000 patients in New Zealand. The 10 covariables included in the final models are easily obtainable in the pre-operative period, making this tool both practical and easy to use during peri-operative planning. The concept of risk assessment to determine surgical fitness and appropriateness for vascular intervention is particularly relevant and challenging today given the increase in the minimally invasive surgical population in both volume and complexity, necessitating improved patient selection.
      • Kolh P.
      • De Hert S.
      • De Rango P.
      The concept of risk assessment and being unfit for surgery.
      NZRISK-VASC shows both excellent calibration and ability to discriminate between patients at low risk and those at high risk when tested against a validation dataset. Addition of one year and two year mortality to 30 day mortality in the prediction model also improves its clinical utility, as a 30 day model alone can underestimate mortality risk associated with surgery in patients at higher risk.
      • Moonesinghe S.R.
      • Mythen M.G.
      • Grocott M.P.
      High-risk surgery: epidemiology and outcomes.
      Furthermore, NZRISK-VASC has superior predictive performance for the vascular surgical population compared with the existing NZRISK tool for non-cardiac surgical patients
      • Campbell D.
      • Boyle L.
      • Soakell-Ho M.
      • Hider P.
      • Wilson L.
      • Koea J.
      • et al.
      National risk prediction model for perioperative mortality in non-cardiac surgery.
      ; the AUC for 30 day, one year, and two year mortality are 0.869, 0.833, and 0.824, respectively, compared with 0.833, 0.762, and 0.748 in the original NZRISK models. Options for less invasive operations, a non-operative strategy, or even palliative treatment may be discussed if a patient’s one or two year mortality is found to be high with the prediction model; however, caution needs to be exercised and clinical correlation of individual patient circumstances must be taken into consideration when making such decisions. This is because while a risk assessment tool like NZRISK-VASC performs well on the validation cohort, it does not incorporate other salient features required in the shared decision making process.
      NZRISK-VASC has been developed and validated on all vascular surgical procedures except procedures that did not require an anaesthetist and were therefore missing anaesthetic codes and ASA score (see Fig. 1 and Appendix Table S5). This is unlike other currently available risk prediction models that are more disease and surgery specific,
      • Carlisle J.B.
      • Danjoux G.
      • Kerr K.
      • Snowden C.
      • Swart M.
      Validation of long-term survival prediction for scheduled abdominal aneurysm repair with an independent calculator using only pre-operative variables.
      and therefore the results are more generalisable. In addition to its clinical utility, it can be used as an audit tool with some of the suggested important risk factors comparing other NZ centres.
      Ethnicity, smoking status, and socioeconomic status did not contribute to the models as previously hypothesised. Ethnicity has been seen to correlate with surgical outcomes in New Zealand,
      • Khashram M.
      • Pitama S.
      • Wiliman J.A.
      • Jones G.T.
      • Roake J.A.
      Survival disparity following abdominal aortic aneurysm repair highlights inequality in ethic and socio-economic status.
      and it was found to be an important covariable in the original NZRISK model.
      • Campbell D.
      • Boyle L.
      • Soakell-Ho M.
      • Hider P.
      • Wilson L.
      • Koea J.
      • et al.
      National risk prediction model for perioperative mortality in non-cardiac surgery.
      There is also increasing appreciation of the importance of socioeconomic status to health outcomes in the medical literature. However, neither were found to be predictive of mortality in the present dataset. Further exploration of these risk factors for future models may be warranted. However, exclusion of ethnicity and socioeconomic status may allow NZRISK-VASC to be externally validated by other international vascular surgery datasets. Further studies will enhance the robustness of the validation process.
      The study has limitations. The data used in development and validation of the three prediction models are all from the New Zealand national dataset, thus their external validity and generalisability to international populations have yet to be established, and they require recalibration before appropriate use. Temporal validation of the risk model is currently being performed to address some of these concerns. The data are also based on the ACHI codes for the surgical procedures, and the accuracy of the use of these codes has not been published apart from studies that have also used the codes for audit purposes.
      • Khashram M.
      • Thomson I.A.
      • Jones G.T.
      • Roake J.A.
      Abdominal aortic aneurysm repair in New Zealand: a validation of the Australasian Vascular Audit.
      However, ACHI codes are the basis of mandatory reporting of surgical procedure type to the Ministry of Health in New Zealand and are the basis of government funding for surgery to the District Health Boards.
      There are other limitations to the use of the procedure codes. For example, for endovascular procedures, there was only one ACHI code available, “endovascular repair of aneurysm” (see Supplementary Appendix 1), thus it did not allow further categorisation of the location of the aneurysm or the complexity of the repair. For the purpose of development of the risk prediction models, this was classified into the abdomen anatomical category.
      The data for the study come from a retrospective administrative dataset, with the limitations associated with datasets of this type, such as possible bias in the data resulting from the way in which they were collected, which would be reflected in the model. For three covariables (i.e. cancer, diabetes, renal failure), a small proportion of patients would have had these diagnoses added at the time of surgical admission. However, the vast majority of these health conditions was present pre-operatively in the database as pre-existing conditions. This may produce a subtle bias in model performance when using the information available at the time. Another limitation is that only the deaths that occurred in New Zealand would have been captured.
      Although a seemingly high number of patients had missing ASA score and were excluded, the characteristics of the participants with missing data and those in the full analysis dataset were very similar, as illustrated in Table S5 (Supplementary Appendix 2). There was also a relatively low proportion of endovascular procedures (9.2%) in the dataset. However, almost half of all procedures in the dataset were non-arterial (veins, AV fistula, amputations) which are not amenable for an endovascular procedure. Another possible explanation is that most elective endovascular procedures are usually performed as day cases in New Zealand without an anaesthetist, thus the missing ASA scores and exclusion from this study. With these limitations, it is important to understand that this tool is to complement clinical assessment rather than to replace it entirely.
      In conclusion, NZRISK-VASC is a simple and accurate multivariable risk prediction model for peri-operative mortality in vascular surgical patients. It requires only 10 easily obtainable covariables in the pre-operative setting and can help with the shared clinical decision making process with patients. The general NZRISK tool
      • Campbell D.
      • Boyle L.
      • Soakell-Ho M.
      • Hider P.
      • Wilson L.
      • Koea J.
      • et al.
      National risk prediction model for perioperative mortality in non-cardiac surgery.
      is currently freely available online for clinicians to use at www.nzrisk.com, and the NZRISK-VASC will become available when the website is upgraded in mid-2021.

      Funding

      This work was supported by a grant from Precision Driven Health (PDH), which provided research funding, work time for L.B. and scientific support.

      Conflict of interest

      D.C. has received research funding from PDH, Auckland District Health Board Research Trust, the Australian and New Zealand College of Anaesthetists and the Neurological Foundation of New Zealand. The remaining authors declare no conflicts of interest.

      Appendix A. Supplementary data

      The following are the Supplementary data to this article:

      References

        • Moonesinghe S.R.
        • Mythen G.M.
        • Das M.P.
        • Rowan W.K.
        • Grocott M.P.
        Risk stratification tools for predicting morbidity and mortality in adult patients undergoing major surgery: qualitative systematic review.
        Anesthesiology. 2013; 119: 959-981
        • Wijeysundera D.N.
        Predicting outcomes: is there utility in risk scores?.
        Can J Anaesth. 2016; 63: 148-158
        • Stewart B.
        • Khanduri P.
        • McCord C.
        • Ohene-Yeboah M.
        • Uranues S.
        • Vega Rivera F.
        • et al.
        Global disease burden of conditions requiring emergency surgery.
        Br J Surg. 2014; 101: e9-e22
        • van Beek S.C.
        • Blankensteijn J.D.
        • Balm R.
        Validation of three models predicting in-hospital death in patients with an abdominal aortic aneurysm eligible for both endovascular and open repair.
        J Vasc Surg. 2013; 58 (1452–7.e1)
        • Ambler G.K.
        • Gohel M.S.
        • Mitchell D.C.
        • Loftus I.M.
        • Boyle J.R.
        The Abdominal Aortic Aneurysm Statistically Corrected Operative Risk Evaluation (AAA SCORE) for predicting mortality after open and endovascular interventions.
        J Vasc Surg. 2015; 61 (e1): 35-43
        • Khashram M.
        • Kvizhinadze G.
        • Khashram Z.
        • Williman J.A.
        • Jones G.T.
        • Roake J.A.
        Development and validation of a predictive model to aid in the management of intact abdominal aortic aneurysms.
        Eur J Vasc Endovasc Surg. 2018; 56: 48-56
        • Keyhani S.
        • Madden E.
        • Cheng E.M.
        • Bravata D.M.
        • Halm E.
        • Austin P.C.
        • et al.
        Risk prediction tools to improve patient selection for carotid endarterectomy among patients with asymptomatic carotid stenosis.
        JAMA Surg. 2019; 154: 336-344
        • Gupta P.K.
        • Ramanan B.
        • Lynch T.G.
        • Sundaram A.
        • Mactaggart J.N.
        • Gupta H.
        • et al.
        Development and validation of a risk calculator for prediction of mortality after infrainguinal bypass surgery.
        J Vasc Surg. 2012; 56 (e1): 372-379
        • Nashef S.A.M.
        • Roques F.
        • Michel P.
        • Gauducheau E.
        • Lemeshow S.
        • Salamon R.
        European system for cardiac operative risk evaluation (EuroSCORE).
        Eur J Cardiothorac Surg. 1999; 16: 9-13
        • Protopapa K.L.
        • Simpson J.C.
        • Smith N.C.
        • Moonesinghe S.R.
        Development and validation of the Surgical Outcome Risk Tool (SORT).
        Br J Surg. 2014; 101: 1774-1783
        • Moses D.A.
        • Johnston L.E.
        • Tracci M.C.
        • Robinson W.P.
        • Cherry K.J.
        • Kern J.A.
        • et al.
        Estimating risk of adverse cardiac event after vascular surgery using currently available online calculators.
        J Vasc Surg. 2018; 67: 272-278
        • Campbell D.
        • Boyle L.
        • Soakell-Ho M.
        • Hider P.
        • Wilson L.
        • Koea J.
        • et al.
        National risk prediction model for perioperative mortality in non-cardiac surgery.
        Br J Surg. 2019; 106: 1549-1557
        • Collins G.S.
        • Reitsma J.B.
        • Altman D.G.
        • Moons K.G.M.
        Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD).
        Ann Intern Med. 2015; 162: 55-63
      1. TRIPOD. The TRIPOD Statement. Available at: https://www.tripod-statement.org. [Accessed 1 August 2017].

        • Moons K.G.
        • Altman D.G.
        • Reitsma J.B.
        • Ioannidis J.P.
        • Macaskill P.
        • Steyerberg E.W.
        • et al.
        Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration.
        Ann Intern Med. 2015; 162 (W1–73)
        • Ministry of Health
        National minimum dataset (hospital events).
        2015 (Available at:)
      2. New Zealand Government. Births, deaths and marriages. Available at: https://www.govt.nz/organisations/births-deaths-and-marriages/ [Accessed 1 August 2017].

      3. Ministry of Health. Ethnicity code tables. Available at: http://www.health.govt.nz/nz-health-statistics/data-references/code-tables/common-code-tables/ethnicity-code-tables [Accessed 1 August 2017].

        • Atkinson J.
        • Salmond C.
        • Crampton P.
        NZDep2013 Index of Deprivation.
        University of Otago, Dunedin2014 (Available at:)
        • Steyerberg E.W.
        • Vickers A.J.
        • Cook N.R.
        • Gerds T.
        • Gonen M.
        • Obuchowski N.
        • et al.
        Assessing the performance of prediction models: a framework for traditional and novel measures.
        Epidemiology. 2010; 21: 128-138
        • Steyerberg E.W.
        • Vergouwe Y.
        Towards better clinical prediction models: seven steps for development and an ABCD for validation.
        Eur Heart J. 2014; 35: 1925-1931
        • Kolh P.
        • De Hert S.
        • De Rango P.
        The concept of risk assessment and being unfit for surgery.
        Eur J Vasc Endovasc Surg. 2016; 51: 857-866
        • Moonesinghe S.R.
        • Mythen M.G.
        • Grocott M.P.
        High-risk surgery: epidemiology and outcomes.
        Anesth Analg. 2011; 112: 891-901
        • Carlisle J.B.
        • Danjoux G.
        • Kerr K.
        • Snowden C.
        • Swart M.
        Validation of long-term survival prediction for scheduled abdominal aneurysm repair with an independent calculator using only pre-operative variables.
        Anaesthesia. 2015; 70: 654-665
        • Khashram M.
        • Pitama S.
        • Wiliman J.A.
        • Jones G.T.
        • Roake J.A.
        Survival disparity following abdominal aortic aneurysm repair highlights inequality in ethic and socio-economic status.
        Eur J Vasc Endovasc Surg. 2017; 54: 689-696
        • Khashram M.
        • Thomson I.A.
        • Jones G.T.
        • Roake J.A.
        Abdominal aortic aneurysm repair in New Zealand: a validation of the Australasian Vascular Audit.
        ANZ J Surg. 2017; 87: 394-398

      Comments

      Commenting Guidelines

      To submit a comment for a journal article, please use the space above and note the following:

      • We will review submitted comments as soon as possible, striving for within two business days.
      • This forum is intended for constructive dialogue. Comments that are commercial or promotional in nature, pertain to specific medical cases, are not relevant to the article for which they have been submitted, or are otherwise inappropriate will not be posted.
      • We require that commenters identify themselves with names and affiliations.
      • Comments must be in compliance with our Terms & Conditions.
      • Comments are not peer-reviewed.