If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Editor's Choice – Development and Validation of a Multivariable Prediction Model of Peri-operative Mortality in Vascular Surgery: The New Zealand Vascular Surgical Risk Tool (NZRISK-VASC)
Risk calculators and prediction models are available to assist clinicians and patients with peri-operative decision making to optimise outcomes. In a vascular surgical setting, the majority of these models is based on open AAA repair outcomes, and in general their clinical use is limited. The objective of this study was to develop and validate a simple and accurate vascular surgical risk prediction model.
Methods
A national administrative database was accessed to collect information on all adult patients undergoing vascular surgery between 1 July 2011 and 30 June 2016 in New Zealand. The primary outcomes were mortality at 30 days, one year, and two years. Previously established covariables including American Society of Anaesthesiologists (ASA) physical status score, sex, surgical urgency, cancer status and ethnicity were tested, and other covariables such as smoking status, presence of renal failure, diabetes, anatomical site of operation, structure operated, and type of procedures (open or endovascular) were explored. LASSO regression was used to select variables for inclusion in the model.
Results
A total of 21 597 cases formed the final risk prediction models, with covariables including ASA score, gender, surgical urgency, cancer status, presence of renal failure, diabetes, anatomical site, structure operated, and endovascular procedure. The area under the receiver operating curve (AUROC) for 30 day, one year, and two year mortality using L-min model was 0.869, 0.833, and 0.824, respectively, demonstrating very good discrimination. Calibration with the validation dataset was also excellent, with slopes of 0.971, 1.129, and 1.011, respectively, and McFadden’s pseudo-R2 statistics of 0.250, 0.227, and 0.227, respectively.
Conclusion
A simple and accurate multivariable risk calculator for vascular surgical patients was developed and validated using the New Zealand national dataset, with excellent discrimination and calibration for 30 day, one year, and two year mortality.
Pre-operative decision making is a crucial part of a patient’s management plan, and risk calculators and prediction models are available to assist clinicians and patients during this process to optimise the outcome. There are numerous surgery specific risk prediction tools for vascular surgical patients; however, their general clinical utility is limited. A multivariable risk prediction model was developed that can predict 30 day, one year, and two year mortality in vascular surgical patients. It requires only 10 easily obtainable covariables in the pre-operative setting, making it potentially practical and easy to use.
Introduction
Pre-operative decision making is becoming more challenging with the increase in elderly, co-morbid patients undergoing major operations. Peri-operative risk prediction models can be used to provide a more accurate estimate of an individual patient’s risk than clinicians’ estimates,
and such risk prediction tools may be useful in shared clinical decision making.
There are a number of prediction models currently available for use in vascular surgery. The majority of instruments is based on open abdominal aortic aneurysm (AAA) repair outcomes to predict in hospital deaths,
The Abdominal Aortic Aneurysm Statistically Corrected Operative Risk Evaluation (AAA SCORE) for predicting mortality after open and endovascular interventions.
However, the routine use of such clinical tools in vascular surgical practice is limited compared with more widely incorporated models such as the European System for Cardiac Operative Risk Evaluation (EuroSCORE) for cardiac surgery,
Other general risk calculators have been used to estimate the risk of adverse cardiac events after vascular surgery including The American College of Surgeons National Surgical Quality Improvement Program (NSQIP), the Revised Cardiac Risk Index (RCRI), and the Vascular Study Group of New England (VSGNE) Cardiac Risk Index.
However, they have shown significant variability in their predictability at procedural and institutional levels, generally underestimating the cardiac risk.
A multivariable prediction model NZRISK has been developed recently, and validated for patients undergoing non-cardiac surgical procedures in New Zealand.
Derived from a large, full coverage, national dataset of more than 360 000 national cases, it has demonstrated excellent calibration and discrimination in predicting 30 day, one year, and two year mortality.
However, NZRISK was designed as a universal calculator and its use may not be optimal in the vascular surgical setting where the nature of some vascular procedures and the higher patient risk profile may differ from other surgical specialties.
The aim of this study was to develop and validate a specific vascular surgical risk prediction model, using readily available pre-operative variables from a national administrative dataset. The goal was to implement a simple and accessible vascular surgical risk calculator to assist clinicians and patients in their pre-operative shared decision making process.
Methods
This manuscript was prepared adhering to the TRIPOD (Transparent Reporting of a multivariable Prediction model for Individual Prognosis Or Diagnosis) guidelines.
The NMDS is a national registry of all inpatient events reported to the New Zealand Ministry of Health, capturing 99% of all surgical procedures performed in New Zealand.
Follow up was completed by 30 June 2018, to allow calculation of two year mortality in all participants.
Participants
All patients aged ≥18 years in the dataset who underwent a vascular surgical procedure, as defined by the Australian Classification of Health Intervention (ACHI) standardised codes were included. The procedures were identified as non-cardiac vascular operations commonly performed by vascular surgeons in New Zealand. Included vascular surgical procedures are listed in the Supplementary material, Appendix 1. Participants who did not have an associated anaesthetic code were excluded. Anaesthetic codes included general anaesthesia, regional anaesthesia, local anaesthesia, and sedation, and indicated the presence of an anaesthetist for the case. If a patient had multiple procedures in a single admission during the study period, only the first vascular surgical procedure was analysed as the “index procedure”, being the first procedure matching the criteria in each admission. If a patient was re-admitted for a revision operation, this was counted as a new operation.
Outcomes
The primary outcomes were mortality at 30 days, one year, and two years, where mortality was defined as the presence of a date of death recorded in the NMDS. There may be a three month delay in the death reporting from New Zealand Births and Deaths Registry to the NMDS,
hence the analysis was performed once full reporting was assured.
Predictors
The recently created NZRISK model used eight predictors including patient’s age, sex, urgency of surgery (elective or acute), American Society of Anesthesiologists (ASA) score, surgical grade, surgical specialty, presence of cancer, and ethnicity.
Six of these confirmatory variables were analysed, excluding the surgical grade and surgical specialty as vascular surgery was found to comprise higher risk groups. Acute surgery included all non-elective operations. ASA score was handled as a categorical predictor with five categories. Cancer status was a binary variable, defined in the NMDS as the presence or absence of a cancer with the potential to affect mortality. Ethnicity was defined by level 1 ethnicity codes reported to the NMDS.
Additional exploratory predictors relevant to vascular surgery were investigated, including anatomical site of operation, structure operated on, and whether it was an endovascular procedure. Two authors, J.Y.K and M.K., reviewed the procedure codes independently and derived specific categories, which were further reviewed by D.C. The anatomical sites were categorised into abdominal, limb unspecified, supra-inguinal lower limbs, infra-inguinal lower limbs, upper limbs, neck, thorax, and other (Supplementary Appendix 1). The three most prevalent procedures in each anatomical category are provided in Supplementary Appendix 2 (Table S1). Operated structures included major amputation, minor amputation, artery, vein, and arteriovenous (AV).
Four further exploratory covariables were tested in model development; socioeconomic status, smoking status, presence of renal failure, and diabetes. Socioeconomic status was categorised as 1 to 10 using a national scoring system, NZDep13.
Renal failure was categorised into acute, chronic, dialysis dependent and unspecified, using the International Statistical Classification of Diseases and Related Health Problems codes (ICD diagnosis codes, provided in Supplementary Appendix 3). All predictors were coded while in hospital and were reported to the Ministry of Health in New Zealand within 21 days of hospital discharge aggregated in the NMDS.
Sample size
A minimum of 10 endpoints per confirmatory covariable and a minimum of 100 endpoints per exploratory covariable are required to construct a robust model.
The model in the present study had six confirmatory covariables and seven exploratory covariables. Therefore, a conservative estimate of 760 events was required in the dataset for a stable model. The 30 day mortality in the present dataset was 3.5%, requiring a sample size of 21 714 to ensure sufficient events. A five year dataset was used, with 21 597 eligible participants, which ensured model stability.
Missing data
It was assumed that all mortality outcomes were recorded because it is mandatory for a date of death to be reported in New Zealand. Participants were excluded if they had missing data. The baseline characteristics of the included and excluded participants were similar in the original modelling,
so it was assumed data were missing at random and a complete case analysis was undertaken.
Statistical analysis
Least absolute shrinkage and selection operator (LASSO) regression analysis helped select the variables for inclusion to construct 30 day, one year, and two year mortality prediction models. The prediction model was derived from a random 75% split of the dataset and validated with the remaining 25%.
The area under the receiver operating characteristic curve (AUROC) was used to assess model accuracy by estimating discrimination, and the calibration plots and McFadden’s pseudo-R2 statistic were used to assess calibration and the goodness of fit. An AUROC of over 0.8 is taken to represent very good discrimination and McFadden’s pseudo-R2 values of between 0.2 and 0.4 to indicate excellent calibration.
The statistical analyses were repeated for optimal performance of the prediction models, with adjustments in the covariables included and the way each covariable was categorised. The lambda-minimum (L-min) model was used to construct the final prediction models for simplicity, statistical accuracy, and clinical plausibility. Model coefficients for each covariable were summed with the intercept constant from the regression model and converted to a predicted mortality risk using the formula:
The final analysis dataset comprised 21 597 adult patients who underwent vascular surgical procedures in New Zealand between 1 July 2011 and 30 June 2016 (Fig. 1). Ten of the initially tested 13 covariables were selected for inclusion in the final models: age, sex, urgency of surgery (elective or acute), ASA score, cancer status, anatomical site, structure, endovascular procedure, and presence of renal failure and diabetes. The descriptive data of the final analysis dataset are presented in Table 1. Ethnicity, smoking status, and socioeconomic status were removed from the final models as they did not show statistically significant effect on mortality, thus did not contribute to improvement in the models’ predictive performance. A final sample size of 16 197 for the derivation cohort and 5 400 for the validation cohort was used in NZRISK-VASC. Descriptive data for these cohorts are presented in Supplementary Appendix 2 (Table S2).
Figure 1Flow diagram showing how the final participants for analysis were selected for the development and validation of a multivariable prediction model of peri-operative mortality in vascular surgery: New Zealand vascular surgical risk tool (NZRISK-VASC).
From this dataset, a multivariable risk calculation model was developed for each of the primary outcomes: 30 day, one year, and two year mortality (Table 2).
Table 2New Zealand vascular surgical risk tool (NZRISK-VASC) multivariable prediction models of mortality at 30 days, one year, and two years
The strongest 30 day mortality predictor was higher ASA score. Patients with ASA scores four and five had odds ratios (OR) of 5.66 and 27.61, respectively, for 30 day mortality, when compared with patients with an ASA score of one (Table 3). Other significant covariables associated with 30 day mortality included urgency of the surgery, anatomical site of the operation, structure operated on, and presence of renal failure. Procedures on abdomen and thorax conferred higher risk (OR 1.94 and 2.47, respectively) when compared with infra-inguinal lower limb operations. Major amputation was also associated with higher risk when compared with operations on venous structures with OR of 2.89. Final one and two year prediction model results are provided in Supplementary Appendix 2 (Tables S3 and S4).
Table 3New Zealand vascular surgical risk tool (NZRISK-VASC) final 30 day mortality prediction model results with model coefficients and odds ratio
The three final risk prediction models were assessed for performance and generalisability with the validation dataset. The AUROC for 30 day, one year, and two year mortality using the L-min model were 0.869, 0.833, and 0.824, respectively, representing very good discrimination across risk groups. Calibration with the validation dataset was excellent, shown by McFadden’s pseudo-R2 statistics of 0.250, 0.227, and 0.227, respectively, and calibration slopes of 0.971, 1.129, and 1.011, respectively (Fig. 2).
Figure 2Calibration plots of New Zealand vascular surgical risk tool (NZRISK-VASC) for mortality prediction models at (A) 30 days, (B) one year, and (C) two years. Intercept −0.0032, −0.015, −0.004; slope 0.971, 1.129, 1.011; area under the receiver operating characteristics (AUROC) 0.869, 0.833, 0.824; McFadden’s pseudo-R2 0.250, 0.227, 0.227 for the three models, respectively.
In this study, one of the first multivariable risk prediction tools specifically designed for the vascular surgical population has been created, based on a large national dataset of more than 21 000 patients in New Zealand. The 10 covariables included in the final models are easily obtainable in the pre-operative period, making this tool both practical and easy to use during peri-operative planning. The concept of risk assessment to determine surgical fitness and appropriateness for vascular intervention is particularly relevant and challenging today given the increase in the minimally invasive surgical population in both volume and complexity, necessitating improved patient selection.
NZRISK-VASC shows both excellent calibration and ability to discriminate between patients at low risk and those at high risk when tested against a validation dataset. Addition of one year and two year mortality to 30 day mortality in the prediction model also improves its clinical utility, as a 30 day model alone can underestimate mortality risk associated with surgery in patients at higher risk.
Furthermore, NZRISK-VASC has superior predictive performance for the vascular surgical population compared with the existing NZRISK tool for non-cardiac surgical patients
; the AUC for 30 day, one year, and two year mortality are 0.869, 0.833, and 0.824, respectively, compared with 0.833, 0.762, and 0.748 in the original NZRISK models. Options for less invasive operations, a non-operative strategy, or even palliative treatment may be discussed if a patient’s one or two year mortality is found to be high with the prediction model; however, caution needs to be exercised and clinical correlation of individual patient circumstances must be taken into consideration when making such decisions. This is because while a risk assessment tool like NZRISK-VASC performs well on the validation cohort, it does not incorporate other salient features required in the shared decision making process.
NZRISK-VASC has been developed and validated on all vascular surgical procedures except procedures that did not require an anaesthetist and were therefore missing anaesthetic codes and ASA score (see Fig. 1 and Appendix Table S5). This is unlike other currently available risk prediction models that are more disease and surgery specific,
and therefore the results are more generalisable. In addition to its clinical utility, it can be used as an audit tool with some of the suggested important risk factors comparing other NZ centres.
Ethnicity, smoking status, and socioeconomic status did not contribute to the models as previously hypothesised. Ethnicity has been seen to correlate with surgical outcomes in New Zealand,
There is also increasing appreciation of the importance of socioeconomic status to health outcomes in the medical literature. However, neither were found to be predictive of mortality in the present dataset. Further exploration of these risk factors for future models may be warranted. However, exclusion of ethnicity and socioeconomic status may allow NZRISK-VASC to be externally validated by other international vascular surgery datasets. Further studies will enhance the robustness of the validation process.
The study has limitations. The data used in development and validation of the three prediction models are all from the New Zealand national dataset, thus their external validity and generalisability to international populations have yet to be established, and they require recalibration before appropriate use. Temporal validation of the risk model is currently being performed to address some of these concerns. The data are also based on the ACHI codes for the surgical procedures, and the accuracy of the use of these codes has not been published apart from studies that have also used the codes for audit purposes.
However, ACHI codes are the basis of mandatory reporting of surgical procedure type to the Ministry of Health in New Zealand and are the basis of government funding for surgery to the District Health Boards.
There are other limitations to the use of the procedure codes. For example, for endovascular procedures, there was only one ACHI code available, “endovascular repair of aneurysm” (see Supplementary Appendix 1), thus it did not allow further categorisation of the location of the aneurysm or the complexity of the repair. For the purpose of development of the risk prediction models, this was classified into the abdomen anatomical category.
The data for the study come from a retrospective administrative dataset, with the limitations associated with datasets of this type, such as possible bias in the data resulting from the way in which they were collected, which would be reflected in the model. For three covariables (i.e. cancer, diabetes, renal failure), a small proportion of patients would have had these diagnoses added at the time of surgical admission. However, the vast majority of these health conditions was present pre-operatively in the database as pre-existing conditions. This may produce a subtle bias in model performance when using the information available at the time. Another limitation is that only the deaths that occurred in New Zealand would have been captured.
Although a seemingly high number of patients had missing ASA score and were excluded, the characteristics of the participants with missing data and those in the full analysis dataset were very similar, as illustrated in Table S5 (Supplementary Appendix 2). There was also a relatively low proportion of endovascular procedures (9.2%) in the dataset. However, almost half of all procedures in the dataset were non-arterial (veins, AV fistula, amputations) which are not amenable for an endovascular procedure. Another possible explanation is that most elective endovascular procedures are usually performed as day cases in New Zealand without an anaesthetist, thus the missing ASA scores and exclusion from this study. With these limitations, it is important to understand that this tool is to complement clinical assessment rather than to replace it entirely.
In conclusion, NZRISK-VASC is a simple and accurate multivariable risk prediction model for peri-operative mortality in vascular surgical patients. It requires only 10 easily obtainable covariables in the pre-operative setting and can help with the shared clinical decision making process with patients. The general NZRISK tool
is currently freely available online for clinicians to use at www.nzrisk.com, and the NZRISK-VASC will become available when the website is upgraded in mid-2021.
Funding
This work was supported by a grant from Precision Driven Health (PDH), which provided research funding, work time for L.B. and scientific support.
Conflict of interest
D.C. has received research funding from PDH, Auckland District Health Board Research Trust, the Australian and New Zealand College of Anaesthetists and the Neurological Foundation of New Zealand. The remaining authors declare no conflicts of interest.
Appendix A. Supplementary data
The following are the Supplementary data to this article:
The Abdominal Aortic Aneurysm Statistically Corrected Operative Risk Evaluation (AAA SCORE) for predicting mortality after open and endovascular interventions.
To submit a comment for a journal article, please use the space above and note the following:
We will review submitted comments as soon as possible, striving for within two business days.
This forum is intended for constructive dialogue. Comments that are commercial or promotional in nature, pertain to specific medical cases, are not relevant to the article for which they have been submitted, or are otherwise inappropriate will not be posted.
We require that commenters identify themselves with names and affiliations.
31123-0&id=gr1.jpg){kind=link}
31123-0&id=gr2.jpg){kind=link}