International Vascunet Validation of the Swedvasc Registry

Vascunet is an international collaboration set up to compare practice between countries for common vascular interventions. Data analysis, using national and regional registry data, has been undertaken for infrarenal abdominal aortic aneurysm (AAA) repair, carotid endarterectomy, infrainguinal bypass, and popliteal aneurysm intervention, and these comparisons of practice have been previously published.^{,  ,  ,  ,} 

These analyses have revealed differences in practices and outcomes between countries but have also highlighted that the extent and quality of data collection vary considerably. Therefore, a persistent concern surrounding this type of registry data comparison, from both within the Vascunet committee and from peer reviewers, is that validation of the data is insufficient.

International audit also poses a particular problem with regard to how representative the data of the population within the country from which they are derived are. It is quite possible for large numbers of patients to be submitted but for this to only be a small proportion of the total number of cases performed. It is therefore important to examine not only the quality of data with respect to accurate recording of clinical information but also to examine the completeness of the data. The latter includes how well individual data fields are completed but also whether all patients treated are included. Two validation questions are therefore important when assessing registry data: (1) Are all the cases recorded in the registry, and (2) Are the recorded data accurate? External validation in the current study answers the first question and internal validation answers the second.

In response to these concerns Vascunet has now embarked on a system of validation of national data, and is committed to improving registry data in order to allow meaningful international comparison of data, thereby enabling improvement of outcomes from vascular intervention. Hungary was the first country to be validated and the results of this validation have been published. The processes used in this initial validation formed the basis for developing a template for use in subsequent validations. The second country to be validated was Sweden and this report details the findings of that validation.

The validation of the Swedvasc registry was performed by two independent validators (T.L., M.V.), who visited five of the 31 hospitals in Swedvasc during two visits to Sweden. The five hospitals cover 36% (342/948) of the carotid artery procedures performed for stroke prevention and reported to the Swedvasc during 2012 and 28% (357/1269) of the AAA procedures. The hospitals were picked randomly but in such a way that allowed the five hospitals to be visited on the two site visits. The hospitals included in the first visit in January 2014 were Uppsala University Hospital (Hospital 1) (T.L. and M.V.), Vasterås Hospital (Hospital 2) (T.L.), and Södersjukhuset in Stockholm (Hospital 3) (M.V.). Visits to Skåne University Hospital in Malmö (Hospital 4) (M.V.) and Helsingborg Hospital (Hospital 5) (T.L.) took place in April 2014. Both external validation and internal validation (described below) were performed in each hospital: 95% confidence intervals were calculated for external validity of all data, and internal validity in each hospital as well as total data.

In Swedvasc, procedures are registered in seven different modules: carotid artery procedures (for stroke prevention), infrarenal abdominal aortic aneurysm disease, other aortic disease (juxta/suprarenal, thoracic, and thoraco-abdominal), infrainguinal occlusive disease (including popliteal artery aneurysms), miscellaneous other arterial procedures, miscellaneous other venous procedures and re-operations. Shortly beforehand the validators had informed the hospitals that procedures registered to the modules for AAA and carotid stenosis performed during the calendar year 2012 would be retrospectively reviewed, so that hospital administrative data would be obtained before the site visit for external validation to be performed.

Vascunet, a subcommittee of the European Society of Vascular Surgery, aims to increase knowledge and understanding of vascular disease, and to promote excellence in vascular surgery by means of international vascular audit. The group comprises vascular representatives from 10 countries and has published extensively on international vascular outcome data using registry data.^{,  ,  ,  ,}  The group believes that vascular registries are of the utmost importance in the quality control of a vascular surgical unit. They serve as a source of data in population based studies and allow monitoring of outcomes in a large number of patients who are treated in daily practice.

It is clear that different countries are at very different stages in recording and publishing outcomes from vascular intervention; nevertheless, in recent years the number of vascular registries has increased and there is a steady trend towards publication of results. In the UK this has recently moved to a level of individual surgical outcome data.

Medical registry data need to be accurate in order to be able to draw meaningful conclusions. The data contained in a registry, and publications and quality reports based on these data, are of little value if the registry data are unreliable. International data comparison poses a particular challenge in ensuring consistency of data between countries. Studies of data quality have suggested various classifications of error, including interpretation errors, documentation errors and coding errors. Despite this there is no internationally agreed method of data validation for registries, although Arts et al. have attempted to define a framework of procedures for data quality assurance in medical registries, involving procedures, in central coordination of data and in local data collection. The Vascunet group have therefore developed a process for international validation of vascular outcome data, based on previous experience in a pilot study.

In this study the Swedvasc registry was validated and both external and internal validation were done. External validation answers the question: Are all the cases recorded in the registry? The Swedvasc registry was found to have very good correlation with local hospital clinical and administrative data for external validation. With only a very small number of exceptions all the patients who were operated on in the validated hospitals during the year examined (2012) were also registered in the Swedvasc registry. Most of the discrepancies that were found between the two data systems were due to either registration in Swedvasc under different modules, or to patients being coded as a different procedure in local administrative data. In some cases it was felt by the local teams that there was no satisfactory code available. There were a handful of cases found in Swedvasc that were not in the hospital administrative data due to errors in the hospital recording systems. There is a long tradition of recording outcomes by a vascular registry in Sweden as Swedvasc was founded almost 30 years ago. The Swedvasc organisation performs their own external validation annually by comparing the number in the Swedvasc with the numbers in the administrative registry. In this external validation process the external validity of Swedvasc for carotid endarterectomy was 96% in 2012, which is comparable with the results here when considering that all cases that did not match individually were checked. This public external validation process performed by Swedvasc is likely to drive the motivation to register patients in Swedvasc.

The second question is: Are the recorded data accurate? This is at least as important, if not more so than external validation. It was found that data accuracy was good in internal validation. The pre- and post-operative data were correct in all hospitals, giving more that 95% internal data validity in all five hospitals. In one of the validated hospitals the pre-operative risk factors (diabetes and smoking) were systematically missing. For the validator this information was easily found from the case registries, so it seemed to be a local cultural matter in registering the pre-operative risk factors. The other variables in internal validation in this particular hospital were correct.

The independence of the validators is a key issue in order to achieve unbiased results in the validation process; hence, validators from different countries were chosen to visit Sweden. In the current project five of 20 hospitals that perform vascular surgery in Sweden and participate in Swedvasc were validated. The crucial question is how well these five hospitals represent the whole of Swedvasc. The validators were able to influence the centres selected (Malmö, Stockholm), although two of them were selected for geographical reasons, that is the hospitals were near the other hospitals that were chosen for validation (Vasterås and Helsingborg). Only one centre was selected by the representatives from Swedvasc (Uppsala). Owing to this selection policy, the validators were confident that the selection of the five hospitals was unbiased and would provide a representative view of vascular surgery in Sweden. The hospitals did know before the visit that aortic and carotid surgery in 2012 would be validated. With the exception of Uppsala, which was the first centre to be visited, the time between informing the centres and the visit was less than 4 weeks. There was therefore a theoretical possibility of influencing the registry data before the visit although the validators found no evidence that this had occurred. Furthermore, it would have been an enormous workload to check the data validity for internal validation as the cases for validation were picked totally randomly. Furthermore, the fact that all the studied pre-operative risk factors were missing in one of the selected hospitals suggests that there was no alteration of registry data prior to the visit. The validators were given very detailed access to relevant data in order to be able to perform the validation and found no evidence of local centres failing to provide the required information. This type of access is essential in order for the validation process to be robust, and validation of other countries in the future will need to include this requirement. The willingness of the Swedish vascular community to allow external scrutiny of their data was considered by the validators to be exceptional. One of the two validators had language skills that allowed her to review the case histories independently but the other needed a local person or representative from Swedvasc to help with the patient records. This could be seen as another limitation of the method (in addition to the fact that the hospitals were aware of the validation year), but is inevitable in international validation. Many medical terms are common to both languages, however, and both the validators felt that any attempt to hide data would have been clear.

Almost all of the few differences between the case histories and Swedvasc data were in the pre-operative data, such as indication for the surgery, aneurysm diameter, degree of carotid stenosis and risk factors (diabetes and smoking). Post-operative complications were recorded carefully and in all cases where discrepancy was found between Swedvasc and case records, the definition of a complication was not fulfilled (e.g. a few cases with significant creatinine elevation without dialysis were recorded as renal complications although the definition of a renal complication in Swedvasc is one that requires dialysis). This indicates that post-operative complications were not underestimated in the registry. The few cases that were missing from Swedvasc were all emergency cases, and all of these patients survived; therefore, the validation did not demonstrate a failure to report cases with an adverse outcome. Mortality data, both at 30 days and at 12 months was correct in all the studied cases.

The most significant problem in the current validation process was obtaining the correct lists of hospital records from official hospital records, and the structure of Swedvasc, which has several different modules. The vast majority of the cases that seemed to be missing from Swedvasc were just recorded to a different module, and a significant number of module “mistakes” were found. This of course does not affect the overall internal and external validity but does have an influence on the reliability of the reports extracted from Swedvasc. As a result, the numbers of different type of operations based on modules in Swedvasc could be seen as unreliable. There was a learning curve to this both with regard to the validating team and also the local team providing the data and the validation process became more efficient as the project progressed.

The fact that most of the cases “missing” from the Swedvasc registry were found in a different module indicates that the registry may be too complex, and for this reason reducing the number of modules within Swedvasc is advisable. Although this issue could in theory have led to cases being missed in the international comparisons that have been performed by Vascunet, in practice the different modules have been interrogated in order to identify all relevant cases. In the future the increasing use of a variety of endovascular techniques for infrarenal, juxtrenal, and suprarenal aneurysms will make it even more difficult to define a true “infrarenal” group and therefore combining these cases into one module seems sensible both within Swedvasc and within other national systems.

This validation process has not examined the relevance or subsequent use of the datafields that have been collected within the Swedvasc database. The perceived impression is that some of the datafields, in particular those that were poorly completed, are of little relevance to the subsequent evaluation of patient care. Further international work is required to define a robust dataset that excludes data fields of little relevance.

The following recommendations for the Swedvasc registry were made in the full report of the validation process provided to the Swedish vascular teams.

• 1.
  Procedure codes for identification of procedures in local hospital data and in Swedvasc should be identical.
• 2.
  Agreement should be reached (or followed more accurately locally if agreement already exists) at a national level regarding which procedures map to which codes.
• 3.
  It should be clear in Swedvasc how to record procedures done for different indications (e.g. an aortobifemoral bypass graft for aneurysmal disease or occlusive disease)
• 4.
  Consideration should be given to revising the different modules within Swedvasc as there appears to be a lack of consistency with regard to which cases are entered into the different modules. Overall, reducing the number of modules is recommended to avoid confusion about which module to use for individual cases.
• 5.
  A small number of data fields are responsible for the majority of the variations found between Swedvasc and local data. However, the validators were able to find this information from the case records. Consideration should be given as to the benefit of collecting the data fields that are prone to missing data. If these are not used for analysis they could be removed, if they are used then centres should try and complete these fields.

The Vascunet team believe that this validation process has demonstrated that international validation of registry data is possible and this template could be used for other similar validation exercises in the future. It is an important step in improving registry data in order to allow meaningful international comparison and thereby improve outcomes from vascular intervention.

None.

The European Society for Vascular Surgery (ESVS) supported this validation financially. The ESVS received a grant from the Swedvasc Registry to enable the realisation of the project. The Swedvasc registry had no influence on how the validation was performed, nor any access to the results prior to publication of the report.

The authors are indebted to all those clinicians in Sweden who assisted them in performing a complete, independent and robust validation of clinical data.