A Delphi Consensus on Patient Reported Outcomes for Registries and Trials Including Patients with Intermittent Claudication: Recommendations and Reporting Standard

reached if (cid:2) 80% of participants agreed that an item was both clinically relevant and practical. Results: Participation rates in two Delphi rounds were 66% (31 participants of 47 invited) and 90% (54 of 60), respectively. Initially, 145 patient reported outcome QIs were documented. Following the two Delphi rounds, 18 quality indicators remained, all of which reached consensus regarding clinical relevance. The VascuQoL questionnaire (VascuQoL-6), currently the most common patient reported outcome measurement (PROM) used within vascular registries, includes a total of six items. Five of these six items also matched with high rated indicators identi ﬁ ed in the Delphi study. Consequently, the panel recommends the use of the VascuQoL-6 survey as a preferred core PROM QI set as well as an optional extension of 12 additional patient reported QIs that were also identi ﬁ ed in this study. Conclusion: The current recommendation based on the Delphi consensus building approach, strengthens the international harmonisation of registry data collection in relation to patient reported outcome quality. Continuous and standardised quality assurance will ensure that registry data may be used for future quality benchmarking studies and, ultimately, positively impact the overall quality of care provided to patients with peripheral arterial occlusive disease.


INTRODUCTION
Peripheral arterial occlusive disease (PAOD) is a widespread health burden worldwide. The number of PAOD related interventions is continuously increasing, 1,2 and PAOD is considered an important marker of risk of subsequent cardiovascular events and death. 3 In 2015, approximately 237 million patients were affected globally. 4 A number of valid guidelines on the treatment of patients with PAOD define the increase of patients' functional status and health related quality of life (HRQoL) as primary treatment goals in patients with intermittent claudication (IC). 5e7 WHO defines quality of life "as an individual's perception of their position in life (.) It is a broad ranging concept affected in a complex way by the persons' physical health, psychological state, level of independence, social relationships, personal beliefs and their relationship to salient features of the environment" (source: https://www.who.int/healthinfo/ survey/whoqol-qualityoflife/en/). An abundance of studies has demonstrated that PAOD is associated with markedly reduced HRQoL 8e10 and can lead to depression and substantial social isolation. 11,12 As demonstrated by Pell et al., 8 the precision with which vascular specialists assess the HRQoL in their patients is "only moderately good". Overall, surgeons estimate their patients' HRQoL lower than patients assess it themselves, and the authors emphasise the need for more accurate outcome measurements in situations when clinical decision making is influenced by the patient's HRQoL status. This discrepancy between surgeons' and patients' HRQoL assessments could also potentially explain why registry data show that approximately 50% of invasive procedures targeting PAOD are undertaken for IC, even though supervised exercise and lifestyle changes are the first line treatment at this PAOD stage, as the reduction of major adverse cardiovascular event rates is the primary aim of intervention. 1,13 Modern evidence based health care of patients with PAOD recommends the creation of patient centred treatment pathways, where patient reported outcomes (PROs) have a key role 8 by providing longitudinal information about limitations and changes in HRQoL status. Many clinical trials also use a variety of patient reported outcome measurements (PROMs) to indicate treatment success and to allow comparison between chronic conditions and health economic analysis. PROMs capture diverse facets of self reported outcomes 14 by using either generic (e.g., short form health 36, SF-36) or disease specific questionnaires (e.g., VascuQoL-6). 15e17 While the former are particularly useful when comparing HRQoL impact across different diseases and therapeutic areas, disease specific instruments are more sensitive in revealing smaller but clinically important changes, because they focus on the specific problems experienced by patients with PAOD. 18 When measuring and comparing quality in health care, three types of quality indicators (QIs) can be employed to capture either the structure, process, or outcome, 19 and can be used to document the overall quality of care, for benchmarking, and in research projects. 20 An important requirement for standardised integration and analysis of PROMs in everyday clinical practice is that the outcome indicators included in the questionnaires are objective, measurable, and comparable, as well as precisely defined in advance. These indicators are intended to measure the quality of the outcome exclusively based on patient self reporting. 24 The availability of multiple varying PROMs for the measurement of HRQoL makes the comparability and validation of results difficult. 22,23 To date, there is neither a widely accepted standard for disease specific PROMs in general 10 nor an existing consensus concerning data collection of PROMs in PAOD registries.
Accordingly, this study aimed to build consensus among international experts in the field of vascular medicine on a set of core indicators for IC regarding PRO data collection in vascular registries. The use of the Delphi method for consensus building is widely accepted and can be found in recent publications of various specialties, including vascular medicine. 21,25e29

METHODS
The study comprised a two stage methodology, in which a comprehensive literature search for the compilation of an indicator index preceded the evaluation of the index by an international expert panel through the Delphi method.
To identify outcome QIs, a detailed literature search (Supplementary Table S1) was conducted including metaanalyses, systematic reviews, and guidelines for the treatment of PAOD. The search was conducted between July and December 2020 and was restricted to online sources available in the English and German languages and included the bibliographic database PubMed/Medline, as well as websites of medical institutions, available guidelines, and databases of vascular medical organisations (Supplementary Table S2). The first (HA) and last (CAB) authors of the manuscript conducted the literature review. In addition, a grey literature search was conducted, to generate data from narrative literature of patient organisations published in non-commercial form (e.g., patient associations of vascular societies or PAOD self help groups). The selected literature was based exclusively on outcome QIs, which can be derived from patient reports. Outcome was defined as patient reported therapy results after a conservative or invasive PAOD treatment of patients suffering from IC. All items identified in the literature were precisely defined and uniformly documented in a structured indicator index of PRO QIs (Supplementary Table S3).
Thereafter, the individual items were presented to an expert panel for approval with additional suggestions by providing a free text option in a preliminary survey. Subsequently the items were included for evaluation and consensus building in a modified Delphi approach. The Delphi method is a structured interactive communication technique that serves to find consensus on a specific topic or question among a group of experts. The goal of the Delphi method is to achieve agreement on specific questions by relying on the expertise of the participants, structured reports of the voting results, as well as discussions among the experts. 28

Expert panel
A wide range of experts was invited to participate, with the goal of including representatives from different countries, institutions, and medical specialties. The expert panel included international vascular specialists, patient representatives, and registry members of the VASCUNET committee of the European Society for Vascular Surgery (ESVS), 30 the International Consortium of Vascular Registries (ICVR), and the Medical Device Epidemiology Network (MDEpiNet). During the more cumbersome preliminary and first rounds, the survey leaned primarily on vascular specialists with membership in the largest international collaborations on vascular registries. Participation in the evaluation process was online and anonymous. Open source software was used to create the questionnaire (LimeSurvey GmbH, Hamburg, Germany, www.limesurvey.org). Invitations with a link to the survey as well as reminders were sent electronically before and during each round by email.
This study made use of a modified two stage Delphi technique to accomplish consensus. Items were rated on a five point Likert scale (strongly disagree/disagree/neutral/ agree/strongly agree) in the first round. Participants were asked to rate each item independently in relation to the parameters "clinical relevance" and "practicability". If at least 80% of the participants rated an item as "strongly agree" or "agree", the item reached consensus and was included for evaluation in the second round. In addition, items that almost formed consensus ( 70% of agreement) were also included for re-evaluation in the modified second Delphi round which included an extended expert group and a rating on a four point forced choice Likert scale (strongly disagree/disagree/agree/strongly agree). Following both rounds, a structured anonymised report of the results using graphical diagrams was compiled and distributed to the experts electronically (Supplementary Table S4). Subsequently, after both the preliminary and first rounds, the results were meticulously discussed at two online video meetings, resulting in the decision to expand the panel in the second round to introduce greater diversity. Voting results for each item were explained with respect to clinical relevance and practicability. An online discussion of the results was held to review and approve the final recommendations for data collection of IC PROMs in vascular registries and trials. The discussion included the research team as well as all Delphi panel members with membership in international vascular registries.

RESULTS
Thirty of the 46 (65%) invited experts participated in the preliminary survey. The panel represented 29 different countries and was expanded by 13 experts including two senior vascular nurses who represented the patients' perspective in the second round. Of 47 invited experts, 31 (66%) completed the first round, and 53 of 60 (90%) participated in the second and final round (Fig. 1).
The preliminary survey results affirmed all 118 items compiled during the literature search and included 27 new items suggested by the panel and added to the evaluation process as potentially useful patient reported outcome QIs. Subsequently, the items were added to the structured indicator index, which finally consisted of eight domains, 29 subdomains, and a total of 145 items (Supplementary Table S3).
In the first Delphi round, 22 items reached at least 80% agreement, the previously defined threshold of consensus building. Following the first round, a discussion among the participants was held, with the decision to re-integrate another 17 items that failed to reach the limit of agreement, but almost formed consensus ( 70% to < 80% agreement) in the second round for re-evaluation to arrive at a high consensus among experts without dismissing variables prematurely in the first round given the high number of variables. As a result, the second and final round consisted of 39 items. Eighteen items reached consensus regarding clinical relevance (Table 1) with over half (56%) belonging to the physical domain. Twelve items exceeded the 80% threshold of consensus building regarding clinical relevance. No consensus was achieved for indicators in the psychological domain. By including 17 items for re-evaluation, the item "perception of IC related QoL" formed consensual agreement (81% agreement for clinical relevance). The Delphi study not only confirmed five of the six VascuQol-6 indicators, but also attested high clinical relevance with ratings of > 90% of agreement for the individual item. The only VascuQoL-6 item that did not reach consensus in the Delphi study was "concerns about PAOD", with 68% and PAOD ¼ peripheral arterial occlusive disease; QoL ¼ Quality of life. * High rated items (> 90% agreement). y Failed to reach consensus for practicability. z These items regarding "presence of." are already covered in VascuQoL-6 (when asking for "intensity of IC" or "degree of walking impairment").
50% of agreement for clinical relevance and practicability, respectively. The panel reviewed the results in an online discussion and recommended the use of a core set of six indicators (included in the VascuQoL-6 survey) but to also consider extending it by a set of 12 additional indicators (the identified consensus items in this study) for national registry and trial data collection of patient reported outcome QIs (Table 2).

DISCUSSION
The current Delphi consensus study strived to harmonise PAOD patient reported outcome assessments across different vascular registries and countries and included a total of 60 panel experts from 29 countries and different medical specialties as well as patient representatives. The goal of this study was to build consensus on PROs to be collected in registries and trials that include patients with IC. Of 145 items initially included in the two Delphi rounds and group discussions, six core items from the VascuQoL-6 questionnaire along with 12 additional, optional items were ultimately recommended by the panel.
Clinical practice guidelines on the treatment of patients with IC aim to guide clinicians through the challenge of identifying optimal treatment pathways for their patients. In PAOD, concepts like lifestyle limiting claudication and the degree of impairment in daily living activities are regarded as important determinants for subsequent treatment decisions. 5,6,31 At the same time, there is a paucity of concurrent recommendations for appropriate PRO tools that enable measurement of such constructs in patients with IC. To the present authors' knowledge, only a now outdated PAOD guideline document covered this issue and recommended using the SF-36 or the Walking Impairment Questionnaire (WIQ) to measure HRQoL in PAOD. 32 Of these, the SF-36 includes no less than 36 questions, and this relative abundance of items limits the usefulness of this particular questionnaire in busy clinical routine scenarios. The WIQ was developed so that it can be completed by study investigators in approximately five minutes, and, while it has remained widely used in PAOD patients, this questionnaire primarily captures IC disease symptom severity (walking distance, walking speed and stair climbing) 32 but fails to include items that measure other important HRQoL constructs; such as pain, discomfort, social and emotional consequences of IC, all of which remain central themes in IC disease. More recent research in this area has reached heterogeneous conclusions on which PROMs to recommend, 15,33 and a fairly recent systematic review also pointed out that the validation process for many of the currently available PAOD specific PROMs has been suboptimal, and such shortcomings should be taken into account when interpreting their results. 17 In a recent comprehensive review of the literature, Raja et al. clearly emphasised the distinct heterogeneity of PROMs currently used in both registries and trials. 34 Overall, it appears challenging to collect PROM data in everyday clinical practice, for both clinical and research purposes. In the prospective GermanVasc cohort study (NCT03098290), which enrolled 5 608 patients with invasive revascularisation for symptomatic PAOD between May 2018 and December 2021, only 73% of patients enrolled during the index treatment completed the baseline PRO questionnaires, and only 21% agreed to complete the questionnaire at the 12 month follow up. One of the most frequently documented reasons was that the 25 item questionnaire was too complicated and cumbersome. Hence, while PRO surveys implemented in clinical routine need to satisfy certain psychometric and scaling standards, they should also allow easy completion by the patient, and furthermore need to be easy for health professionals to administer, score, and analyse. In an attempt to improve the assessment of PROs after PAOD interventions, a pragmatic instrument, the VascuQoL-6, originally derived from the VascuQoL 25 items questionnaire, was developed and empirically tested for content validity, construct validity, and test retest reliability. 33,35,36 The VascuQoL-6 was developed using modern development principles including item response theory, and furthermore multiple validation studies have confirmed its validity. 33,36e38 To date, the questionnaire is available in numerous languages and there are established minimally important difference and substantial clinical benefit thresholds both following either supervised exercise therapy or lower limb revascularisation. Consequently, this facilitates clinical interpretation in both clinical trials and routine clinical care. 35,39 This current Delphi study confirmed its widespread use in 30% of clinical registries currently participating in the VASCUNET and ICVR, and five of the six items included in the VascuQoL-6 survey matched items that reached consensus during the Delphi process. The only VascuQoL-6 item that did not reach full consensus was "concerns about PAOD". Interestingly, all proposed items included in the psychological domain of the Delphi process failed to reach consensus and were subsequently excluded. This was not only due to low practicability ratings but also for clinical relevance, with the highest overall rating given to "anxiety caused by PAOD" (74% agreement), an item closely related to "concerns about PAOD" available in the VascuQoL-6 survey. In contrast, the items "presence of intermittent claudication in daily living" as well as "smoking" reached highest ratings in both categories and were included among the 12 optional additional items also recommended by the expert panel.
Considering the focus on walking impairment in recent guidelines, it is interesting to note that all respective items on walking distance received rather moderate ratings for both clinical relevance and practicability. This may point to an incongruity between guideline recommendations and daily practice. The question arises how to communicate an achievable improvement of maximum walking distance to a patient with claudication after 200 m. Are 50 m enough? The rather modest improvements in numerous clinical trials on innovative medical devices and therapy illustrate that this aspect deserves more reflection by the community. From a clinical standpoint, patient reported walking distance estimations also remain notoriously inaccurate, 40 and this is arguably one reason why the expert panel downgraded the clinical relevance for the proposed items aiming to capture IC walking distance.
This study has limitations. Firstly, a Delphi consensus study, although commonly accepted and broadly used in the past, can only achieve consensus among the panel experts included in the process. The present study aimed to be as inclusive as possible and involved experts from all medical specialties as well as patient representatives. However, it cannot be ruled out that another panel would decide differently in the future. The lack of patient representatives and panel representativeness from disciplines other than medicine most probably led to a bias that cannot be rectified retrospectively. During all steps of the process, it had to be accepted that the enormous amount of time necessary to participate as a panel expert along with further requirements (e.g., language barriers) are obstacles that are not easy to overcome. To compensate for the shortcomings in the distribution of panellists and to include experts who reached out after the preliminary round, it was accepted that the composition of the Delphi panel as well as the parameters (e.g., consensus thresholds) differed between the rounds. This, however, may have introduced another bias. The inclusion of only English and German language publications introduces another bias. By including the largest international collaborations in the vascular registry field, attempts were made to cover many global regions and societies. However, as the virtual meetings were on a voluntary basis, not all panel experts participated in these meetings. The final recommendation deserves another critical discussion. Against the background of the 18 consensus items, the panel decided to recommend the VascuQoL-6 as a set of core indicators along with 12 additional indicators. Although it probably introduced another bias, this decision was made due to the fact that the VascuQoL-6 is a psychometric construct that has been repeatedly validated and has been shown to perform well across different countries, regions, and languages. It seems reasonable to underscore that the current study was not conducted to develop PROMs but to reach a consensus on what to use in trials and registries on PAOD. Moreover, it appears challenging and time consuming to implement this consensus on PROMs into established registry structures. The present authors will monitor whether this recommendation leads to an adaptation of registries in the future. Finally, although patient representatives were involved in this process, further strengthening of patient involvement as well as empowerment are important aims of future projects within the PAOD field. The vascular community needs to find ways to include patients not only in consensus processes but also in the decision making in everyday clinical practice.

Conclusions
In the current study, a rigorous modified Delphi method was applied to find consensus among an international panel of experts with respect to core patient reported outcome quality indicators to be collected in registries and used in trials on the treatment of patients with IC. At this stage, a broad integration of the VascuQoL-6 core PROM set into vascular registries and trials including patients with IC was supported by this Delphi consensus study and would represent an important step in the development of patient centred management pathways for patients with PAOD.

FUNDING
This work was supported by the German Federal Joint Committee (grant number 01VSF16008 and 01VSF18035 to CAB).