User-testing of a decision support tool for multiple treatment options with NMA data to facilitate guideline development: A qualitative study
2Department of Medicine, Lovisenberg Diaconal Hospital., Norway
3Department of Medicine, University Hospitals of Geneva, Switzerland
4MAGIC Evidence Ecosystem Foundation, Oslo, Norway
Background:
While network meta-analysis (NMA) presents unprecedented opportunities for comparative effectiveness research, data output from NMAs is often overwhelming, even for healthcare professionals and researchers. Little is known on how to best present NMA data to guideline panelists.
Objectives:
To employ and user test the MATCH-IT tool – an interactive Summary of Findings table (iSoF) presenting data from NMA for multiple interventions – in guideline panels to further answer their information needs and facilitate guideline development.
Methods:
We performed user testing sessions with semi-structured interviews of panel members in preparation for or after panel meetings in 4 different guideline panels. We also observed the use of the MATCH-IT tool during panel meetings. Interviews and meetings were transcribed, coded and analyzed using directed content analysis to capture the user experience and use of the tool. The analysis informed iterative developments of the tool.
Results:
User-testing of 15 panel members (3 chairs, 9 clinical experts, 3 patient partners) and 3 observed panel meetings resulted in 4 iterations of the tool including both new features and refinements from the initial prototype. Key improvements included adding color coding as well as new data elements such as 95 % confidence interval. In the final two rounds of development, chairs and participating panelists expressed that the tool contributed significantly to facilitating the meeting. Our analysis suggests that key features of MATCH-IT include: (1) the interactivity that provides the possibility of tailoring the tool to fit the needs of the panel live during the meeting, (2) the tool’s ability to provide an overview of the evidence by presenting the effects of multiple interventions across multiple outcomes, and (3) the color coding of these effects to deal with the information overload.
Conclusions:
This study suggests that iSoFs, in this case the MATCH-IT tool, can facilitate guideline development by making data from NMA in systematic reviews easier to understand and navigate. MATCH-IT has already been published in a Cochrane review and multiple clinical practice guidelines.
Patient, public and/or healthcare consumer involvement:
Study participants included patients, practicing clinicians and methodological experts, contributing to user-friendliness for these consumer populations.