Evaluation and development of a novel interactive Summary of Findings table for network meta-analysis. A qualitative user-testing study with clinicians
Per Olav Løvsletten, Lovisenberg Diaconal Hospital
2Faculty of Medicine, University of Oslo, Oslo, Norway; MAGIC Evidence Ecosystem Foundation, Oslo, Norway
3Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium; Belgian Centre for Evidence Based Medicine (Cebam), Leuven, Belgium; Cochrane Belgium, Leuven, Belgium
4Academic Centre for General Practice, Department of Public Health and Primary Care, Faculty of Medicine, KU Leuven, Leuven, Belgium
5Harvard Medical School, Boston, United States of America
6Department of Anesthesia, McMaster University, Hamilton, Canada
7MAGIC Evidence Ecosystem Foundation; Department of Medicine,Geneva University Hospital; Department of Medicine, Faculty of Medicine,University of Geneva; Department of Health Research Methods, Evidence,Evidence&Impact, McMasterUniversity, Canada
8Department of Medicine, Lovisenberg Diakonal Hospital, Oslo, Norway; MAGIC Evidence Ecosystem Foundation, Oslo, Norway
9Department of Medicine, Lovisenberg Diakonal Hospital, Oslo,Norway; Faculty of Medicine, University of Oslo, Oslo Norway; MAGIC Evidence Ecosystem Foundation, Oslo, Norway
10MAGIC Evidence Ecosystem Foundation, Oslo, Norway
Background: Clinicians often need to evaluate comparative effectiveness of relevant treatment options for their patients. Network meta-analyses (NMAs) provide an important resource for clinicians to guide decision-making; however, these analyses are complex, and interpretation may prove challenging for users.
Objectives: To evaluate and develop a novel interactive Summary of Findings table (the MATCH-IT tool) for network meta-analysis results, to support clinicians in interpreting and applying comparative effectiveness evidence into decision-making.
Methods: We performed qualitative user-testing with a convenience sample of practicing physicians working in hospitals and general practice in Norway, Belgium, and Canada. User testing entailed a brief introduction with a clinical scenario, which was a think aloud session with participant-tool-interaction followed by a semi-structured interview. The tests were recorded, transcribed, and analysed using directed content analysis. The results informed the iterative development process of the MATCH-IT tool.
Results: Five rounds of user-testing, with 26 participants in total, have resulted in four iterations with updates in the tutorial, layout, navigation functionality, and interphase, as well as the practical-issues-section. We have also added an FAQ section. Clinicians who participated in user-testing had been in clinical practice from 0 to 30 years (median 6, IQR 3-11). Most had little or no prior experience with interpretation of NMA results and sparse knowledge about GRADE methodology. Most clinicians perceived MATCH-IT as easy to interpret and navigate and appreciated its ability to provide overview of the evidence, although some of the interactive features proved to be difficult to discover intuitively. They perceived pictograms and inclusion of practical-issues-information as potentially useful features when interacting with patients. Categorization of results with colour coding to indicate both certainty of evidence and relative effectiveness of interventions was appreciated, and filtering functionality allowed participants to efficiently browse and focus in on the interventions believed to be most relevant for a given clinical scenario.
Conclusions: MATCH-IT shows promise in supporting decision-making for clinicians when presented with results from NMAs. The tool has already been published within Cochrane systematic reviews and clinical practice guidelines. Patient, public, and/or healthcare consumer involvement: Clinicians were the subject for user testing, ensuring user friendliness for this key consumer group.