Introduction to the WISEST (Which Systematic Evidence Synthesis is best) Project: Developing an automated clinical decision-support algorithm to choose amongst systematic review(s) on the same topic
2Ottawa Hospital Research Institute, canada
3Ottawa Hospital Research Institute, Canada
4Canadian Association of Radiologists, canada
5Institute for Health Services and Health Systems Research, Center for Health Services Research, Brandenburg Medical School, Germany
6Department of Electrical and Computer Engineering, Toronto Metropolitan University (TMU), canada
7Nova Scotia Health, Canada
8The University of British Columbia, Canada
9Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, Canada
10McMasterPlus, McMaster University, Canada
11CrowdCARE, University of Melbourne, Australia
12Public Health Agency of Canada, Canada
13National Collaborating Centre for Methods and Tools & Health Evidence, Canada
14Università di Bologna, Italy
15McGill University, canada
16University of Toronto, canada
17Beijing Yealth Technology, China
18National Advisory Committee on Immunization (NACI), canada
19Robert Koch Institute, Germany
20Robert Koch Institute, germany
21Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, canada
22Therapeutics Initiative, The University of British Columbia (UBC), canada
23Therapeutics Initiative, The University of British Columbia (UBC), Canada
24Cochrane Hypertension Group, The University of British Columbia, Canada
25British Columbia Ministry of Health, Canada
26World Health Organisation, Switzerland
27Canadian Agency for Drugs and Technology in Health, Canada
28The Federal Joint Committee (G-BA), Germany
29Agency for Healthcare Research and Quality (AHRQ), USA
30Cochrane Consumer Network, United Kingdom
31SPOR Evidence Alliance, Canada
32SFU, Canada
33BC Support Unit, Canada
Background: Knowledge users (KUs) need the highest-quality studies to make decisions about which interventions and policies should be used. The most reliable way to answer questions is with a systematic review (SR). AMSTAR and ROBIS tools are used to assess quality/bias in SRs. However, no automated tool currently exists to assess the quality/biases in SRs (Figure 1).
Objectives: (1) Develop a set of features to extract from SRs related to quality/bias; (2) develop a labelled dataset of 10,000 SRs; and (3) test, train and validate models and compare their accuracy.
Methods: A structure was proposed by the international steering group, and features were collected using a methods review and a survey. We used the results of a previous study comparing the tools which mapped an item’s concept across the three tools. A core team reviewed them and determined their feasibility in being automatically identifiable and predictable by our model (rather than through a manual process). A flowchart of activities is proposed (Figure 2). Five organisations with databases of preappraised SRs will supply the SRs. Duplicates will be removed and topics will be mapped. If topic fields, settings or conditions are missing, we will fill these gaps with a search for SRs. We will use ML Random Forests and DL Neural Network classification models such as Facebook’s StarSpace and Fasttext. We will use a ‘supervised’ learning model which learns by making predictions given examples of data.
Results: The proposed structure is found in Figure 1. The 21 items mapped from AMSTAR 1 and 2 and ROBIS will be used as the quality features. Other features chosen include PROGRESS-Plus items related to sex, gender and equity, 20 methods features (e.g., number of databases searched, PICO, and meta-analysis model used) and 10 results features (e.g., effect estimates and adverse events). SRs were collected and cleaned and the other features were extracted in duplicate. Testing will begin in March 2024.
Conclusions: An artificial intelligence tool that critically appraises SRs would dramatically reduce the financial and human resources currently needed to appraise SRs and update SR databases (e.g., McMasterPlus, HealthEvidence and SysVac).
Patient, public and/or healthcare consumer involvement: None.