Introduction to the WISEST (Which Systematic Evidence Synthesis is best) Project: Developing an automated clinical decision-support algorithm to choose amongst systematic review(s) on the same topic

Session Type
Oral presentation
Evidence synthesis innovations and technology
Lunny C1, Veroniki AA1, Shea B2, Hutton B3, Hamel C4, Pieper D5, Bagheri E6, Reid E7, Zhang JH8, Watt J9, Lavis J10, Downie L11, Tunis M12, Dobbins M13, Ferri N14, Kanjii S2, Whitelaw S15, Strauss S16, Chi Y17, Stevens A18, Pilic A19, Harder T20, Pham B21, Dormuth C22, Bassett K23, Baxter D12, Wright J24, McFarlane J25, Waddell L12, Moja L26, Mittmann N27, Lorenz R28, Iyer S29, Minogue V30, Zarin W31, Gerrish S32, Bryan S33, Tricco AC9
1Knowledge Translation Program, St Michael's Hospital, Unity Health Toronto, Canada
2Ottawa Hospital Research Institute, canada
3Ottawa Hospital Research Institute, Canada
4Canadian Association of Radiologists, canada
5Institute for Health Services and Health Systems Research, Center for Health Services Research, Brandenburg Medical School, Germany
6Department of Electrical and Computer Engineering, Toronto Metropolitan University (TMU), canada
7Nova Scotia Health, Canada
8The University of British Columbia, Canada
9Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, Canada
10McMasterPlus, McMaster University, Canada
11CrowdCARE, University of Melbourne, Australia
12Public Health Agency of Canada, Canada
13National Collaboratin g Centre for Methods and Tools & Health Evidence, Canada
14Università di Bologna, Italy
15McGill University, canada
16University of Toronto, canada
17Beijing Yealth Technology, China
18National Advisory Committee on Immunization (NACI), canada
19Robert Koch Institute, Germany
20Robert Koch Institute, germany
21Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, canada
22Therapeutics Initiative, The University of British Columbia (UBC), canada
23Therapeutics Initiative, The University of British Columbia (UBC), Canada
24Cochrane Hypertension Group, The University of British Columbia, Canada
25British Columbia Ministry of Health, Canada
26World Health Organisation, Switzerland
27Canadian Agency for Drugs and Technology in Health, Canada
28The Federal Joint Committee (G-BA), Germany
29Agency for Healthcare Research and Quality (AHRQ), USA
30Cochrane Consumer Network, United Kingdom
31SPOR Evidence Alliance, Canada
32SFU, Canada
33BC Support Unit, Canada

Background: Knowledge users (KUs) need the highest quality studies to make decisions about which interventions and policies should be used. The most reliable way to answer questions is with a systematic review (SR). AMSTAR and ROBIS tools are used to assess quality/bias in SRs. However, no automated tool currently exists to assess the quality/biases in SRs (Fig 1).
Objectives: (1) develop a set of features to extract from SRs related to quality/bias; (2) develop a labelled dataset of 10,000 SRs; (3) Test, train and validate models and compare their accuracy.
Methods: A structure was proposed by the international steering group, and features were collected using a methods review, and a survey. We used the results of a previous study comparing the tools which mapped an item’s concept across the 3 tools. A core team reviewed them and determined their feasibility in being automatically identifiable and predictable by our model (rather than through a manual process). A flowchart of activities is proposed (Figure 2). Five organisations with databases of pre-appraised SRs will supply the SRs. Duplicates will be removed and topics will be mapped. If topic fields, settings or conditions are missing, we will fill these gaps with a search for SRs. We will use ML Random Forests and DL Neural Network classification models such as Facebooks’ StarSpace and Fasttext. We will use a “supervised” learning model which learns by making predictions given examples of data.
Results: The proposed structure is found in Figure 1. The 21 items mapped from the AMSTAR 1, 2 and ROBIS will be used as the quality features. Other features chosen: PROGRESS-Plus items related to sex, gender and equity, 20 methods features (e.g. number of databases searched, PICO, meta-analysis model used) and 10 results features (eg. effect estimates, adverse events). SRs were collected, cleaned and the other features extracted in duplicate. Testing will begin in March 2024.
Conclusions: An AI tool that does the critical appraisal of SRs would dramatically reduce the financial and human resources currently needed to appraise SRs and update SR databases (e.g. McMasterPlus, HealthEvidence, SysVac).
Patient, public and/or healthcare consumer involvement: None

Fig 1 and 2 2023 Colloquium.PNG