Integration of Machine Learning in a living systematic review of baseline risks of Venous Thromboembolism complications in hospitalized patients with COVID-19

Session Type
Oral presentation
Evidence synthesis innovations and technology
Lotfi T1, Nowak A2, Charied R1, Solo K1, Santesso N1, Schünemann H1, Nieuwlaat R1
1Michael G. DeGroote Cochrane Canada Centre, Health Research Methods, Evidence & Impact, McMaster University, Canada
2Evidence Prime, Poland

Background: Living Systematic Reviews (LSRs) of prognostic studies rely on screening many observational studies that are not clearly labeled.
Objectives: To assess the performance of a Machine Learning (ML) algorithm for screening in a LSR for Venous Thrombo-Embolism (VTE)-related outcomes baseline risks in COVID-19 patients.
Methods: As part of a guideline development project for the American Society of Hematology (ASH) on the use of anticoagulation for thromboprophylaxis in COVID-19 patients, the team conducted a LSR to establish and maintain relevance of the baseline risk for VTE-related outcomes. The search was conducted in September 2020 (baseline search) and updated monthly until July 2021. At baseline, the search identified 69,560 citations. The team trained a ML classifier algorithm using the manual screening of the baseline search to partially automate the screening process for the next search iterations. The algorithm ranked captured citations based on likelihood for inclusion, with those appearing on top as most likely to be included. The algorithm was integrated in a new software “Laser AI” which will allow the team to screen prioritized citations in future updates. In this study, we screened manually, in duplicate and independently, a sample of 5% (n=3478) of captured citations at two iterations of the living search and that were not allocated the highest likelihood for inclusion. In parallel, we retrospectively applied model trained on the most recently screened documents to the initial set of search results to explore how the data distribution changed over time.
Results: We manually screened in duplicate and independently 3478 citations, out of which 377 were included at title/abstract level and full text screening is ongoing. We will compare these results with the algorithm's classification to measure the algorithm’s performance (precision, recall, accuracy and specificity). We will also assess whether there were identified studies eligible for inclusion that were not selected for screening, and if they affected the pooled baseline risk for VTE. Results will be ready by March 31, 2023.
Conclusions: Efficiency and relevance of LSRs for prognostic studies can be enhanced when combining manual with ML-directed screening.
Patient, public and/or healthcare consumer involvement: N/A