Background: Living systematic reviews (LSRs) of prognostic studies rely on screening many observational studies that are not clearly labeled.
Objectives: To assess the performance of a machine learning (ML) algorithm for screening in an LSR for venous thromboembolism (VTE)–related outcomes and baseline risks in patients with COVID-19.
Methods: As part of a guideline development project for the American Society of Hematology (ASH) on the use of anticoagulation for thromboprophylaxis in patients with COVID-19, the team conducted an LSR to establish and maintain relevance of the baseline risk for VTE-related outcomes. The search was conducted in September 2020 (baseline search) and updated monthly until July 2021. At baseline, the search identified 69,560 citations. The team trained an ML classifier algorithm using the manual screening of the baseline search to partially automate the screening process for the next search iterations. The algorithm ranked captured citations based on likelihood for inclusion, with those appearing on top as most likely to be included. The algorithm was integrated in a new software, “Laser AI,” which will allow the team to screen prioritized citations in future updates. In this study, we screened manually, in duplicate and independently, a sample of 5% (n=3,478) of captured citations at two iterations of the living search and that were not allocated the highest likelihood for inclusion. In parallel, we retrospectively applied a model trained on the most recently screened documents to the initial set of search results to explore how the data distribution changed over time.
Results: We manually screened in duplicate and independently 3,478 citations, of which 377 were included at title/abstract level and full text screening is ongoing. We will compare these results with the algorithm’s classification to measure the algorithm’s performance (precision, recall, accuracy, and specificity). We will also assess whether there were identified studies eligible for inclusion that were not selected for screening and if they affected the pooled baseline risk for VTE. Results will be ready by March 31, 2023.
Conclusions: The efficiency and relevance of LSRs for prognostic studies can be enhanced when combining manual with ML-directed screening.
Patient, public and/or healthcare consumer involvement: N/A.