Can the Cochrane RCT classifier be used to speed up screening for a qualitative evidence synthesis? A retrospective evaluation
Christine Hillestad Hestevik
Patricia Sofia Jacobsen Jardim
Background: Study selection is time and resource intensive. Using machine learning to identify studies that do not meet the inclusion criteria can decrease time use. The Cochrane RCT classifier has been trained to recognize and classify RCTs into two categories, likely or unlikely to be an RCT. The quick identification of references, for example RCTs, which are unlikely to meet the inclusion criteria of a qualitative evidence synthesis (QES) can speed up the study selection process.
Objectives: The first objective of this study is to evaluate if we can use the Cochrane RCT classifier to identify non-qualitative studies that can be excluded or deprioritized during screening for a QES. The second objective is to evaluate whether there is a cutoff point in the likely to be an RCT category beyond which no qualitative studies are lost.
Methods: Using a pre-identified number of QES, we will: - Extract the included studies - Run them through the Cochrane RCT Classifier in EPPI Reviewer - Map studies that are predicted “likely to be an RCT” - Create a combined map of the predictions for all included studies across the included QES - Explore the studies that are classified as having a high likelihood of being an RCT to determine how they are different to the other included studies that were not given this classification
Results: This retrospective evaluation is ongoing (completion June 2023). The results will be the map over qualitative studies that were classified as likely to be an RCT as well as the overview of how these studies are different from the studies that did not receive this classification.
Conclusions: Based on the mapping of the studies we will conclude if there is a threshold which reviewers can apply to automatically exclude or deprioritise studies based on their high likelihood of being an RCT. This would enable reviewers to screen faster and prioritize certain search hits for screening above others.
Patient, public and/or healthcare consumer involvement: Speeding up the review process allows evidence to become available faster to consumers. Qualitative research brings consumer perspectives forward.