Background: Cochrane reviews are regularly updated to reflect the most up-to-date evidence on health interventions. In 2021, Cochrane introduced a new policy to manage potentially problematic studies that are eligible for inclusion in a Cochrane review. This includes the management of studies with retractions, expressions of concern, and concerns about the trustworthiness of the data. It remains unclear how well such a policy is implemented for published Cochrane review updates.
Objectives: To provide an overview of the frequency of problematic studies included in Cochrane reviews and whether they have been excluded in Cochrane review updates according to Cochrane’s new policy.
Methods: We searched the Cochrane Database of Systematic Reviews to identify intervention review updates published in 2022 and their previous version. We compared these two versions to identify studies that were included in the previous version and excluded in the 2022 update. We also identified retracted studies that were included in the previous version by searching the Retraction Watch Database and assessed whether these were excluded from the updated version. Additionally, we recorded whether and how Cochrane’s new policy was implemented in the updates.
Results: We identified 123 pairs of Cochrane reviews from 37 Cochrane groups. The previous version of these reviews was published between 1996 and 2021, with a median of six included studies and 607 participants. There were 16 (13%) reviews with at least one previously included study excluded in the 2022 updates, including 4 (3%) excluding at least one (median 4, range 1-54) potentially problematic study. The previous version of two (2%) reviews included at least one retracted study, and one (1%) review did not exclude them in the 2022 update. Sixteen (13%) updates clearly specified the methods to assess and/or handle problematic studies, including 4 (3%) using a trustworthiness assessment tool and 12 (10%) mentioning the handling of retracted studies.
Conclusions: To facilitate the production of trustworthy evidence, it is imperative to identify and appropriately manage problematic studies eligible for Cochrane reviews. This study reflects the suboptimal implementation of Cochrane’s new research integrity policy in review updates, suggesting further guidance is needed.