Background: There is increasing concern that a significant proportion of randomised trials included in Cochrane reviews may not be trustworthy. Applying a Trustworthiness Screening Tool (TST) has already had a clinically important effect on several reviews published by the Cochrane Pregnancy and Childbirth Group.
Objectives: We wanted to assess the impact of removing untrustworthy randomised trials from already published Cochrane reviews on a defined clinical area (nutritional interventions during antenatal and postnatal period).
Methods: We applied the Cochrane Pregnancy and Childbirth TST to 375 randomised trials included in 18 Cochrane reviews. The TST has five domains (is the research governance trustworthy; are the baseline characteristics trustworthy; is the study feasible; are the results plausible; and is all relevant information available?). When additional information was needed, authors were contacted using a standard template. At least two attempts were made to contact the authors. At the end of the evaluation process, each study was classified as i) included (YES to all domains); ii) excluded (retracted study); or iii) awaiting classification (any NO to the TST questions).
Results: 95/375 studies (25%) were removed, affecting 14/18 (78%) reviews. 13/18 reviews (72%) showed a difference in the Summary of Findings tables (direction and size of effects and/or GRADE ratings). 6/18 Cochrane reviews (33%) were judged to require updating because of important differences in their conclusions, implication for practice, and/or implication for research.
Conclusions: Formal assessment of trustworthiness and inclusion only of studies that satisfy prespecified criteria for trustworthiness affects conclusions in a relatively large number of Cochrane reviews, with potentially important clinical implications for practice and research. The lack of consensus regarding the best tool(s) for assessing trustworthiness cannot be an excuse for ignoring this issue in future Cochrane reviews.
Patient, public and/or healthcare consumer involvement: N/A. Statement on relevance and importance to patients: Our work found that removal of studies that did not fulfil prespecified trustworthiness criteria affected the conclusions of a third of the Cochrane reviews that we assessed.