Background: Risk of bias (RoB) assessment of randomised clinical trials (RCTs) is a critical step in the conduction of systematic reviews (SRs) because it directly impacts interpretation of findings and certainty of evidence. The Cochrane RoB tool (version 1) has been widely used historically, but some criticism remains in terms of possible disagreements between different assessors.
Objectives: To describe the agreement of RoB assessment of a set of RCTs made by authors of different SRs using the RoB1 tool.
Methods: Within an overview of SRs regarding the effects of anticancer drugs versus supportive care for hepatobiliary cancers, we extracted all RoB assessments made by SR authors that used the RoB1. We describe the percentage of severe disagreements (simultaneous low and high-risk assessments across different SRs) and non-severe disagreements (simultaneous low-unclear or unclear-high assessments) across all domains.
Results: We identified 18 SRs including a total of 22 unique RCTs, of which 16 were evaluated with the RoB1 tool by at least two different reviews. We found severe disagreements in 50% of the assessments of the “selective reporting” domain, 37.5% of the “blinding of outcome assessors” domain, and 25% of the “blinding of participants and personnel” domain. Non-severe disagreements were observed in all domains, especially in the domains related to selection bias.
Conclusions: RoB1 tool showed mostly discrepancies in three specific domains. Overview authors should extract and consider the RoB assessments of RCTs made from every SR included within an overview to explore these discrepancies. Patient, public, and/or healthcare consumer involvement: None.