Risk of bias of a set of primary studies across systematic reviews showed highly heterogeneous assessments: A methodological analysis within an overview

Date & Time
Tuesday, September 5, 2023, 12:30 PM - 2:00 PM
Location Name
Session Type
Overviews of reviews and scoping reviews
Bracchiglione J1, Rodríguez-Grijalva G2, Requeijo C2, Santero M2, Salazar J2, Bonfill X1, Urrútia G1
1Institut d’Investigació Biomèdica Sant Pau (IIB SANT PAU), Cochrane Iberoamerican Centre, CIBERESP, Barcelona, Spain
2Institut d’Investigació Biomèdica Sant Pau (IIB SANT PAU), Cochrane Iberoamerican Centre, Barcelona, Spain

Background: Risk of bias (RoB) assessment of randomised clinical trials (RCTs) is a critical step in the conduction of systematic reviews (SRs) because it directly impacts interpretation of findings and certainty of evidence. The Cochrane RoB tool (version 1) has been widely used historically, but some criticism remains in terms of possible disagreements between different assessors.
Objectives: To describe the agreement of RoB assessment of a set of RCTs made by authors of different SRs using the RoB1 tool.
Methods: Within an overview of SRs regarding the effects of anticancer drugs versus supportive care for hepatobiliary cancers, we extracted all RoB assessments made by SR authors that used the RoB1. We describe the percentage of severe disagreements (simultaneous low and high-risk assessments across different SRs) and non-severe disagreements (simultaneous low-unclear or unclear-high assessments) across all domains.
Results: We identified 18 SRs including a total of 22 unique RCTs, of which 16 were evaluated with the RoB1 tool by at least two different reviews. We found severe disagreements in 50% of the assessments of the “selective reporting” domain, 37.5% of the “blinding of outcome assessors” domain, and 25% of the “blinding of participants and personnel” domain. Non-severe disagreements were observed in all domains, especially in the domains related to selection bias.
Conclusions: RoB1 tool showed mostly discrepancies in three specific domains. Overview authors should extract and consider the RoB assessments of RCTs made from every SR included within an overview to explore these discrepancies. Patient, public, and/or healthcare consumer involvement: None.