Background:RoB2, the revised Cochrane tool for assessing risk of bias in randomized controlled trials (RCTs) aims to increase the reliability of assessments.
Objectives:To share our experience of using RoB2 in a systematic review on remote alcohol and drug misuse therapies.
Methods:An experienced review team conducted 53 RoB2 assessments of 45 RCTs for two substance use outcomes (several studies reported both outcomes).
Results:We experienced the following challenges: 1. Pre-specifying the outcome: Our review included heterogenous studies evaluating complex interventions with a range of outcomes. With no preferred outcome indicated by review commissioners, to pre-specify an outcome we had to familiarise ourselves with the studies prior to quality appraisal, despite this not being considered best practice. Recommendation: Guidance should recognise that not all reviews include a homogenous study set, with a standardised outcome measure. As an alternative to pre-specification of outcomes, reviewers could pre-specify ‘decision rules’, for example to select the most reported outcome. 2. Assessing and reporting multiple outcomes: Studies with multiple eligible outcomes, required time-consuming and largely repetitive assessments. Although some aspects of RoB2 are specific to each outcome included, others apply to the whole study. Recommendation: Adapt RoB2 to allow for multiple assessments of a single study. 3. Interpreting statistics and trial methodology: RoB2 demands sophisticated understanding of methodological aspects such as randomization procedures and handling missing data. Recommendation: More examples and scenarios should be included in guidance. 4. Difficulty navigating the guidance: The application of RoB2 to our set of complex studies necessitated frequent referral to the training videos and the full guidance, as we found the crib sheet lacked sufficient detail. Repeated rounds of piloting and development of a review-specific guidance document were also necessary to ensure consistency of application. Recommendation: Consolidate guidance into a single, multi-layered resource, making it easier to delve deeper into explanations and examples when needed.
Conclusions:Application of RoB2 is challenging and time-consuming even for an experienced review team. Broad and/ or complex reviews may be particularly susceptible to these challenges. We propose user-friendly ways to revise the RoB2 tool and guidance documents.
Patient, public and/or healthcare consumer involvement:None