Background: RoB 2, the revised Cochrane tool for assessing risk of bias in randomized controlled trials (RCTs), aims to increase the reliability of assessments.
Objectives: To share our experience of using RoB 2 in a systematic review on remote alcohol and drug misuse therapies.
Methods: An experienced review team conducted 53 RoB 2 assessments of 45 RCTs for two substance use outcomes (several studies reported both outcomes).
Results: We experienced the following challenges: 1. Prespecifying the outcome: Our review included heterogeneous studies evaluating complex interventions with a range of outcomes. With no preferred outcome indicated by review commissioners, to prespecify an outcome we had to familiarise ourselves with the studies prior to quality appraisal, despite this not being considered best practice. Recommendation: Guidance should recognise that not all reviews include a homogeneous study set, with a standardised outcome measure. As an alternative to prespecification of outcomes, reviewers could prespecify ‘decision rules’, for example, to select the most reported outcome. 2. Assessing and reporting multiple outcomes: Studies with multiple eligible outcomes, required time-consuming and largely repetitive assessments. Although some aspects of RoB 2 are specific to each outcome included, others apply to the whole study. Recommendation: Adapt RoB 2 to allow for multiple assessments of a single study. 3. Interpreting statistics and trial methodology: RoB 2 demands sophisticated understanding of methodological aspects such as randomization procedures and handling missing data. Recommendation: More examples and scenarios should be included in the guidance. 4. Difficulty navigating the guidance: The application of RoB 2 to our set of complex studies necessitated frequent referral to the training videos and the full guidance, as we found the crib sheet lacked sufficient detail. Repeated rounds of piloting and development of a review-specific guidance document were also necessary to ensure consistency of application. Recommendation: Consolidate guidance into a single, multilayered resource, making it easier to delve deeper into explanations and examples when needed.
Conclusions: Application of RoB 2 is challenging and time-consuming even for an experienced review team. Broad and/or complex reviews may be particularly susceptible to these challenges. We propose user-friendly ways to revise the RoB 2 tool and guidance documents.
Patient, public and/or healthcare consumer involvement: None.