Background: Systematic reviews could inform about the impact of different health systems’ arrangements on processes of care and patients’ health outcomes. As for any type of evidence, users should make judgments about how much confidence to place in their findings. AMSTAR (Assessment Methodological quality of SysTemAtic Reviews) is a tool for assessing the methodological quality of reviews. An updated version—AMSTAR-2—was developed in response to some limitations of the original tool. ROBIS (Risk Of Bias In Systematic reviews) was recently developed to assess risk of bias (RoB) in reviews. They have not been compared in assessing health systems reviews.
Objectives: To compare two tools to assess how much confidence to place in the findings of health systems reviews.
Methods: In preparing four overviews assessing different health systems arrangements, we previously identified 124 reviews. We assessed a random sample of them using AMSTAR-2 and ROBIS. We converted the AMSTAR-2 overall confidence ratings and the ROBIS overall RoB ratings into numerical values. We calculated a mean score across the raters for each review for each tool and used them to calculate Spearman’s rank correlation coefficient (rs). Additionally, we compared the concordance in the overall categorical ratings on confidence and RoB.
Results: Twenty-eight reviews were assessed by at least two raters with each tool. AMSTAR-2’s overall confidence ratings were strongly correlated with ROBIS’s overall ratings. The rs between both tools was 0.71 (p=0.000023). Regarding the concordance between overall categorical ratings, the 11 reviews assessed with moderate/high confidence by AMSTAR-2 were also assessed with low RoB by ROBIS. However, 8 out of 14 reviews assessed with low/critically low confidence by AMSTAR-2 were assessed with low RoB by ROBIS.
Conclusions: The tools showed correlation in the overall numerical scores, but there was not clear concordance between their overall categorical assessments, especially in the group of reviews with more limitations. More work is needed to disentangle the relationship between the tools currently available to assess reviews. Patient and healthcare consumer relevance: Making judgments about how much confidence to place in reviews’ findings is key to informing decisions with a potential high impact on patients’ health.