How much confidence to put on health systems reviews?: a comparative assessment using AMSTAR-2 and ROBIS
2School of Medicine, Pontificia Universidad Catolica de Chile, Chile
3UC Evidence Center, School of Medicine, Pontificia Universidad Catolica de Chile, Chile
4Faculty of Medical Sciences, School of Obstetrics and Childcare, Universidad de Santiago de Chile, Chile
5McMaster Health Forum, McMaster University, Canada
Systematic reviews could inform about the impact of different health systems arrangements on processes of care and patients’ health outcomes.As for any type of evidence, users should make judgments about how much confidence to place on their findings. AMSTAR (Assessment Methodological quality of SysTemAtic Reviews) is a tool for assessing the methodological quality of reviews. An updated version – AMSTAR-2 – was developed in response to some limitations of the original tool. ROBIS (Risk Of Bias In Systematic reviews) was recently developed to assess risk of bias (RoB) in reviews. They have not been compared in assessing health systems reviews.
Objectives: To compare 2 tools for assessing how much confidence to place on the findings of health systems reviews.
Methods: In preparing 4 overviews assessing different health systems arrangements, we previously identified 124 reviews. We assessed a random sample of them using AMSTAR-2 and ROBIS. We converted the AMSTAR-2 overall confidence ratings and the ROBIS overall RoB ratings into numerical values. We calculated a mean score across the raters for each review for each tool and use them to calculate Spearman’s rank correlation coefficient (rs). Additionally, we compared the concordance in the overall categorical ratings on confidence and RoB.
Results: Twenty-eight reviews were assessed by at least two raters with each tool. AMSTAR-2 overall confidence ratings were strongly correlated with ROBIS overall ratings. The rs between both tools was 0.71 (p=0.000023). Regarding the concordance between overall categorical ratings, the 11 reviews assessed with moderate/high confidence by AMSTAR-2 were also assessed with low RoB by ROBIS. However, 8 out of 14 reviews assessed with low/critically-low confidence by AMSTAR-2 were assessed with low RoB by ROBIS.
Conclusions: The tools shown correlation in the overall numerical scores but there was not clear concordance between their overall categorical assessments especially in the group of reviews with more limitations. More work is needed to disentangle the relationship between the tools currently available to assess reviews. Patient and healthcare consumer relevance: Making judgments about how much confidence to place on reviews’ findings is key for informing decisions with a potential high impact on patients’ health.