Applying a stepwise appraisal tool to determine the usefulness of systematic reviews in health technology assessments
2Region Skåne, HTA Syd, Lund, Sweden
3Region Västra Götaland, HTA-centrum, Sahlgrenska University Hospital; Department of Obstetrics and Gynecology, Instute of Clinical Science, University of Gothenburg; Department of Obstetrics snd Gynecology, Sahlgrenska University Hospital, Sweden
4Region Västra Götaland, HTA-centrum, Sahlgrenska University Hospital; Department of Health and Rehabilitation, Institute of Neuroscience and Physiology, Sahlgrenska Academy, University of Gothenburg, Sweden
5Region Västra Götaland, HTA-centrum, Sahlgrenska University Hospital; Department of Obstetrics and Gynecology, Instute of Clinical Science, University of Gothenburg, Department of Obstetrics snd Gynecology, Sahlgrenska University Hospital, Sweden
6Medical library, Sahlgrenska University Hospital, Gothenburg, Sweden
7Region Västra Götaland, HTA-centrum, Sahlgrenska University Hospital; Department of Pharmacology, Sahlgrenska Academy, University of Gothenburg, Sweden
Background To facilitate quick appraisal of the usefulness of systematic reviews (SRs), the Swedish Agency for Health Technology Assessment (HTA) and Assessment of Social Services (SBU) has developed SNABBSTAR. This tool, based on A MeaSurement Tool to Assess systematic Reviews (AMSTAR), implies a stepwise assessment of the methodological/reporting quality. The six steps concern the following: 1. definition of population, intervention, control, and outcomes (PICO), literature search; 2. inclusion/exclusion according to PICO, listing of included studies; 3. risk of bias assessments; 4. evidence synthesis/meta-analyses; and 5. certainty of evidence considerations; 6. documentation of excluded studies, conflicts of interest, and SR protocol. Assessment is stopped if an SR does not meet the quality criteria of a step. SRs not fulfilling the first step are considered not useful, and those reaching the fifth step and beyond can be used in HTAs with limited or no complementary work. Objectives To apply SNABBSTAR to evaluate the usefulness of SRs systematically identified in an HTA for evidence-based decision-making. Methods We included all SRs published in 2018-2020 identified in the literature search in four databases, and fulfilling the PICO, of an HTA (Sahlgrenska University Hospital, Gothenburg, Sweden, 2021:123) investigating efficacy and safety outcomes for clopidogrel versus ticagrelor in patients subjected to dual antiplatelet therapy in acute coronary syndrome. Four assessors independently assessed all SRs using SNABBSTAR, and diverging assessments were resolved in consensus discussions. Results A total of 13 SRs were assessed (China, n=7; Italy, n=2; USA, n=2; Poland, n=1; Kazakstan, n=1), 10 of which were described to be reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). The SRs included 0‒16 randomised controlled trials (RCTs) and 0‒8 non-RCTs. In all, 12 SRs were assessed to fail already at step 1 in SNABBSTAR; the literature search could not be reproduced with the information provided. One SR passed step 4 but not step 5 because the certainty of evidence was not reflected in the conclusions. Conclusion This analysis suggests that SRs often fail to meet fundamental quality criteria required to be useful in evidence-based decision-making; the reporting of the literature search process deserves further attention in SR publications. A stepwise appraisal, starting with the definition of the PICO/literature search, may facilitate quick assessments of the usefulness of SRs.