Risk of bias assessment in systematic review with network meta-analysis: a meta-research study with AMSTAR-2, ROBIS and the ROB-NMA tool
2Department of Health Product Policy and Standards, World Health Organization, Geneva,, Switzerland
3Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Cochrane Hypertension Review Group, the Therapeutics Initiative, University of British Columbia, Canada
Background. A MeaSurement Tool to Assess systematic Reviews (AMSTAR 2) and the Risk of Bias in Systematic Reviews (ROBIS) are currently the most commonly used tool for systematic reviews of health care interventions. A new tool to assess the risk of bias in network meta-analysis (RoB NMA) is under development: this pilot tool is recommended to be used in conjunction with ROBIS, although network meta-analysis (NMA) authors might use it also with AMSTAR 2. Objectives. To assess methodological quality and risk of bias by using AMSTAR-2, ROBIS and the pilot RoB-NMA tool. Methods. A cross-sectional study. We searched for all NMAs indexed in Pubmed in January 2023. We rated NMAs methodological quality and risk of bias using AMSTAR 2 and ROBIS. Selection, data extraction and quality assessments were performed independently by two researchers. Data were analyzed descriptively. The RoB NMA tool will be available for the piloting phase by April 2023; therefore, analyses will be completed by June 2023. Stratified analysis and correlations will be conducted to explore factors that might affect the quality of conduct. Study protocol is stored at the following link: https://osf.io/pa6dz/. Results. A total of 110 NMAs with a median of 21 studies (IQR 12-43) and 3695 participants (IQR 1525-12404) were included. Preliminary results on 10% NMAs (n=12) showed that 83.3% (n=10) were judged by critically low quality by AMSTAR 2, with only one of moderate quality and one of high quality. Among critical domains, the three least adequately addressed were item 2 (58,3% NMAs), item 4 (66.7% NMAs), and item 15 (66.7% NMAs) (Figure 1). ROBIS judged 83.3% (n=10) of NMAs as high risk of bias, one as unclear risk and one as low risk. Domain 2 was at highest risk in 50% NMAs, whereas domain 3 was judged as low risk of bias in 75% NMAs that resulted in the better conducted. Table 1 reported AMSTAR 2 and ROBIS assessments. Conclusions. Preliminary results showed that the majority of recently published NMAs had an overall low methodological quality. Patient involvement. Poor control of potential source of bias in NMAs might affect results’ trustworthiness and translation into clinical practice, which could be beneficial for patients.