Background: COVID-19 has driven the need for timely evidence to inform decision-making. Rapid reviews allow for the timely production of evidence for stakeholders. Therefore, systematic review teams may consider mechanisms to streamline the methods despite the lack of evidence about the impact of these strategies on the conclusions of the review. To our knowledge, there is no relevant empirical evidence regarding differences in meta-analytic effect estimates when using data abstracted by only one reviewer versus the consensus.
Objectives: To evaluate the impact of data extraction by a single reviewer versus two reviewers on effect estimates from meta-analyses.
Methods: We conducted a meta-epidemiological study using data abstracted from randomized trials included in the living network meta-analysis (LNMA) that informs the recommendations of the World Health Organization COVID-19 therapeutics living guideline. We included comparisons and outcomes used in the guidelines. As part of the LNMA, for each eligible trial, pairs of reviewers, following training and calibration exercises, extracted data independently using a standardized, pilot-tested data extraction form. Subsequently, a data cleaning team checked all the extracted information for accuracy. For each comparison and outcome, we will conduct three pairwise meta-analyses using a frequentist approach and random effects model: 1) using the reference standard (i.e., the final data after cleaning), 2) using data abstracted by one reviewer, and 3) using the consensus data of the reviewers. To determine whether there are important differences between the results of the three analyses, we will use the threshold above which the guideline panel considered the effect important (see Figure 1). Finally, we will calculate the proportion of outcomes for which the results are importantly different when using the data abstracted by one reviewer versus two reviewers and versus the reference standard.
Results: This study is ongoing, and the results will be presented at the Colloquium as available.
Conclusions: This study is ongoing, and the results will be presented at the Colloquium as available.
Patient, public and/or healthcare consumer involvement: Patients and or the public were not involved in this study, either in planning or in the study’s design.