Comparison of effect estimates between preprints and peer-reviewed publications: a meta-epidemiological study of COVID-19 trials
2Université Paris Cité and Université Sorbonne Paris Nord, Inserm, INRAE, Center for Research in Epidemiology and Statistics (CRESS), F-75004 Paris; Centre d’Epidémiologie Clinique, AP-HP, Hôpital Hôtel Dieu, F-75004, Paris; Cochrane France, Paris, France
3Université Paris Cité and Université Sorbonne Paris Nord, Inserm, INRAE, Center for Research in Epidemiology and Statistics (CRESS), F-75004 Paris; Cochrane France, Paris, France
Background: Preprints have emerged as a major source of research communication during the COVID-19 pandemic. However, questions were raised concerning the reliability of their results.
Objectives: To evaluate whether effect estimates differ between preprint and peer-reviewed journal randomised controlled trials (RCTs).
Methods: Data were derived from the COVID-NMA (covid-nma.com) initiative, a living systematic review of RCTs evaluating preventive interventions, treatments and vaccines for COVID-19. Meta-analyses with at least one preprint and one peer-reviewed journal article evaluating pharmacological treatments vs. standard of care/placebo were included up to July 20, 2022. Predefined COVID-NMA ‘critical outcomes’ at 28 days were considered. A meta-epidemiological analysis estimated the difference in effect estimates [expressed as the ratio of odds ratio (ROR)] between preprint and peer-reviewed journal RCTs. An ROR of <1 indicated that preprint trials yielded larger effect estimates.
Results: Thirty-seven meta-analyses (114 RCTs—44 preprints, 70 peer-reviewed journal articles) were selected. Twenty-four meta-analyses assessed hospitalized patients (81 RCTs), and 13 assessed outpatients (33 RCTs). The median number of RCTs per meta-analysis was 2 (IQR, 2-4; maximum, 11). The median sample size of RCTs was 199 (IQR, 99-478), 68% were prospectively registered, 67% received industry or mixed funding, 79% were multicentric trials, and 75% had some concerns of overall risk of bias. Overall, there was no significant difference in effect estimates between preprint and peer-reviewed journal trials (ROR, 0.88; 95% CI, 0.71-1.09; I2 = 17.8%; τ2= 0.06) (Figure 1). Post-hoc sensitivity analyses of meta-analyses with only two trials (ROR, 0.86; 95% CI, 0.51-1.45; I2 = 22.2%; τ2= 0.19) and at least three trials (ROR, 0.98; 95% CI, 0.84-1.14; I2 = 0.0%; τ2= 0.00) found consistent results.
Conclusions: No important difference in the treatment effect between preprints and peer-reviewed publications was found, particularly in meta-analyses with at least three trials. Systematic reviewers and meta-analysts should assess preprint inclusion individually, accounting for risk of bias and completeness of reporting.
Patient, public and/or healthcare consumer involvement: Peer review is a lengthy process that, during a pandemic, can inadvertently cost lives. Because of the demand for faster access to scientific knowledge, preprints offer a solution, particularly to patients.