Background: The availability of unsound or scientifically invalid work is a concern associated with preprint articles. The popularity of preprints in prevention research has increased due to the COVID-19 pandemic.
Objectives: The objectives of this study were to assess the consistency of results and conclusions of preprints in prevention research compared to peer-reviewed articles and to explore the perception of key stakeholders regarding the growing number of published preprints.
Methods: The study employed a mixed-methods approach that involved the development of a Python-based Web crawler to search MedRxiv for prevention preprints from January 1 to September 30, 2020. We ran an update search one year later to identify which preprints were published. We dually screened all results for prevention articles and developed a scheme to classify changes in effect sizes and conclusions. We also conducted and analyzed 19 qualitative online interviews with stakeholders who have expertise in prevention.
Results: The WebCrawler retrieved a total of 2,238 preprints, of which 594 were prevention research studies and 329 were epidemiological studies. Nearly half of these studies (48.9%) were published in peer-reviewed journals within one year, with a median time from upload to publication of 5.3 months (range: -0.1 to 11.3 months). Among published preprints, 16.8% of articles showed a change in effect size, which was a major change in 4.4% articles (i.e., a change greater than 25% of the original effect size). The conclusions changed in 43% of the studies, mostly in terms of style or wording (39%). Stakeholders perceive that preprints have advantages over peer-reviewed articles, including free access, fast submission, transparency, and feedback from colleagues. However, they also feel that preprints are not compatible with current quality standards and feedback culture and may have a detrimental effect on publication and career. Conclusion: A few prevention research articles experience changes in effect estimates and conclusions when they are published. Although these changes are small in number, they affect one in every 23 articles. Although preprints may seem like an attractive publishing option, there is still skepticism about their quality and potential consequences. We therefore warrant caution of using preprints of prevention research in decision-making.