Background: For decades, statisticians and methodologists have severely criticized the undue reliance on strict p-value thresholds (usually 0.05) and the misinterpretation of statistical significance in medical research. Despite legitimate criticism, primary research and systematic reviews continue to rely heavily on statistical significance without taking a nuanced approach when interpreting nonsignificant effect estimates.
Objectives: We aimed to evaluate the language employed by Cochrane authors to emphasize differences in nonsignificant treatment effects they considered relevant. We also sought to measure the magnitude of these differences and determine whether they differed from statistically nonsignificant effect estimates that authors interpreted as similar or not different.
Methods: We screened all Cochrane reviews of interventions published between November 2017 and 2022 for statistically nonsignificant effect estimates that authors presented as meaningful differences. We classified interpretations qualitatively and assessed them quantitatively by calculating two areas under the curve (AUC): one for the confidence interval (CI) exceeding the null, indicating a greater effect of one intervention, and another for the CI exceeding a minimal important difference. We compared the AUC of nonsignificant effect estimates that authors interpreted as meaningful with those interpreted as nonmeaningful.
Results: We screened 2,337 Cochrane reviews, finding 139 cases that emphasized meaningful differences of nonsignificant results. Most commonly, authors used qualifying words (e.g., may/might/could) to express uncertainty (47.5%) and sometimes emphasized lack of significance (19.4%). In 27% of cases, authors made absolute claims about the greater efficacy or harm of one intervention without acknowledging statistical uncertainty, which could be misleading. The AUC analysis of the CIs is ongoing and will be available at the Colloquium.
Conclusions: Nuanced interpretations of statistically nonsignificant results were rare in Cochrane reviews. Authors should consider these subtleties when interpreting nonsignificant effect estimates to ensure that meaningful differences are not overlooked and that conclusions drawn are as accurate and informative as possible. Future research should explore the most effective language for distinguishing and interpreting effect estimates so that knowledge users interpret this information correctly.
Patient, public and/or healthcare consumer involvement: The choice of how to present statistically nonsignificant results can substantially impact the interpretation of results made by decision-makers.