Background: Prior expectations of effect estimates may affect randomized clinical trials (RCTs), leading to expectation bias. When assessing the same clinical question, the design choice between superiority RCTs (S-RCTs) and noninferiority RCTs (NI-RCTs) may be based on researchers’ expectations.
Objectives: To estimate the impact of expectation bias by comparing the effect estimates between S-RCTs and NI-RCTs.
Methods: This was a retrospective cohort study of RCTs. We screened Cochrane reviews for meta-analyses that assessed the efficacy of clinical interventions, produced statistically significant results, and included at least one NI-RCT and one S-RCT. In each meta-analysis, S-RCTs were included in the exposure group, while NI-RCTs were in the control group. S-RCTs should share the same primary outcome with NI-RCTs. The main outcome was the ratio of risk ratio (RRR), hazard ratio (RHR), or odds ratio (ROR, with OR transformed from standardized mean difference) between S-RCTs and NI-RCTs. RRRs, RHRs, and RORs were pooled to form a single estimate by random-effects meta-analyses. Potential confounders and effect modifiers were assessed in a linear mixed-effect regression model.
Results: We identified 56 meta-analyses from 9,018 Cochrane reviews. A total of 405 RCTs were included, consisting of 74 NI-RCTs and 331 S-RCTs. Among meta-analyses using OR (transformed from SMD), RR, and HR as the effect measure, S-RCTs produced an effect estimate 1.63 (1.21-2.19, I2=84.5%), 1.16 (1.07-1.25, I2=16.2%), and 1.08 (0.94-1.23, I2=3.9%) times greater than NI-RCTs, respectively. On average, S-RCTs produced an effect estimate 1.31 (95% CI: 1.17-1.48, I2=73.3%) times greater than NI-RCTs. When adjusting for the covariates, S-RCTs produced an effect estimate 1.25 (1.05, 1.47) times greater than NI-RCTs. Publication bias was assessed as an effect modifier: among meta-analyses where publication bias was detected, undetected, or unclear, S-RCTs produced an effect estimate 2.27 (1.72-2.99), 1.25 (1.05-1.47), and 1.10 (0.95-1.29) times greater than NI-RCTs, respectively.
Conclusions: S-RCT generally produced an effect size 31% higher than NI-RCT, implying that expectation bias may significantly distort the effect estimates. Such bias may not be adequately controlled by the current procedures to detect or minimize bias.
Patient, public and/or healthcare consumer involvement: Patient, public and/or healthcare consumers were not involved.