Impact of Expectation Bias on Effect Estimates between Non-inferiority and Superiority Randomized Clinical Trials: Retrospective Cohort Study

Session Type
Jia Y1, Jiang Y1, Yang Z1, Robinson K2, Tang J1
1Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China
2Department of Medicine, School of Medicine, Johns Hopkins University, USA

Background: Prior expectations of effect estimates may affect randomized clinical trials (RCTs), leading to expectation bias. When assessing the same clinical question, the design choice between superiority RCTs (S-RCTs) and non-inferiority RCTs (NI-RCTs) may be based on researchers’ expectations.
Objectives: To estimate the impact of expectation bias by comparing the effect estimates between S-RCTs and NI-RCTs.
Methods: This was a retrospective cohort study of RCTs. We screened Cochrane reviews for meta-analyses that assessed the efficacy of clinical interventions, produced statistically significant results, and included at least one NI-RCT and one S-RCT. In each meta-analysis, S-RCTs were included in the exposure group, while NI-RCTs were in the control group. S-RCTs should share the same primary outcome with NI-RCTs. The main outcome was the ratio of risk ratio (RRR), hazard ratio (RHR), or odds ratio (ROR, with OR transformed from standardized mean difference) between S-RCTs and NI-RCTs. RRRs, RHRs, and RORs were pooled to form a single estimate by random-effects meta-analyses. Potential confounders and effect modifiers were assessed in a linear mixed-effect regression model.
Results: We identified 56 meta-analyses from 9,018 Cochrane reviews. A total of 405 RCTs were included, consisting of 74 NI-RCTs and 331 S-RCTs. Among meta-analyses using OR (transformed from SMD), RR, and HR as the effect measure, S-RCTs produced an effect estimate 1.63 (1.21-2.19, I2=84.5%), 1.16 (1.07-1.25, I2=16.2%), and 1.08 (0.94-1.23, I2=3.9%) times greater than NI-RCTs, respectively. On average, S-RCTs produced an effect estimate 1.31 (95%CI: 1.17-1.48, I2=73.3%) times greater than NI-RCTs. When adjusting for the covariates, S-RCTs produced an effect estimate 1.25 (1.05, 1.47) times greater than NI-RCTs. Publication bias was assessed as an effect modifier: among meta-analyses where publication bias was detected, undetected, or unclear, S-RCTs produced an effect estimate 2.27 (1.72-2.99), 1.25 (1.05-1.47), and 1.10 (0.95-1.29) times greater than NI-RCTs, respectively.
Conclusions: S-RCT generally produced an effect size 31% higher than NI-RCT, implying that expectation bias may significantly distort the effect estimates. Such bias may not be adequately controlled by the current procedures to detect or minimize bias.
Patient, public and/or healthcare consumer involvement: Patient, public and/or healthcare consumers were not involved.