Towards More Scientific Meta-Analyses

Session Type
Oral presentation
Statistical methods
Zhang L1, Konstantinidis M2, Bind M3, Rubin D4
1New York University, USA
2University of Toronto, Canada
3Harvard Medical School, USA
4Harvard University, Tsinghua University, Temple University Fox School of Business, USA, China

Background: Meta-analysis typically estimates a quantity that differs from the implicitly intended estimand; typically, standard approaches estimate the average effect of a treatment for a population of imperfect studies, rather than the true scientific effect that would be measured in a population of hypothetical perfect studies. We advocate for an alternative approach, called response-surface meta-analysis, which models the relationship between study design quality and effect size in order to estimate the effect in the hypothetical ideal study.
Objectives: We demonstrate how we can extend meta-regression to perform response-surface meta-analysis, as well as how resulting estimates can differ from estimates obtained by traditional methods.
Methods: We perform response-surface meta-analysis by performing a meta-regression with design quality as a covariate and reporting the predicted mean and confidence interval at the ideal design quality. We apply this approach to a simulation study with a known true effect, as well as an empirical example published in the Cochrane Library. Using a univariate measure of design quality in the synthetic example and risk of bias as a proxy for quality in the empirical example, we compare estimates from response-surface meta-analysis to those of traditional approaches.
Results: In the simulation study, response-surface meta-analysis usually covers the true effect within one standard error. In contrast, we obtain overly biased results from traditional fixed-effects and random effects meta-analyses. In the empirical study, although the original analysis favors the intervention, response-surface meta-analysis does not provide strong evidence for the surperiority of the intervention.
Conclusions: Response-surface meta-analysis reframes meta analysis as an endeavor to estimate the true scientific effect that would be measured under a perfect study, rather than to summarize the existing population of imperfect studies. More work is needed to standardize the response-surface approach, such as determining a rating system for design quality. We hope this work will help to encourage clarity on the causal estimand of interest in meta-analysis.
Patient, public and/or healthcare consumer involvement: N/A