Background: Meta-analysis typically estimates a quantity that differs from the implicitly intended estimand; typically, standard approaches estimate the average effect of a treatment for a population of imperfect studies, rather than the true scientific effect that would be measured in a population of hypothetical perfect studies. We advocate for an alternative approach, called response-surface meta-analysis, which models the relationship between study design quality and effect size in order to estimate the effect in the hypothetical ideal study.
Objectives: We demonstrate how we can extend meta-regression to perform response-surface meta-analysis, as well as how resulting estimates can differ from estimates obtained by traditional methods.
Methods: We perform a response-surface meta-analysis by performing a meta-regression with design quality as a covariate and reporting the predicted mean and confidence interval at the ideal design quality. We apply this approach to a simulation study with a known true effect, as well as an empirical example published in the Cochrane Library. Using a univariate measure of design quality in the synthetic example and risk of bias as a proxy for quality in the empirical example, we compare estimates from response-surface meta-analysis with those of traditional approaches.
Results: In the simulation study, response-surface meta-analysis usually covers the true effect within one standard error. In contrast, we obtain overly biased results from traditional fixed-effects and random effects meta-analyses. In the empirical study, although the original analysis favors the intervention, response-surface meta-analysis does not provide strong evidence for the superiority of the intervention.
Conclusions: Response-surface meta-analysis reframes meta-analysis as an endeavor to estimate the true scientific effect that would be measured under a perfect study, rather than to summarize the existing population of imperfect studies. More work is needed to standardize the response-surface approach, such as determining a rating system for design quality. We hope this work will help to encourage clarity on the causal estimand of interest in meta-analysis. Patient, public, and/or healthcare consumer involvement: N/A.