What we do in the shadows: Interviews with methodologists on synthesis methods when meta-analysis is not possible for all studies

Session Type
Synthesis of evidence without meta-analysis
Cumpston MS1, Brennan SE2, Ryan R3, McKenzie JE1
1Methods in Evidence Synthesis Unit, School of Public Health & Preventive Medicine, Monash University, Australia
2Cochrane Australia, School of Public Health & Preventive Medicine, Monash University, Australia
3Centre for Health Communication & Participation, La Trobe University, Australia

Background: Systematic review (SR) authors commonly encounter situations where meta-analysis is not possible for some or all studies (e.g. when studies report only the direction of the effect). In a survey of published SRs of public health and health systems interventions, 60% used a method other than meta-analysis, but the methods were rarely specified (5%) and were limited to those described in Cochrane guidance. Greater understanding is needed of the range of other synthesis methods considered by authors, and factors that influence their decisions.
Methods: We interviewed 12 experienced SR authors, editors and methodologists. We presented four scenarios in which it was not possible to combine results from all studies in a meta-analysis. Scenarios varied in the number and size of included studies; completeness of the reported results (i.e. fully reported [effect estimate and measure of precision] and partially reported [e.g. only a statement of statistical significance]); and other factors (e.g. risk of bias). Participants described the methods they considered to summarise, synthesise and present the results; whether they would combine all available studies in a single synthesis; and how they would draw an overall conclusion. Transcripts were coded to develop a framework of synthesis methods and factors that influenced decisions.
Results: Factors that influenced participants’ decisions included their views on the purpose of synthesis, existing beliefs about study results and synthesis methods, trust in the available data, and the decision-making needs of end users. Participants differed in whether they would or would not meta-analyse some of the available results. Commonly participants considered explicit or implicit vote counting, alongside or instead of meta-analysis, when it was not possible to meta-analyse all studies. Their description and interpretation of the results varied.
Conclusions: A diverse range of methodological approaches and influencing factors were identified. While synthesis methods other than meta-analysis and their rationale are often not described in SRs, they could be described explicitly in protocols and SRs to enhance transparency. Patient, public and/or consumer involvement: Patients were not involved in this research. These findings are relevant in promoting transparency of methods, which will inform future guidance to strengthen SRs.