Background: Systematic review (SR) authors commonly encounter situations in which meta-analysis is not possible for some or all studies (e.g., when studies report only the direction of the effect). In a survey of published SRs of public health and health systems interventions, 60% used a method other than meta-analysis, but the methods were rarely specified (5%) and were limited to those described in Cochrane guidance. Greater understanding is needed of the range of other synthesis methods considered by authors and factors that influence their decisions.
Methods: We interviewed 12 experienced SR authors, editors and methodologists. We presented four scenarios in which it was not possible to combine results from all studies in a meta-analysis. Scenarios varied in the number and size of included studies; completeness of the reported results (i.e., fully reported [effect estimate and measure of precision] and partially reported [e.g., only a statement of statistical significance]); and other factors (e.g., risk of bias). Participants described the methods they considered to summarise, synthesise and present the results; whether they would combine all available studies in a single synthesis; and how they would draw an overall conclusion. Transcripts were coded to develop a framework of synthesis methods and factors that influenced decisions.
Results: Factors that influenced participants’ decisions included their views on the purpose of synthesis, existing beliefs about study results and synthesis methods, trust in the available data and the decision-making needs of end users. Participants differed in whether they would or would not meta-analyse some of the available results. Commonly, participants considered explicit or implicit vote counting, alongside or instead of meta-analysis, when it was not possible to meta-analyse all studies. Their description and interpretation of the results varied.
Conclusions: A diverse range of methodological approaches and influencing factors were identified. Although synthesis methods other than meta-analysis and their rationale are often not described in SRs, they could be described explicitly in protocols and SRs to enhance transparency. Patient, public and/or consumer involvement: Patients were not involved in this research. These findings are relevant in promoting transparency of methods, which will inform future guidance to strengthen SRs.