An assessment of the design-by-treatment interaction model for network meta-analysis inconsistency
2Department of Health Research Methods, Evidence, and Impact, McMaster University, Canada
3Department of Primary Education, School of Education, University of Ioannina. Methods Support Unit, Cochrane Central Executive Team, Greece
4Medical Research Council Clinical Trials Unit at UCL, Institute of Clinical Trials and Methodology, University College London, England
5Population Health Sciences, NIHR Applied Research Collaboration West (ARC West), University Hospitals Bristol and Weston NHS Foundation Trust, England
6Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center, University of Freidburg, Germany
7Department of Primary Education, School of Education, University of Ioannina, Greece
8Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto. Division of Epidemiology and Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, Canada
Background: Network meta-analysis (NMA) is a powerful method that simultaneously synthesizes evidence from studies addressing the same clinical question comparing multiple interventions. The method allows inferences based on direct and indirect comparisons in a network. However, NMA results are reliable only when the prerequisite assumptions are met. Of interest, the consistency assumption requires that direct and indirect evidence in a network is in agreement. The design-by-treatment (DBT) interaction model is considered the best method to date; however, its statistical properties have not been well studied for complex networks.
Objectives: To assess the Type I error and Power of the inconsistency estimator from the DBT interaction model in triangular networks with arms denoted A, B, and C.
Methods: A simulation study in which, over 10,000 repeated iterations, we will simulate network meta-analysis data over a range of scenarios, fit frequentist, random-effect network meta-analysis models, and estimate the inconsistency of the network using the DBT model. From the 10,000 iterations, we estimate the estimator’s Power and Type I error based on the observed p-values. We consider varying values for the true odds ratio for the AB- (i.e., 0.65, 1.2, and 1.4) and AC-comparison (0.75, 1, and 1.4), inconsistency factor (0, 0.3, and 1), and number of studies between comparisons (1, 2, 5, and 10). Preliminary
Results: The power of the inconsistency test ranged from approximately 0.5 to 0.75, depending on the simulation scenario. Furthermore, the Type I error of the test ranged from approximately 0.4 to 0.45. Preliminary results indicate that the main driver of Power and Type I error is the assumed inconsistency factor in the data-generating mechanism.
Conclusions: Preliminary results indicate that the DBT inconsistency estimator suffers from a high Type I error and lacks sufficient Power to reliably detect inconsistency in a network. This suggests that further methodological work in assessing network inconsistency may be necessary so that NMAs used in informing decision-making are trustworthy. We intend to expand the simulations in our study to reflect other types of networks observed in practice (e.g., different geometries).
Patient, public and/or healthcare consumer involvement: Patients were not involved at this stage of the project.