Background: When investigators analyze data from observational studies, they make numerous potentially justifiable, but still subjective, analytic decisions on which direction, magnitude, and statistical significance of findings can be contingent. This allows investigators to test many alternative analytic specifications and selectively report results for the specification that yields the most statistically significant or interesting results. Objective: We present a novel approach to interpret the results of observational studies in the context of the variation expected because of analytic flexibility. We apply this new method to the effect of unprocessed red meat on all-cause mortality.
Methods: We reviewed all observational studies addressing the effect of red meat and all-cause mortality from a recently published systematic review and documented variations in analytic choices across studies. We used data from the National Health and Nutrition Examination Survey (NHANES) 2007 to 2014 linked with National Death Index and applied specification curve analysis—a novel analytic method that involves defining and implementing all plausible and justifiable analytic specifications—to investigate the effect of unprocessed red meat on all-cause mortality. Our choice of analytic specifications was informed by analytic methods used in published studies addressing the same or similar questions.
Results: We applied specification curve analysis to NHANES, including 10,661 participants. In total, we performed 1,208 unique analyses, 48 (3.97%) of which produced statistically significant results, 40 (83.33%) of which indicated red meat reduced all-cause mortality, and 8 (16.67%) of which indicated red meat to increase mortality. The specification curve analysis produced a median hazard ratio of 0.94 [IQR: 0.83 to 1.05].
Conclusions: Our results suggest that inconsistency in the reported results in the literature may be explained by differences in analytic methods. We encourage evidence users to interpret the results of observational studies in the context of variation expected due to analytic flexibility. Patient, public involvement: We did not involve consumers in this investigation.

VoE Figure.pdf