Background: Systematic reviews (SRs) of randomised controlled trials (RCTs) often provide the highest certainty evidence for decision-making. However, if RCTs are not available to inform specific health questions, other study designs should be considered. This includes nonrandomised studies of interventions (NRSIs), which are frequently used in the field of vaccination.
Objectives: The key objectives are to illustrate the limitations and opportunities of including NRSIs in SRs using an exemplary review on vaccination against COVID-19 in children.
Methods: The exemplary review (https://doi.org/10.1016/S2352-4642(23)00078-0) followed Cochrane methodological standards. Drawing on the experience gained in the review process, practical challenges from study selection to interpretation of results will be illustrated. Further, limitations and opportunities of using NRSIs for informed decision making will be explored.
Results: The following key challenges in conducting our review with NRSIs were observed: (1) diversity of study designs (e.g., traditional cohort, ecological or test-negative case-control studies) in the field of vaccination, introducing clinical and methodological heterogeneity. (2) Additional risks of bias beyond the ones known from RCTs that originate from several factors pre-, at-, and postintervention level, particularly relating to methods used for selecting the study population, assignment of intervention groups and implementation of the intervention. (3) NRSIs often present different measures of adjusted effects (e.g., risk ratios, odds ratios or incidence risk ratios), challenging the synthesis of vaccine effectiveness estimates. (4) The evidence derived from NRSIs in a fast-changing pandemic situation requires very careful interpretation in relation to applicability and transferability (e.g., baseline immunity of study population and general population or investigated virus variants).
Conclusions: NRSIs are usually at a higher risk of bias and more heterogeneous in design than RCTs. Especially in fast-changing epidemiological landscapes, as observed during the COVID-19 pandemic, available RCT data were quickly outdated, whereas some NRSIs timely addressed changes in epidemiology (e.g., shift of variants). When considering concomitant limitations, our experience showed that NRSIs in this exemplary SR could be successfully used for timely information of health policy and practice. Consumer involvement: The paediatric working group of the Standing Committee on Immunisation (STIKO) in Germany was involved from prioritisation of the review question to interpretation of the findings.