Background: There is increasing concern that a significant proportion of randomised trials included in Cochrane reviews may not be trustworthy. Applying a Trustworthiness Screening Tool (TST) has already had a clinically important effect on several reviews published by the Cochrane Pregnancy and Childbirth Group.
Objectives: We wanted to assess the impact of removing untrustworthy randomised trials from already published Cochrane reviews on a defined clinical area (nutritional interventions during antenatal and postnatal period).
Methods: We applied the Cochrane Pregnancy and Childbirth TST to 375 randomised trials included in 18 Cochrane reviews. The TST has five domains (is the research governance trustworthy; are the baseline characteristics trustworthy; is the study feasible; are the results plausible; and is all relevant information available?). When additional information was needed, authors were contacted using a standard template. At least two attempts were made to contact the authors. At the end of the evaluation process, each study was classified as i) included (YES to all domains); ii) excluded (retracted study); or iii) awaiting classification (any NO to the TST questions).
Results: 95/375 studies (25%) were removed, affecting 14/18 (78%) reviews. 13/18 reviews (72%) showed a difference in the Summary of Findings tables (direction and size of effects and/or GRADE ratings). 6/18 Cochrane reviews (33%) were judged to require updating because of important differences in their conclusions, implication for practice, and/or implication for research.
Conclusions: Formal assessment of trustworthiness and inclusion only of studies that satisfy prespecified criteria for trustworthiness affects conclusions in a relatively large number of Cochrane reviews, with potentially important clinical implications for practice and research. The lack of consensus regarding the best tool(s) for assessing trustworthiness cannot be an excuse for ignoring this issue in future Cochrane reviews.
Patient, public and/or healthcare consumer involvement: N/A. Statement on relevance and importance to patients: Our work found that removal of studies that did not fulfil prespecified trustworthiness criteria affected the conclusions of a third of the Cochrane reviews that we assessed.

Background: Although it is critical to consider downstream consequences of changing disease definitions and to avoid the perils of overdiagnosis, disease definitions and diagnostic criteria are sometimes required to be developed, modified, or updated. In 2021, a group in Australia was tasked with updating the guideline for the diagnosis and assessment of Fetal Alcohol Spectrum Disorder. However, there is currently no universally agreed approach to developing diagnostic criteria or assessment guidelines.
Objectives: Our aim was to ensure that any new criteria or definitions proposed by our guideline group were as transparent as possible and supported by a rigorous development process. In line with best practice in guideline development, we aimed to use GRADE to assist with our guideline. However, there was no specific GRADE guidance currently for this type of guideline.
Methods: We used a novel application of the GRADE approach to inform the development of our guideline. Firstly, to determine potential diagnostic criteria for FASD, we performed a range of systematic reviews using the GRADE for establishing certainty in prognostic factors to determine the association between particular exposures and outcomes with FASD. We then used adapted evidence to decision frameworks (EtDF) to make a recommendation regarding whether a particular diagnostic criteria should be considered in our final set of criteria. Following this, we created an overarching EtDF for our proposed diagnostic criteria for FASD. Results and Conclusion: Our proposed revised definition, criteria, and assessment guidelines for FASD are informed by rigorous systematic reviews and a transparent decision-making process. We believe these methods may be suitable for adoption or adaptation for other groups creating diagnostic criteria and revising disease definitions.
Patient, public and/or healthcare consumer involvement: Extensive engagement with people with FASD and their caregivers occurred during this project.

Background: Decolonisation of research methods refers to the undoing of colonial ideologies within academic thinking and broadening critical and theoretical positions within methodological approaches to address power imbalances and inherent biases. Trust in systematic reviews is underpinned by transparency, rigour, and objectivity at all stages of the review process. Frameworks, such as GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) and GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research), aim to improve transparency surrounding certainty or confidence in the evidence. A key element of these tools is to assess the overall directness or relevance of the synthesis findings to examine whether they offer a ‘restricted’ answer to the review question because of the populations, interventions, comparators, or outcomes studied. Application of GRADE and CerQual could therefore be used to identify underrepresentation of important ethnic groups or marginalised populations in research.
Objectives: To explore how GRADE and GRADE-CERQual could be employed to support decolonisation of systematic reviews.
Methods: An exploration of how systematic reviews on issues known to disproportionately affect certain ethnic groups could use GRADE and CERQual to support interpretation of the evidence and decolonisation of research. We use case examples of reviews on conditions known to disproportionately affect people of African, African-Caribbean, and South-Asian descent, such as diabetes, to illustrate how reviewers might consider the directness of the evidence to those groups.
Conclusions: Systematic review teams should routinely consider the potential for underrepresentation of important or marginalised ethnic groups as a restricting factor when applying the ‘directness’ dimension of GRADE or the ‘relevance’ dimension of CERQual. Review teams should consider providing equity statements in their summary of findings table to mitigate issues of colonial bias in research and to reduce the risk of perpetuating underrepresentation of marginalised populations. Patient, public, and/or healthcare consumer involvement: No specific patient or healthcare consumer involvement was sought for this abstract because of the theoretical nature of the work. However, the authors both value and advocate for diverse stakeholder involvement in the systematic review process.

Background: Evidence Gap Maps (EGMs) are visual tools that present the available evidence on different thematic areas and highlight gaps for future research. EGMs, constructed by international Non-governmental Organisation Sightsavers, summarize, appraise, and present evidence from systematic or literature reviews across different eye health conditions, including cataract, refractive error, glaucoma, diabetic retinopathy, and trachoma. Updating an EGM periodically allows us to monitor changes to the extent and quality of the evidence base and to refine the focus for future research. This presentation will focus on the methods used to develop and update the EGMs and discuss their potential in promoting rigorous, high-quality research and countering misinformation.
Methods: After conducting a comprehensive literature search, we sift and extract data from all relevant reviews. Critical appraisal is conducted independently using the Supported Use of Research Evidence checklist, allowing us to attribute a confidence level of low, medium, or high to the review. We assessed the strength of evidence of the effect of each review. If the review reaches a conclusive answer to their research question using the available evidence, the evidence is classified as strong. If they were unable to answer the question because of insufficient evidence, the evidence is classified as weak. Otherwise, the evidence is inconclusive. Each review is displayed in a matrix where the columns show thematic areas relevant to the eye health condition, labelled as sectors and subsectors, and the rows show the strength of evidence. Updates are conducted in the same way as the original EGMs. Changes to the evidence base were assessed by comparing the numbers and percentages of included reviews across a number of areas, including the thematic focus, geographic representation, strength of evidence of effect, and methodological quality. Conclusion and patient and healthcare consumer involvement: EGMs allow policymakers and clinicians to plan their interventions based on the most up-to-date knowledge and high-quality research, which helps ensure that patients receive the best possible care. Updating the EGMs periodically allows researchers to monitor changes in the extent and quality of systematic review evidence and to prioritise future work to improve the evidence base and to close evidence gaps.

Refractive error evidence gap map details tab.png

Background: The global burden associated with common mental disorders is high, especially for people living in low resource settings. Although psychosocial interventions delivered by locally available lay or community health workers are effective, mechanisms of intervention response are poorly understood. One of the greatest barriers is that psychosocial interventions are administered as complex, multi-component “packages of care”.
Objectives: Our aim is to systematically review all the randomized controlled trials (RCTs) that have tested the efficacy of psychosocial interventions delivered through the task shifting modality to treat people suffering from common mental disorders (depression, anxiety, and related somatic complaints) in low resource settings, dismantle the intervention protocols creating a taxonomy of active intervention components, and reevaluate their efficacy.
Methods: We will use the component network meta-analysis (cNMA) methodology. The major benefit of cNMA is the possibility to disentangle intervention components and explore their effectiveness separately or in various combinations (even in disconnected networks). cNMA increases statistical power by combining direct and indirect comparisons while fully respecting the randomized structure of the evidence. According to the additive cNMA model which we will implement, adding a component “c” to a composite intervention “X” will lead to an increase (or decrease) of the effect size by an amount only dependent on “c”, and not on “X”. We will denote the corresponding component specific incremental standard mean difference (iSMD) so that iSMDc = SMD(X+c) v. (X). Combining these component-specific iSMDs will allow the estimation of SMD between any two composite interventions.
Results: A network of comparisons and a hierarchy that includes all intervention components expressed as iSMD, indicating the added benefit of adding a component to an intervention, will be presented. By selecting the most effective components it will be possible to outline a novel task shifting psychosocial intervention to be tested in future RCTs.
Conclusions: These findings will set the basis for further investigations in the field of precision medicine. This project is funded by the European Union’s HORIZON EUROPE research programme under grant agreement No 101061648 and is prioritized by Cochrane Global Mental Health. Patient, public, and/or healthcare consumer involvement: not applicable.

Abstract Cochrane_componentNMA.pdf

Background: The WHO and the Declaration of Helsinki require that every clinical study must be prospectively registered in a publicly accessible study registry (i.e., registration before enrolling the first patient). However, most evidence syntheses ignore violations of this fundamental moral and ethical principle. The Research Integrity Assessment (RIA) (doi:10.1002/jrsm.1599), a tool to assess adherence of randomized-controlled trials (RCTs) to the principles of Good Clinical Practice, and key elements of research integrity in clinical trials, assesses prospective registration and may help evidence synthesis authors to deal with studies which do not comply with the requirements. RIA was piloted in the updated Cochrane review “Ivermectin for preventing and treating COVID-19” revealing that lack of prospective registration was the most frequent reason for the exclusion of studies according to RIA.
Objectives: To evaluate trial registration in RCTs included in the study pool of COVID-19 systematic reviews (SRs) on interventional drugs and to analyse the impact with the application of RIA.
Methods: We searched for SRs investigating at least 1 of 13 different interventions for treatment or prevention of COVID-19 in Medline/PubMed until 09 June 2022. We selected Cochrane and non-Cochrane SRs with the largest RCT pool, extracted all included RCTs, evaluated trial registration data, and applied the RIA domain 2 (prospective trial registration) to all RCTs. RIA was independently applied by two reviewers; conflicts were resolved by a third reviewer.
Results: We identified 188 RCTs for RIA domain 2 assessment. Twelve RCTs were not registered and 176 RCTs were registered in at least 1 of 10 clinical trial registries. Using RIA, we assessed 109 prospectively registered RCTs as ‘no concernʼ, 51 non- or retrospectively registered RCTs as ‘excludeʼ, and 28 RCTs with inconsistent information between publication and registry record, missing information, inaccessible registry record, or no publication as ‘awaiting classificationʼ (Table 1). Conclusion: Almost one in three studies on interventional treatments for COVID-19 has not complied with the international standard of prospective trial registration. Authors of evidence synthesis can use the RIA tool as a transparent mechanism to manage studies that do not adhere to the international standard. Patient and/or public involvement: none.

Table -1_2023-03-06.pdf

Background: The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach assesses certainty of evidence (CoE) in a systematic review (SR) and facilitates drawing appropriate conclusions that consider limitations of the evidence. However, GRADE is not widely used in dentistry SRs.
Objectives: To evaluate the impact of using GRADE on conclusions in dentistry SRs.
Methods: From a sample of 200 dentistry SRs of randomized controlled trials included in a larger methodological study, we selected a subsample based on the most frequently reported outcomes. We will use the odds ratio and its 95% confidence interval to determine whether SRs not using GRADE are more likely to formulate inappropriate conclusions (i.e, conclusions relying on statistical significance, not considering limitations or formulating recommendations). For SRs not using GRADE, we will compare conclusions made by authors to conclusions we made based on our own GRADE assessments using the same effect thresholds as SR authors. We will calculate the number of conclusions that changed with respect to the described certainty (i.e., the extent to which authors recognized uncertainty, analogous to high, moderate, low, or very low CoE in GRADE) and magnitude of effect (i.e., whether the conclusions state there was a presence or absence of effect, negligible or important effect) after utilizing GRADE.
Results: We present the results of a subsample of 19 SRs reporting the outcome of pain. Six (32%) used GRADE, and 13 (68%) did not. Two SRs formulated inappropriate conclusions; both did not use GRADE. For SRs without GRADE, our GRADE assessments changed 62% of conclusions with respect to the described CoE. Our GRADE assessments changed the described magnitude in 25% of conclusions. We will present results for additional outcomes currently under analysis.
Conclusions: After completing all analyses, our conclusions will focus on the influence of GRADE on formulating appropriate conclusions which will provide insight into potential limitations that may arise when SRs formulate conclusions without GRADE assessments. Developing more transparent SR conclusions is relevant to patients as SRs are a valuable source of evidence for decision-makers. Patient, public, and/or healthcare consumer involvement: Patients were not involved.

Background: To reduce inequity, knowing whether interventions ‘work’ is not enough; we also need to know whether vulnerable subgroups experience them differently. However, vulnerable groups may not have equal opportunity to participate in research and, hence, may be underrepresented or missing entirely. Unless study findings are reported by subgroup, any differences among those included will be masked.
Objectives: As part of a systematic review (SR) exploring the effectiveness of remote therapies for drug/alcohol misuse, we undertook equity-related work with the objectives of understanding (1) who was excluded from the research, (2) who was represented in the study populations and (3) whether sub-groups experienced outcomes differently.
Methods: For each included primary study we assessed (1) the exclusion criteria; 2) the sociodemographic characteristics reported using the PROGRESS-plus framework; and (3) whether substance use outcomes were reported by equity-relevant characteristics and, if so, whether there was any variation.
Results: A total of 52 studies were included in the review, of which 47 had eligibility criteria with the potential to result in the exclusion of vulnerable subgroups, most notably those with mental health conditions (MHCs) and those without access to digital technology. Few studies justified the exclusions or reported how many people were excluded based on each criterion. Sociodemographic characteristics of the study populations were reported for all studies, although what the characteristics were and how they were measured varied. Most commonly and consistently reported were gender, age and ethnicity but it was rare that the study outcomes were reported by these or other factors.
Conclusions: The exclusion of certain vulnerable subgroups from the research process and the lack of reporting of outcomes by potential axes of inequalities have important implications for the interpretation and generalizability of findings of SRs. Particularly troubling in our review was the frequency with which those with MHCs were excluded. Understanding how different groups experience interventions is key to allow practitioners to be able to make informed decisions about their adoption in specific vulnerable groups and to ensure that interventions do not inadvertently produce, or exacerbate, inequities.
Patient, public and/or healthcare consumer involvement: An advisory group from policy and practice inputted to all aspects of the SR.

Background: Evidence and Gap Maps (EGMs) sit within a family of evidence synthesis methods that seek to address broader research questions. EGMs are a valuable tool in which a wider understanding is needed of existing research in a topic area and locating evidence gaps. They do not synthesise existing evidence, but by locating, categorizing, coding, and presenting the evidence in an interactive web based tool, with links to the primary research, they offer a valuable visualisation of existing evidence. Their popularity is growing, and they are particularly valued by policy makers.
Objectives: We undertook a scoping review to address the following questions: 1) What methods are currently used in their development and updating; 2) to what extent do they adhere to recommended guidance; 3) what procedures are in place for updating and maintaining EGMs; and 4) what methods are in place to evaluate the use of web based EGMs.
Methods: We developed a search strategy that included websites as well as database searches. We contacted hosting organisations to locate published EGMs. We used a piloted data extraction table to gather data. We only included EGMs with public facing online maps. We also retrieved any linked or supporting documentation.
Results: We included 96 EGMs addressing a wide range of topics, though they were predominantly in the field of international development, social care, and public health. Their potential in health care is currently less well realised. Approximately 43% did not undertake critical appraisal of the evidence. More than half of the EGMs were not linked to a description of their methodology or information that described the search dates or plans to update the map.
Conclusions: EGMs are an increasingly adopted approach in evidence synthesis and valued by policy makers. They are valuable tools for visualising evidence and highlighting gaps. Yet the guidance for their development is limited, and there is little standardisation in the processes of linking maps to the methods underpinning them. EGM processes must be more transparent and rigorous. Patient, public, and/or healthcare consumer involvement: Extensive consultation with the public on views on EGMs

Background: Network meta-analysis (NMA) is a powerful method that simultaneously synthesizes evidence from studies addressing the same clinical question comparing multiple interventions. The method allows inferences based on direct and indirect comparisons in a network. However, NMA results are reliable only when the prerequisite assumptions are met. Of interest, the consistency assumption requires that direct and indirect evidence in a network is in agreement. The design-by-treatment (DBT) interaction model is considered the best method to date; however, its statistical properties have not been well studied for complex networks.
Objectives: To assess the Type I error and Power of the inconsistency estimator from the DBT interaction model in triangular networks with arms denoted A, B, and C.
Methods: A simulation study in which, over 10,000 repeated iterations, we will simulate network meta-analysis data over a range of scenarios, fit frequentist, random-effect network meta-analysis models, and estimate the inconsistency of the network using the DBT model. From the 10,000 iterations, we estimate the estimator’s Power and Type I error based on the observed p-values. We consider varying values for the true odds ratio for the AB- (i.e., 0.65, 1.2, and 1.4) and AC-comparison (0.75, 1, and 1.4), inconsistency factor (0, 0.3, and 1), and number of studies between comparisons (1, 2, 5, and 10). Preliminary
Results: The power of the inconsistency test ranged from approximately 0.5 to 0.75, depending on the simulation scenario. Furthermore, the Type I error of the test ranged from approximately 0.4 to 0.45. Preliminary results indicate that the main driver of Power and Type I error is the assumed inconsistency factor in the data-generating mechanism.
Conclusions: Preliminary results indicate that the DBT inconsistency estimator suffers from a high Type I error and lacks sufficient Power to reliably detect inconsistency in a network. This suggests that further methodological work in assessing network inconsistency may be necessary so that NMAs used in informing decision-making are trustworthy. We intend to expand the simulations in our study to reflect other types of networks observed in practice (e.g., different geometries).
Patient, public and/or healthcare consumer involvement: Patients were not involved at this stage of the project.

Background: Cochrane reviews are regularly updated to reflect the most up-to-date evidence on health interventions. In 2021, Cochrane introduced a new policy to manage potentially problematic studies that are eligible for inclusion in a Cochrane review. This includes the management of studies with retractions, expressions of concern, and concerns about the trustworthiness of the data. It remains unclear how well such a policy is implemented for published Cochrane review updates.
Objectives: To provide an overview of the frequency of problematic studies included in Cochrane reviews and whether they have been excluded in Cochrane review updates according to Cochrane’s new policy.
Methods: We searched the Cochrane Database of Systematic Reviews to identify intervention review updates published in 2022 and their previous version. We compared these two versions to identify studies that were included in the previous version and excluded in the 2022 update. We also identified retracted studies that were included in the previous version by searching the Retraction Watch Database and assessed whether these were excluded from the updated version. Additionally, we recorded whether and how Cochrane’s new policy was implemented in the updates.
Results: We identified 123 pairs of Cochrane reviews from 37 Cochrane groups. The previous version of these reviews was published between 1996 and 2021, with a median of six included studies and 607 participants. There were 16 (13%) reviews with at least one previously included study excluded in the 2022 updates, including 4 (3%) excluding at least one (median 4, range 1-54) potentially problematic study. The previous version of two (2%) reviews included at least one retracted study, and one (1%) review did not exclude them in the 2022 update. Sixteen (13%) updates clearly specified the methods to assess and/or handle problematic studies, including 4 (3%) using a trustworthiness assessment tool and 12 (10%) mentioning the handling of retracted studies.
Conclusions: To facilitate the production of trustworthy evidence, it is imperative to identify and appropriately manage problematic studies eligible for Cochrane reviews. This study reflects the suboptimal implementation of Cochrane’s new research integrity policy in review updates, suggesting further guidance is needed.

Background: We recently completed a Cochrane review of diabetes quality improvement (QI) trials that aimed to identify promising QI strategies (or combinations of strategies) to deploy in practice or examine in future research. We coded interventions according to the presence of absence of 12 QI strategies and assessed the association between intervention components and postintervention outcomes with Bayesian meta-regressions. This is atypical for a Cochrane review—our analysis and output do not match the usual Cochrane processes—and thus necessitated adjustments in our reporting of results to align with Cochrane procedures.
Methods: We will describe the misalignment between the goals and outputs of our analysis with the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) approach for producing Cochrane Summary of Findings (SoF) tables and our operationalizations and solutions.
Results: The standard approach for producing an SoF table is to have a table for each intervention of interest. This was not feasible in our review, as studied interventions were diverse and rarely replicated and did not capture the full range of possible interventions that could comprise combinations of the 12 QI strategies. Most importantly, the standard SoF table was inconsistent with the goal of our analyses, which was learn across (diverse combinations of QI strategy) interventions to develop a theory of which QI strategies (or combinations) of strategies may be most promising for future practice or research. We therefore adjusted GRADE tables to report findings as they pertained to each QI strategy for each outcome. Additionally, we parsed data to inform GRADE assessments for each component (e.g., sample size, risk of bias, precision, consistency and directness) to rate certainty of evidence. Discussion: GRADE is a transparent tool for communicating review findings. Although the goal of our analyses did not match the standard setup of GRADE SoF tables, we found it feasible to adapt SoF tables to better align with our synthesis objective. Further GRADE guidance may be required to help guide future and ongoing similar reviews aimed at teasing apart active ingredients of interventions, rather than interventions as a whole.

Background: The Immunization Agenda 2030 prioritizes the populations without access to vaccines. Health equity has been increasingly incorporated into economic evaluations of vaccines to foster equitable access. Robust and standardized methods are needed to evaluate health equity impact of vaccination programs to ensure monitoring and effective addressing of inequities. However, methods currently vary and potentially affect the application of findings to inform policy decision-making.
Objectives: To identify and summarize economic evaluations of the health equity impact of vaccines and immunization programs, focusing on the methods and applications.
Methods: We searched PubMed, Embase, Econlit, and CEA Registry up to December 15, 2022. We included articles that met the following eligibility criteria: full-text articles of economic evaluations estimating costs, outcomes, and health equity impact of vaccines across equity-relevant subgroups in any context. We summarized how health equity was incorporated and evaluated in the selected studies. We performed reporting quality assessment using the Consolidated Health Economic Reporting Standard (CHEERS) 2022 statement.
Results: Twenty-one studies were included that performed health equity impact analysis to estimate the distributional impact of vaccines, such as deaths averted and financial risk protection, across equity-relevant subgroups. Eleven studies performed only health equity impact analysis as part of cost-effectiveness analyses to estimate the distributional impact and subpopulation incremental cost-effectiveness ratios of vaccines. Nine studies are Extended Cost-Effectiveness Analyses (ECEAs) that performed health equity impact analysis of vaccines with an estimation of the distributional financial risk protection. One study is a Distributional Cost-Effectiveness Analysis (DCEA) that performed a health equity impact analysis of vaccines incorporating equity-weighting and opportunity costs as the money was displaced to be spent on vaccines instead of other health services. These studies showed similar findings that introducing vaccines or improving vaccination coverage resulted in fewer deaths and higher financial risk benefits in subpopulations with higher disease burdens and lower vaccination coverage, particularly poorer income groups and those living in rural areas.
Conclusions: Methods to incorporate equity have been evolving progressively. Vaccination programs can enhance equity if their design and implementation address existing inequities in order to provide equitable vaccination coverage and achieve health equity. Patient, public, and/or healthcare consumer involvement: None.

Background: Scoping reviews, mapping reviews and evidence and gap maps (EGMs) are evidence synthesis methodologies that address broad research questions, aiming to describe a bigger picture rather than address a specific question about intervention effectiveness. They are being increasingly used to support a range of purposes including guiding research priorities and decision-making. There is, however, a confusing array of terminology used to describe these different approaches.
Objectives: In this commentary we aim to describe where there are differences in terminology and where this equates to differences in meaning. We demonstrate the different theoretical routes that underpin these differences. We suggest ways in which the approaches of scoping and mapping reviews may differ in order to guide consistency in reporting and method.
Methods: We undertook a critical analysis of the methods and cited methods used in published scoping, mapping and EGM reviews and guidance.
Results: Scoping reviews, mapping reviews and evidence maps are terms that are not used consistently within the literature, with different terms used to describe similar approaches and review objectives and sometimes the same term is used to describe different approaches. We show the scientific traditions that have shaped these approaches. Scoping reviews tend to have a more focused question and data extraction and analysis is richer than in mapping reviews and EGMs. EGMs have a unique place within the range of evidence synthesis approaches for greater transparency in identifying gaps in the evidence. EGMs are visual and interactive web-based tools. We propose that mapping and scoping reviews and EGMs have similarities that unite them as a group but with their unique differences. The accompanying graphic summarises these differences (Figure 1).
Conclusions: Understanding these similarities and differences is important for informing the development of methods used to undertake and report these types of evidence synthesis. We hope this work will help to improve consistency in description and reporting.
Patient, public and/or healthcare consumer involvement: None.

Figure 1.pdf

Background: Differences in baseline risk (BR) across studies may be a source of heterogeneity in meta-analysis (1; see attachment for a full list of references); thus, generalizability of findings across subgroups of patients requires constancy of treatment effects across BRs (2). In the case of risk ratio (RR), its independence from BR cannot hold due to its range limitations (2). Odds ratio (OR), instead, is theoretically independent from BR (3), but this property may not hold in practice (4). Moreover, in the specific case of using OR in network meta-analysis (NMA), even factors affecting baseline risk but not conditional effect may lead to inconsistency (5). Thus, regardless of the outcome measure adopted, in the case of binary outcomes, treatment effects should be considered as likely dependent from BR.
Objectives: To suggest a method making results from NMA for binary outcomes more reliable and generalizable, by expressing their dependence on BR, to equip clinicians and patients with better predictions of potential outcomes for each treatment option.
Methods: The arm-based (AB) model from (6) estimates log-odds of risks for each treatment arm by allowing both for dependence between baseline risk and treatment effect and for random effects (guaranteeing evaluation of heterogeneity and inconsistency). After estimating the logit of the probability of baseline risk and treatment effect, marginal treatment and BRs can be derived and marginal estimates of the desired effect size calculated (4).
Results: We will present an example of use of the AB model from (6) with a dichotomous outcome, showing pitfalls of the comparison-based approach using both OR and RR as outcome measures.
Conclusions: The never-ending debate about whether OR or RR should be used as an effect measure seems to be focused lately on ‘portability’, i.e., on their possible independence from BR (7). Owing to the general implausibility of such assumption in practice for both outcomes, we argue that NMA for binary events should adopt an AB approach, in order to express how results depend on BR, regardless of the outcome measure adopted.
Patient, public and/or healthcare consumer involvement: No patient, public or healthcare consumer has been involved.

references_abstract_Federico_Tedeschi_Colloquium_2023.pdf

Background: The availability of unsound or scientifically invalid work is a concern associated with preprint articles. The popularity of preprints in prevention research has increased due to the COVID-19 pandemic.
Objectives: The objectives of this study were to assess the consistency of results and conclusions of preprints in prevention research compared to peer-reviewed articles and to explore the perception of key stakeholders regarding the growing number of published preprints.
Methods: The study employed a mixed-methods approach that involved the development of a Python-based Web crawler to search MedRxiv for prevention preprints from January 1 to September 30, 2020. We ran an update search one year later to identify which preprints were published. We dually screened all results for prevention articles and developed a scheme to classify changes in effect sizes and conclusions. We also conducted and analyzed 19 qualitative online interviews with stakeholders who have expertise in prevention.
Results: The WebCrawler retrieved a total of 2,238 preprints, of which 594 were prevention research studies and 329 were epidemiological studies. Nearly half of these studies (48.9%) were published in peer-reviewed journals within one year, with a median time from upload to publication of 5.3 months (range: -0.1 to 11.3 months). Among published preprints, 16.8% of articles showed a change in effect size, which was a major change in 4.4% articles (i.e., a change greater than 25% of the original effect size). The conclusions changed in 43% of the studies, mostly in terms of style or wording (39%). Stakeholders perceive that preprints have advantages over peer-reviewed articles, including free access, fast submission, transparency, and feedback from colleagues. However, they also feel that preprints are not compatible with current quality standards and feedback culture and may have a detrimental effect on publication and career. Conclusion: A few prevention research articles experience changes in effect estimates and conclusions when they are published. Although these changes are small in number, they affect one in every 23 articles. Although preprints may seem like an attractive publishing option, there is still skepticism about their quality and potential consequences. We therefore warrant caution of using preprints of prevention research in decision-making.

Background: COVID-19 has had an unprecedented impact worldwide on health, healthcare, societies, and economies. Evidence for interventions emerged rapidly but was difficult to examine systematically in a short span of time. Few guidelines used formal evidence synthesis and GRADE approaches, particularly when tailoring to low- and middle-income countries (LMICs). To address this gap, Christian Medical College, Vellore, India, partnered with the Clinical Infectious Diseases Society of India (CIDS), Cochrane Infectious Diseases Group, Prof BV Moses Center for evidence-informed healthcare, and experts from 16 institutions in India and globally to form The Covid Management Guidelines India Group.
Objectives: To produce guidelines for the management of patients with acute COVID-19 in secondary and tertiary care settings in India.
Methods: The process incorporated GRADE and Cochrane Rapid Review approaches to identify and prioritise questions in areas of equipoise, screen and extract data from available studies, synthesise and determine certainty in the evidence, and enable expert working groups to produce recommendations or best practice statements tailored for India and other LMICs using the GRADE Evidence to Decision framework, which was all disseminated on a widely accessible platform: https://indiacovidguidelines.org.
Results: In the course of the project, clinicians across the country were trained in the development of guidelines using formal evidence synthesis and the GRADE approach. The guideline group comprising core, steering, methodology, evidence synthesis, dissemination, and intervention expert working groups with clearly defined roles and responsibilities synthesised guidelines pertaining to the use of interventions used in the treatment of COVID-19, including antivirals, anti-inflammatory, anticoagulation, antibodies, respiratory support, and other supportive management. Although acceptability and applicability of the guidelines is being investigated formally through a survey, one example of impact is that one state (Kerala, India) adopted our recommendation on anticoagulation in the state’s COVID-19 guidelines. An external advisory panel ensured scientific challenge and applicability to settings outside of those in which the experts practice.
Conclusions: This unique partnership applied a transparent GRADE approach to ensuring scientific rigor and an evidence-based approach taking into account local contextual factors in making clear timely guidelines relevant to India.

Background: Systematic reviews of interventions often fail to adequately consider equity. This impairs their ability to inform policy and practice decisions that aim to account for differential impacts. Central to this issue is how the dimensions of equity impacts (e.g., socioeconomic status) are conceptualised, as this shapes how equity is treated in the conduct and reporting of reviews.
Objectives: To describe the methods and frameworks (e.g., PROGRESS-Plus) used in systematic reviews to conceptualise the dimensions of equity impacts of public health interventions and issues encountered in their application. This complements a Cochrane review (Welch 2022) that surveyed the descriptive and analytic methods used to investigate health equity via PROGRESS-Plus.
Methods: Umbrella review of systematic reviews that include a focus on equity impacts of public health interventions. We used electronic database searches supplemented with automated searches of the OpenAlex dataset. An active learning algorithm was used to prioritise title-abstract records for manual screening. We extracted data from a purposively selected sample of reviews.
Results: We manually screened 2,060 prioritised title-abstract records, from which 322 full-text reports were assessed. We included 120 reports of systematic reviews. PROGRESS-Plus was the only formal method used for conceptualising dimensions of equity impacts, although most reviews that did not use it used equivalent dimensions (68/75 (91%)). Where intended methods were unable to be applied fully, this was usually because primary research studies did not report the necessary information (55/68 (81%)). Most reviews (87/120 (73%)) included an explicit rationale for focusing on equity impacts in general, but only 7% (8/120) justified their focus on (or exclusion of) specific dimensions. Authors highlighted challenges with applying these methods, including investigating constructs, such as socioeconomic status, that lack standardised operationalisation and measurement.
Conclusions: PROGRESS-Plus is the predominant method for conceptualising dimensions of equity impacts of public health interventions and mostly appears to be sufficiently comprehensive to encompass the scope of such investigations. However, significant practical and conceptual challenges need to be addressed if reviews assessing these impacts are to more meaningfully contribute to cumulative and useful evidence bases. Patient, public, and/or healthcare consumer involvement: None, but can contribute to more robust evidence production.

Background: The COVID-19 pandemic accentuated the need for comprehensive evidence synthesis to guide decisions regarding infection disease control and preparedness. In Sweden, an investigator was tasked by the government with reviewing the Infection Control Act and analyzing the need for new regulations for future pandemics, with special considerations to measures that were effective during the COVID-19 pandemic. Against this background, SBU was requested to provide support to the investigator.
Objectives: To facilitate policymaking by conducting an overview of systematic reviews of nonpharmacological interventions (NPIs) and present the results in an interactive, user-friendly format.
Methods: A PICO was formulated comprising 10 NPIs used to prevent the spread of infections during outbreaks of COVID-19, SARS, MERS, Influenza and Ebola. A structured literature search for systematic reviews was conducted in Medline and Scopus. Eligible reviews were assessed for risk of bias using the AMSTAR checklist. The reviews were mapped according to risk of bias level and type of interventions and infections targeted and incorporated in a digital map.
Results: Seventy-eight systematic reviews were identified and included in the map. Of those, 43 were assessed as having high risk of bias and 35 as having low to moderate risk of bias. Most reviews focused on the effectiveness of various physical distance interventions or the effectiveness of facemasks. The vast majority of reviews and primary studies were conducted during the COVID-19 pandemic. The digital map facilitated navigation among the reviews and was used to support the legislative process.
Conclusions: Digital maps may be a useful tool to present and visualize available evidence and evidence gaps in an intuitive and interactive format that facilitates navigation. They can aid decision-making in complex fields such as public health measures, where it is challenging but important to identify and take into account the best available evidence of potential benefits and harms.
Patient, public and/or healthcare consumer involvement: Not applicable.

Background: Network meta-analysis (NMA) allows synthesising the evidence simultaneously on multiple treatments. A key output of NMA is the relative ranking of the treatments; nevertheless, it has attracted a lot of criticism. This is mainly because ranking is a very influential output and, thus, prone to over-interpretations even when relative effects imply small differences between the alternative treatments. To date, common ranking methods rely on score metrics which are calculated based on the summary effects. Such metrics lack a straightforward interpretation, although it is still unclear how to measure their uncertainty.
Objectives: To introduce a new modelling approach for networks of interventions that produces treatment hierarchies accounting for the clinical importance of the study-specific relative effects as well as for their uncertainty.
Methods: We adapt methodology previously suggested for ranking in sports tournaments into the context of NMA. We use extensions of the so called ‘Bradley-Terry’ models, which are a family of probabilistic models that aim to predict the outcome of pairwise comparisons. We first translate the study-specific relative effects and their confidence intervals into wins, losses, and ties based on predefined minimally clinically important differences between treatments. Then, based on the number of wins, our model estimates the ‘worth’ of each treatment which is used as an intuitive measure to rank the treatments. This approach naturally captures the uncertainty of ranking because the estimates of treatment worth are accompanied by confidence intervals. The model also allows to consider simultaneously multiple outcomes in ranking by implementing a vector of several contrasts for each outcome. Finally, study precision and other important characteristics, such as risk of bias, can be incorporated by an additional parameter that represents the ‘importance’ of each study. Results and
Conclusions: We illustrate our model using a Cochrane NMA comparing 7 treatment classes for chronic plaque psoriasis. Our model is able to clearly indicate the two classes with the largest worth but also classes with similar worth. The latter is a major strength of our approach as it preserves from exaggerating unimportant differences between treatments and drawing spurious conclusions. Patient, public, and/or healthcare consumer involvement: None.

Background: Preprints have emerged as a major source of research communication during the COVID-19 pandemic. However, questions were raised concerning the reliability of their results.
Objectives: To evaluate whether effect estimates differ between preprint and peer-reviewed journal randomised controlled trials (RCTs).
Methods: Data were derived from the COVID-NMA (covid-nma.com) initiative, a living systematic review of RCTs evaluating preventive interventions, treatments and vaccines for COVID-19. Meta-analyses with at least one preprint and one peer-reviewed journal article evaluating pharmacological treatments vs. standard of care/placebo were included up to July 20, 2022. Predefined COVID-NMA ‘critical outcomes’ at 28 days were considered. A meta-epidemiological analysis estimated the difference in effect estimates [expressed as the ratio of odds ratio (ROR)] between preprint and peer-reviewed journal RCTs. An ROR of <1 indicated that preprint trials yielded larger effect estimates.
Results: Thirty-seven meta-analyses (114 RCTs—44 preprints, 70 peer-reviewed journal articles) were selected. Twenty-four meta-analyses assessed hospitalized patients (81 RCTs), and 13 assessed outpatients (33 RCTs). The median number of RCTs per meta-analysis was 2 (IQR, 2-4; maximum, 11). The median sample size of RCTs was 199 (IQR, 99-478), 68% were prospectively registered, 67% received industry or mixed funding, 79% were multicentric trials, and 75% had some concerns of overall risk of bias. Overall, there was no significant difference in effect estimates between preprint and peer-reviewed journal trials (ROR, 0.88; 95% CI, 0.71-1.09; I2 = 17.8%; τ2= 0.06) (Figure 1). Post-hoc sensitivity analyses of meta-analyses with only two trials (ROR, 0.86; 95% CI, 0.51-1.45; I2 = 22.2%; τ2= 0.19) and at least three trials (ROR, 0.98; 95% CI, 0.84-1.14; I2 = 0.0%; τ2= 0.00) found consistent results.
Conclusions: No important difference in the treatment effect between preprints and peer-reviewed publications was found, particularly in meta-analyses with at least three trials. Systematic reviewers and meta-analysts should assess preprint inclusion individually, accounting for risk of bias and completeness of reporting.
Patient, public and/or healthcare consumer involvement: Peer review is a lengthy process that, during a pandemic, can inadvertently cost lives. Because of the demand for faster access to scientific knowledge, preprints offer a solution, particularly to patients.

Figure1.jpeg

Background: Assessing how much confidence users can place in qualitative evidence synthesis (QES) findings is important for producing trusted qualitative evidence on patients’ experiences and perspectives of health issues and interventions. Applying the GRADE-CERQual approach to assess confidence in the evidence is a required step in a Cochrane QES. Use of GRADE-CERQual has increased rapidly globally.
Objectives: We evaluated how GRADE-CERQual has been used in the literature by, firstly, describing GRADE-CERQual’s uptake in QES; and, secondly, developing and applying reporting and fidelity criteria to identify good examples and areas for improvement.
Methods: We undertook citation searches in six databases for 18 key publications on the GRADE-CERQual approach and also used keyword searches. Two researchers screened the output, and publications identified as evidence syntheses that used GRADE-CERQual went on to full-text coding and charting. Reporting and fidelity criteria were developed and then applied using NVivo12 software. We used qualitative content analysis and descriptive statistical approaches to analyse the data.
Results: Our searches identified 1,312 records. Two hundred thirty-three of these were reviews claiming to have applied GRADE-CERQual. Approximately 41.6% (97 studies) had seriously misapplied the GRADE-CERQual approach and were excluded from further fidelity and reporting assessment. Serious misuses were applying GRADE-CERQual in quantitative evidence synthesis, interpreting it to be a critical appraisal tool, and applying it to review findings as a whole, instead of individual review findings. One hundred thirty-six studies applied it as intended to individual review findings. We categorised the most common reporting issues into three broad areas: labelling, terminology, and completeness. The most common fidelity issues centred on the conceptualisation and application of the four GRADE-CERQual components. Fewer fidelity concerns were identified in syntheses citing the most recent guidance compared to those citing earlier publications. Conceptual challenges were identified around distinguishing between quality and confidence, between themes and review findings, and between making assessments at the study level versus the review finding level.
Conclusions: Findings from this evaluation help to inform new agendas for the field of QES, identify topics for further GRADE-CERQual guidance development, and support review authors to avoid common pitfalls and improve reporting and fidelity.

Background: Equity is at the forefront of clinical and public health practice and policy. The incorporation of equity in guideline recommendation requires additional considerations for development methodology. The GRADE working group proposed the consideration of equity in several stages of the guideline development process, such as applying an equity lens in the conduct of the evidence syntheses. However, there is no pragmatic guidance on how to collect evidence from systematic reviews to inform equity considerations in guideline recommendations.
Objectives: We operationalized the GRADE-Equity criterion for collecting and appraising evidence from primary studies of systematic reviews to inform guideline recommendations. We demonstrated the application of this plan in a clinical practice guideline on prescribing medical cannabis for patients with chronic pain.
Methods: We developed the operationalization plan through three steps. First, we reviewed guidance related to guideline development, including the GRADE-Equity series, GRADE evidence to decision (EtD) frameworks. Second, we drafted the operationalization plan using the four signalling questions for the “impact on health equity” criterion of the GRADE EtD and refined the draft based on analyses of reported equity considerations in primary studies. Third, we presented the plan to the steering committee of the medical cannabis guideline and developed an evaluation framework to pilot our approach using primary studies identified from systematic reviews of benefits and harms, values, and preferences.
Results: We propose the following seven-step operationalization plan: 1) identify populations experiencing inequities, 2) examine available data for specific populations, 3) evaluate population baseline risk for primary outcomes, 4) assess representation of these populations in primary studies, 5) appraise analyses, 6) identify barriers to implementation of effective interventions for these populations, and 7) suggest supportive strategies to facilitate implementation of effective interventions. The evaluation of each study ranged from 10 to 30 minutes, depending on the amount of detail provided in the studies. We present examples describing its application to the cannabis guideline.
Conclusions: We present a pragmatic approach to inform equity considerations of systematic reviews informing guideline recommendations. The application of this approach in different topics is needed to test its reliability and feasibility.
Patient, public and/or healthcare consumer involvement: no.

Background: Evidence gap maps (EGMs) are a novel approach to establishing the breadth and depth of evidence on health. Their visual nature promotes knowledge translation to key stakeholders, including patients, practitioners and policymakers. The methods used to produce EGMs are systematic and rigorous but are open to challenges owing to their emerging nature in the canon of evidence synthesis. These challenges resulted in novel methods used within the production of an EGM surrounding interventions for obstetric fistula.
Objectives: To describe two novel methods used in conducting an EGM of interventions for treating obstetric fistula.
Methods: Although development of a framework is mandatory for EGMs, how stakeholders should be involved in establishing the framework is currently unclear. As such, a survey of key stakeholders (including patients and practitioners) was produced using Qualtrics to establish their perspectives on the most important interventions and outcomes to prioritise within the framework. The EGM was then produced in line with recommendations from the Campbell Collaboration and followed a prepublished protocol. Data synthesis was conducted in EPPI-Mapper. Sensitivity analyses using filters applied to the EGM were performed to determine where sufficient homogeneity to perform future evidence syntheses may be possible; this is not currently part of EGM guidance.
Results: Involving key stakeholders in the development of the EGM formed the structure of the evidence map by arranging the framework by what was considered most important to them. However, only 39 responses were received; focus groups or individual interviews may have garnered further insight, particularly if carried out in areas of high fistula prevalence. The reporting of the sensitivity analyses highlighted deficiencies in the evidence base; only 7 of 28 included studies [a combination of randomised controlled trials (RCTs) and non-RCTs] remained in the EGM and further illuminated the type of new primary data needed.
Conclusions: Further guidance on stakeholder involvement in EGMs, as well as development in the technology and methodologies supporting the production of EGMs, may help to address some of the current challenges faced in their production.
Patient, public and/or healthcare consumer involvement: Patients and the public were involved in answering the survey that determined the framework.

Background: The reliability of the results of a network meta-analysis (NMA) lies in the plausibility of the key assumption of transitivity, which implies that the effect modifiers’ distribution is similar across treatment comparisons. Transitivity is statistically manifested through the consistency assumption, which suggests that direct and indirect evidence are in agreement. Several methods have been suggested to evaluate consistency. A common approach for testing network consistency suggests adding inconsistency factors to the NMA model.
Objectives: To evaluate the consistency assumption of NMA in a Bayesian framework.
Methods: We describe each inconsistency factor with a candidate covariate whose inclusion on the model relies on variable selection techniques. Our proposed method, Stochastic Search Inconsistency Factor Selection (SSIFS), evaluates the consistency assumption both locally and globally by applying the stochastic search variable selection method to determine whether the inconsistency factors should be included in the model. The posterior inclusion probability of each inconsistency factor quantifies how likely a specific comparison is to be inconsistent. We use posterior model odds or the median probability model to decide on the importance of inconsistency factors.
Results: Differences between direct and indirect evidence can be incorporated into the inconsistency detection process. A key point of our proposed approach is the construction of a reasonable “informative” prior concerning network consistency. The prior is based on the elicitation of information-derived historical data from 201 published network meta-analyses. The performance of our proposed method is evaluated in two published network meta-analyses.
Conclusions: SSIFS is a novel Bayesian method that evaluates the consistency assumption both globally and locally. The proposed method is publicly available in an R package called ssifs, developed and maintained by the authors of this work. Patient, public, and/or healthcare consumer involvement: -

Background: Synthesizers of evidence are increasingly likely to encounter studies published in predatory journals during the evidence synthesis process. Predatory journals and the studies published within them have caused significant disruption across the scientific landscape and present unique concerns within academia. This is due to the lack of transparent editorial oversight they employ, which may increase the risk that the studies published within these journals are erroneous or fraudulent.
Objectives: Under the broader aim to develop methodological guidance for the use of studies published within predatory journals in evidence syntheses, the objective of this research was to explore the attitudes, opinions and experiences of experts in the synthesis of evidence regarding predatory journals.
Methods: A global, descriptive survey-based cross-sectional study was carried out between April 1 2021 and June 1 2021. The survey was sent to prominent bodies in the field of evidence-based healthcare and systematic reviews, including JBI, Cochrane, Guidelines International Network (GIN), Campbell Collaboration and GRADE. The study utilized a self-administered questionnaire that was coded and disseminated online through Survey Monkey for data collection from the target participant. Because the target audience was those who had experience in evidence synthesis, survey logic was used to remove responses from respondents who lacked this experience.
Results: Two hundred and sixty-four evidence synthesis experts responded to this survey. Most respondents agreed with the definition of a predatory journal (86%); however, several (19%) responded that this definition was difficult to apply practically. Many respondents believed that studies published in predatory journals are still eligible for inclusion into an evidence synthesis project. However, this was only after the study had been determined to be ‘high-quality’ (39%) or if the results were validated (13%). Only 32% of the respondents previously used a checklist or tool to identify a predatory journal.
Conclusions: The results of this project identify that there is a need for more consensus-based guidance regarding the inclusion of studies published within predatory journals into an evidence synthesis project. While critical appraisal of these studies is an expected quality control method, evidence synthesis authors are urged to consider additional steps in their future evidence synthesis projects.

Background: GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research) was developed to support the increasing use of review findings from qualitative evidence syntheses within policy and decision-making. To date, the approach has mainly been applied to aggregative synthesis methodologies and descriptive review findings. Current GRADE-CERQual guidance suggests there is a need for testing with more diverse review methods and outputs, which will provide important insights for the ongoing evolution of the approach.
Objectives: This presentation aims to contribute to the evolution of GRADE-CERQual by reflecting on our experiences of applying the approach to the findings that emerged from a Cochrane meta-ethnography on childhood vaccination acceptance. Specifically, we describe the similarities and differences, challenges, and dilemmas we experienced when applying the approach to the more interpretive versus more descriptive findings that emerged from our review.
Results: We found that we were able to apply the core criteria and principles of GRADE-CERQual in ways that were congruent with the methodologies and epistemologies of a meta-ethnographic approach and associated interpretive findings. We also found that the practical application processes were similar for the more descriptive versus interpretive findings. The main differences we experienced related to the level of demand placed on the evidence supporting the review finding and the level of complexity involved with the decisions. Compared with the more descriptive findings, the more interpretive findings required supporting data that were richer, thicker, more contextually situated, and methodologically stronger for us to have the same level of confidence in them. Making the assessments for the more interpretive findings also involved greater dilemmas and more complicated forms of judgement, particularly for the component of coherence. We provide practical examples to illustrate these complexities and how we approached them.
Conclusions: Our experiences of applying GRADE-CERQual to more interpretive review findings generated various challenges and quandaries which we highlight and hope will offer a platform for further engagement. This could enhance the usability of GRADE-CERQual and in turn the kinds of knowledge that count within health decision-making, with ultimate positive effect on patients and other evidence users. Patient, public, and/or healthcare consumer involvement: None.

Background: Journal editors are the gatekeepers of our research. With increasing evidence that Equity, Diversity, and Inclusion (EDI) are beneficial to research and academia, some editors have expressed their desire to improve EDI of their journals. The Royal Society of Chemistry (RSC) established a minimum set of requirements aimed at improving EDI in scholarly publishing through a joint commitment action plan with more than 56 publishing organizations. There are several commitments in place to improve EDI in journals, however, their effectiveness is yet to be determined.
Objectives: We provide practical approaches for editors, journal publishers, and peer reviewers to improve EDI in academic journals based on the six minimum standards set by the RSC.
Methods: We searched publisher websites, empirical studies, and relevant resources that evaluated the impact of any approach to improve EDI in journals.
Results: We suggest the following six approaches that journal editors can adopt to meet the RSC minimum criteria: (1) adopt a journal EDI statement with clear, actionable steps to achieve it; (2) promote the use of inclusive and bias-free language; (3) appoint a journal’s EDI director or lead; (4) establish an EDI mentoring approach; (5) monitor adherence to EDI principles; and (6) publish reports on EDI actions and achievements. Peer reviewers need to support the author’s needs whether they were language needs by directing them to language services or their clinical conditions that require accommodations. We also provide examples of journals that have implemented some of these strategies to improve EDI.
Conclusions: EDI issues in academia are tightly intertwined with systemic oppression that is integrated in policies and regulations of academic progression. Publishers and journal editors need to test the feasibility of implementing these approaches and assess their impact on improving EDI. Patient, public, and/or healthcare consumer involvement: no.

Background: Evidence and gap maps (EGMs) are used to summarize the quantity, quality, and main characteristics of an evidence base and are an interactive resource for evidence users to identify research that meets their specific interests and requirements. EGMs are of great potential utility for health and social care policy makers who wish to gain an overview of a broad evidence base to inform decision making. However, differences in expectations and understanding of EGMs and systemic review processes between researchers and policy makers means production of EGMs and ensuring their future utility is not always straightforward.
Objectives: To discuss the challenges experienced by the Exeter Policy and Research Programme Evidence Review Facility whilst working alongside policy makers to produce EGMs to inform health and social care policy.
Methods: To establish if an EGM is appropriate to meet the needs of policy makers, we work closely with them to understand the size and main features of the existing literature base through extensive scoping. We provide examples of previous EGMs to support policy makers’ understanding of their functionality and consider how they may use the EGM. Policy makers are consulted on designing the framework and are given the opportunity to revise its format prior to publication of the final report.
Results: Working with policy makers to create EGMs that are relevant and useful can be a protracted process, especially when a focused research question has not been identified. It is not always possible to structure the EGM according to the needs of policy makers or to identify the features of the evidence of interest because of limitations in reporting, which may only become fully apparent after commencing the mapping process.
Conclusions: A better understanding of how EGMs are used to inform policy making would enable researchers to determine the potential strengths and limitations of this approach in different contexts.
Patient, public and/or healthcare consumer involvement: During development of the map, feedback from patients and members of the public is sought to inform the scope and format and ascertain the accessibility of the EGM.

Background: Clinicians often need to evaluate comparative effectiveness of relevant treatment options for their patients. Network meta-analyses (NMAs) provide an important resource for clinicians to guide decision-making; however, these analyses are complex, and interpretation may prove challenging for users.
Objectives: To evaluate and develop a novel interactive Summary of Findings table (the MATCH-IT tool) for network meta-analysis results, to support clinicians in interpreting and applying comparative effectiveness evidence into decision-making.
Methods: We performed qualitative user-testing with a convenience sample of practicing physicians working in hospitals and general practice in Norway, Belgium, and Canada. User testing entailed a brief introduction with a clinical scenario, which was a think aloud session with participant-tool-interaction followed by a semi-structured interview. The tests were recorded, transcribed, and analysed using directed content analysis. The results informed the iterative development process of the MATCH-IT tool.
Results: Five rounds of user-testing, with 26 participants in total, have resulted in four iterations with updates in the tutorial, layout, navigation functionality, and interphase, as well as the practical-issues-section. We have also added an FAQ section. Clinicians who participated in user-testing had been in clinical practice from 0 to 30 years (median 6, IQR 3-11). Most had little or no prior experience with interpretation of NMA results and sparse knowledge about GRADE methodology. Most clinicians perceived MATCH-IT as easy to interpret and navigate and appreciated its ability to provide overview of the evidence, although some of the interactive features proved to be difficult to discover intuitively. They perceived pictograms and inclusion of practical-issues-information as potentially useful features when interacting with patients. Categorization of results with colour coding to indicate both certainty of evidence and relative effectiveness of interventions was appreciated, and filtering functionality allowed participants to efficiently browse and focus in on the interventions believed to be most relevant for a given clinical scenario.
Conclusions: MATCH-IT shows promise in supporting decision-making for clinicians when presented with results from NMAs. The tool has already been published within Cochrane systematic reviews and clinical practice guidelines. Patient, public, and/or healthcare consumer involvement: Clinicians were the subject for user testing, ensuring user friendliness for this key consumer group.

MatchIt for Clinicians.png

Background: Summarizing all available evidence while decreasing bias raises important methodological challenges for systematic reviews (SRs). Grey literature in SRs is important for minimizing the risk of publication bias but is often inadequately reported, which may reduce the credibility of the evidence synthesis.
Objectives: To evaluate the extent of consistency between conference abstracts (CAs) and their final publications and the impact of including results from CAs on meta-analytic effect estimates and the certainty of the evidence (CoE) in our living systematic review and network meta-analysis (SRNMA) for COVID-19.
Methods: We will conduct a retrospective review of all CAs included in our living SRNMA of drug treatments for COVID-19 until January 2023. For all CAs included in our living SRNMA with a subsequent final publication, two reviewers, working independently, will compare the results and risk of bias assessments of the CA and its final publication. To assess the impact of the inclusion of CAs on effect estimates and the CoE, we will include the outcomes most frequently reported in the CAs reporting data for interventions with existing recommendations according to the January 2023 World Health Organization guidelines. We will conduct two analyses, one including the CAs and one excluding them. We will compare the results of these analyses to evaluate the magnitude of the change in the pooled estimate and identify any changes to the CoE.
Results: This work is in progress. We will describe the nature and frequency of discrepancies between CAs and their final publications. We will present how effect estimates and the CoE changed with and without the CAs. We will consider the direction of effect to be different, whether the effect changes from beneficial to harmful or vice versa, considering the thresholds of effectiveness established in the SRNMA. Conclusion: The results of this analysis will highlight the impact of including CAs in a living SRNMA, which can inform eligibility decisions for future evidence syntheses. Understanding the influence of including CAs in network meta-analyses can assist decision-makers with interpreting the results of evidence syntheses with CAs.
Patient, public and/or healthcare consumer involvement: Patient partners were not involved in this study.

Background: Cochrane systematic reviews are a valuable source of evidence-based information for healthcare decision-making. However, it is important to ensure that the analysis of these reviews considers equity, diversity, and inclusion (EDI) to represent diverse populations better and address disparities in health outcomes.
Objectives: This study aimed to assess the feasibility and effectiveness of using artificial intelligence (AI) to incorporate EDI considerations into the analysis of Cochrane systematic reviews.
Methods: A sample of Cochrane systematic reviews was analyzed using AI algorithms to identify EDI-related issues, such as disparities in study populations, missing data on diverse populations, and the under-representation of marginalized groups in the evidence base. The algorithms used machine learning techniques to analyze the data and provide insights into these disparities.
Results: The study found that AI algorithms effectively identified EDI-related issues in the analysis of Cochrane systematic reviews. The AI-powered analysis provided insights into disparities in study populations and the under-representation of marginalized groups in the evidence base. The algorithms also provided recommendations for improving EDI considerations in future studies and systematic reviews.
Conclusions: The study highlights the importance of considering EDI in analyzing Cochrane systematic reviews. Using AI to incorporate EDI considerations is a feasible and effective way to ensure that the information is representative of diverse populations and addresses disparities in health outcomes. The study demonstrates the potential of AI to support EDI in the analysis of evidence-based information and to improve the quality and representativeness of Cochrane systematic reviews. In this study, patient and public involvement was incorporated through consultation with stakeholders and subject matter experts to understand the barriers and facilitators to incorporating EDI considerations in the analysis of Cochrane systematic reviews. This engagement helped to ensure that the AI algorithms developed in this study effectively addressed the needs and perspectives of diverse populations and that the recommendations provided were relevant and meaningful to patients, the public, and healthcare consumers.

Background: In the current absence of direct “evidence on” rehabilitation interventions for people with symptoms of post-COVID-19 condition (PCC), we can search and synthesize the indirect “evidence relevant to” coming from interventions effective for these symptoms in other health conditions. The World Health Organization (WHO) required this information to inform expert teams and provide specific recommendations in their Guidelines. So we defined “evidence relevant to” as the synthesis of evidence focusing on the rehabilitative management of impairments, activity limitations and participation restrictions, informed by different health conditions presenting with similar symptoms.
Objectives: We were asked by the WHO to synthesize the Cochrane evidence relevant to rehabilitation for PCC-related symptoms: fatigue, postexertional malaise, orthostatic intolerance dyspnea, arthralgia, dysphagia, dysphonia, olfactory disease, cognitive impairment, anxiety and depression.
Methods: We searched the last 5 years’ Cochrane Systematic Reviews (CSRs) using specific terms for each symptom, “rehabilitation” and their synonyms in the Cochrane Library. We extracted the available evidence and summarized it using maps. We grouped the included CSRs for health conditions and interventions, indicating the effect and the certainty of evidence.
Results: Out of 5,734 CSRs, we found 32 for fatigue, 4 for exercise intolerance, 10 for arthralgia, 15 for dyspnea, 1 for dysphagia, 17 for cognitive impairment and 37 for anxiety and depression, published between 2016 and 2021. They provided data from several health conditions, such as cancer, chronic respiratory diseases, osteoarthritis and neurological disorders. Effective interventions for fatigue, exercise intolerance, dyspnea and arthralgia included exercise training and physical activities, telerehabilitation, multicomponent and educational interventions. Dysphagia was mainly treated with swallowing therapy, while cognitive aspects were managed with exercise, cognitive training and educational programs. The overall certainty of evidence was low to very low and moderate/high in a few cases. We did not identify CSRs that specifically addressed postexertional malaise, orthostatic intolerance, dysphonia and olfactory disease.
Conclusions: The current findings served as the basis for the recommendations on treatments for PCC symptoms published in the current WHO Guidelines for clinical practice. These results are the first step of indirect evidence able to generate helpful hypotheses for clinical practice and future research. Patient involvement: NA.

Background: For decades, statisticians and methodologists have severely criticized the undue reliance on strict p-value thresholds (usually 0.05) and the misinterpretation of statistical significance in medical research. Despite legitimate criticism, primary research and systematic reviews continue to rely heavily on statistical significance without taking a nuanced approach when interpreting nonsignificant effect estimates.
Objectives: We aimed to evaluate the language employed by Cochrane authors to emphasize differences in nonsignificant treatment effects they considered relevant. We also sought to measure the magnitude of these differences and determine whether they differed from statistically nonsignificant effect estimates that authors interpreted as similar or not different.
Methods: We screened all Cochrane reviews of interventions published between November 2017 and 2022 for statistically nonsignificant effect estimates that authors presented as meaningful differences. We classified interpretations qualitatively and assessed them quantitatively by calculating two areas under the curve (AUC): one for the confidence interval (CI) exceeding the null, indicating a greater effect of one intervention, and another for the CI exceeding a minimal important difference. We compared the AUC of nonsignificant effect estimates that authors interpreted as meaningful with those interpreted as nonmeaningful.
Results: We screened 2,337 Cochrane reviews, finding 139 cases that emphasized meaningful differences of nonsignificant results. Most commonly, authors used qualifying words (e.g., may/might/could) to express uncertainty (47.5%) and sometimes emphasized lack of significance (19.4%). In 27% of cases, authors made absolute claims about the greater efficacy or harm of one intervention without acknowledging statistical uncertainty, which could be misleading. The AUC analysis of the CIs is ongoing and will be available at the Colloquium.
Conclusions: Nuanced interpretations of statistically nonsignificant results were rare in Cochrane reviews. Authors should consider these subtleties when interpreting nonsignificant effect estimates to ensure that meaningful differences are not overlooked and that conclusions drawn are as accurate and informative as possible. Future research should explore the most effective language for distinguishing and interpreting effect estimates so that knowledge users interpret this information correctly.
Patient, public and/or healthcare consumer involvement: The choice of how to present statistically nonsignificant results can substantially impact the interpretation of results made by decision-makers.

Background: Conducting Cochrane reviews can be challenging. These reviews are often carried out by healthcare professionals and experienced researchers because of their complexity and associated workload. University students must complete a research project to obtain an undergraduate or postgraduate degree. Thus, there is a vast potential for collaboration between students and Cochrane. Cochrane is present in universities and hospitals, so one potential partnership may be the realization of end-of-degree research projects and master/PhD theses based on Cochrane activities.
Objectives: (1) To explore how to incorporate Cochrane activities into the undergraduate and postgraduate curricula, (2) to share the experiences of centres in promoting the participation of university students in Cochrane activities and (3) to promote networking among Cochrane trainers.
Description: The workshop is directed at Cochrane trainers, coordinators and undergraduate and postgraduate curricula teachers. The session will highlight new opportunities to link Cochrane activities and university students. The workshop will include two short presentations (10 minutes each) and one interactive session (40 minutes). The group will reconvene for the final 30 minutes to gather feedback about synergies between students and Cochrane. The first presentation (Cochrane Sweden and Cochrane Austria) will focus on experiences with the Cochrane International Mobility scheme (see also https://www.cochrane.org/news/cochrane-international-mobility) and how this could be integrated into the university curriculum. The second presentation will show the experience of different Cochrane Centres (Cochrane Madrid, Iberoamerican Cochrane Centre-Barcelona and Cochrane Complementary Medicine-Switzerland) in integrating Cochrane activities and evidence synthesis methods into the medical degree curriculum. The main section of the workshop will be interactive. Participants will conduct a SWOT analysis to identify strengths, weaknesses, opportunities and threats related to the previously exposed proposals. Relevance to diversity: The integration of Cochrane activities in the university represents a unique opportunity to involve different stakeholders to support informed decisions in healthcare.

Background: Clinicians, the public (including patients), policymakers, payers, regulators, and science communities invest considerable amounts of resources in making decisions at various levels using systematic reviews, health technology assessments (HTA), guideline recommendations, coverage decisions, selection of essential medicines and diagnostics, quality improvement, and policy and evidence briefs. With the World Health Organization (WHO), we have developed a conceptual approach that we call “Theory of Everything (ToE) in health decision-making” to connect the actors in this ecosystem. Our work, focusing on high-income countries, shows that criteria and methods that these actors use differ but that they frequently overlap and exist together (Schünemann et al., Lancet PH 2022). The widely used Evidence to Decision (EtD) frameworks form the connecting framework for our conceptual approach (Figure 1). This work has received broad attention, in particular by regulators, and in the guideline, HTA, and health policy community. These actors need to be linked together, and capacity is required to create interaction and decision consensus in and across jurisdictions. However, particularly in emerging economies and low and middle income countries (LMIC), the risk of sparse, redundant or conflicting decisions across the decision ecosystem may be costly. In particular, without coordination, actions by different stakeholders might contribute to decision instability and volatile national health policies.
Objectives: We are evaluating the flexibility, gradualism, and completeness of the ToE through consideration of new perspectives and exploration of its applicability at LMIC level to understand barriers-enablers and necessary prerequisites.
Methods: We are using a mixed-methods approach including surveys of existing and new stakeholders who play a role in the ecosystem, including regulators and industry, in high-income countries and LMIC, a Delphi approach, and actual work with countries to gain further insights regarding the applicability of the approach. Anticipated results and conclusions: At the Cochrane Colloquium we will report on 1) the update and expansion of the ToE; 2) case examples for countries of connecting the decision-making actors, and 3) a draft blueprint and roadmap for operationalization of the ToE in various settings.

ToE slide original.pdf

Objectives: In preparation of a clinical trial, the existing evidence on the topic needs to be systematically reviewed to avoid research waste and to provide a rationale for the planned trial. If a current systematic review (SR) is not available, a new one should be conducted. However, such an evidence-based research approach is rarely used, and no guidance for clinical researchers exists. Therefore, we aimed 1) to develop a practical guide for clinical researchers to rapidly review the literature before starting a new trial and 2) to conduct formal usability testing of the guidance.
Methods: We developed the guidance for clinical researchers in several steps: 1) literature search for existing rapid review (RR) methods guidance, guidance on conducting literature reviews specifically for clinical research, and relevant methods research; 2) literature selection including a relevancy rating; 3) data extraction; 4) consultation with our stakeholder panel on the first draft; 5) usability testing of the first interim version; and 6) final version of the guidance.
Results: We screened 602 documents and included 24 existing guidances and 29 methods studies. We extracted and synthesized recommendations for each literature review domain (i.e., research question, eligibility criteria, screening, etc.). Our interim guidance consists of two documents—the practical guide listing essential and optional tasks, explanations, and recommended tools, and a template with signaling questions, which allows users to efficiently complete and document the work process. The suggested approach consists of two parts: Part A “Evidence from existing SRs and ongoing trials” and Part B “Evidence from primary studies.” Decision points for “How to proceed from here” with several possible scenarios are presented.
Conclusions: To make clinical research more evidence based, we developed a systematic approach for clinical researchers to rapidly review the literature when planning a new clinical trial.
Patient, public and/or healthcare consumer involvement: The interim guidance and template will undergo formal usability testing, where the main target group of the guide—researchers in the planning phase of a clinical trial—will apply the guide, use the template, and provide feedback thereafter.

Background: Proportion meta-analyses are frequently used in epidemiology to estimate the burden of disease. They are usually based on transformed proportions using Freeman‐Tukey double arcsine transformations (FTT). Schwarzer et al. generated some controversy because they considered that this method produces seriously misleading results and proposed the generalized linear mixed models (GLMM) as a more elaborate approach.[1] However, Suhail et al., using the same set of studies, reanalyzed the data and concluded that the FTT is the most reliable approach and remains the preferred transformation in proportion meta-analysis.[2]
Objectives: To compare the reliability and robustness of FTT with GLMM in a large set of proportion meta-analyses.
Results: Results will be shown at the colloquium.
Methods: We will conduct GLMM[1] and FTT over a large set of proportions from a living systematic review and meta-analysis about safety, immunogenicity, and effectiveness of COVID-19 vaccines for pregnant people (https://safeinpregnancy.org/lsr/) applying recommended safeguards[2]: a) avoiding the use of the average of the double arcsine and its variance for synthesis; b) using the inverse of the variance of the pooled FTT proportion; and c) modifying the confidence intervals to prevent numerical inaccuracies.
Conclusions: It is important to verify the best approach to undertake proportions meta-analysis, which is critical for estimations in epidemiology and decision-making. References 1. Schwarzer G, Chemaitelly H, Abu-Raddad LJ, Rücker G. Seriously misleading results using inverse of Freeman-Tukey double arcsine transformation in meta-analysis of single proportions. Res Synth Methods. 2019;10: 476–483. 2. Doi SA, Xu C. The Freeman-Tukey double arcsine transformation for the meta-analysis of proportions: Recent criticisms were seriously misleading. J Evid Based Med. 2021;14: 259–261.

Background: Previous studies demonstrated limitations in the reporting of time-to-event (TTE) analyses in trial publications. The challenges that systematic review authors face in trials they include for TTE meta-analysis remain unexplored.
Objectives: To explore the characteristics, reporting and methods of trials included in aggregate data TTE meta-analyses of Cochrane and non-Cochrane reviews.
Methods: We extracted data from trials included in pairwise, hazard ratio (HR)–based meta-analyses of primary outcomes and overall survival/all-cause mortality of 50 systematic reviews. Reviews were identified from the Cochrane Database and Core Clinical Journals through systematic searches (02/2020; MEDLINE). For reviews, trials and their outcomes, we extracted, in duplicate, data on review/trial characteristics, outcome definitions, general and TTE analyses and specific characteristics relevant to TTE analysis.
Results: Reviews included 235 trials in eligible meta-analyses, resulting in 315 individual trial analyses. Most prominently assessed in trials (91%; 214/235) and reviews (88%; 44/50) was all-cause mortality/overall survival. Outcome definitions (61%; 132/315), censoring reasons (41%; 130/ 315) and follow-up specifications (56%; 175/ 315) for trial outcomes were frequently missing. Available TTE outcome data per trial were most frequently an HR combined with a log-rank p-value, survival curves and either time-point specific (16%; 50/315) or median survival times (15%; 47/315). The most prominently used individual measures were survival curves (83%; 263/315). HRs were reported for 76% (240/315) of trial outcomes. Review authors most frequently used reported HRs or recalculated TTE data from p-values (each 5%; 15/ 315), although TTE data sources were specified sporadically for individual trial outcomes. Reviews included most frequently intention-to-treat (64%; 202/315) and unadjusted analyses (25%; 80/315) but often did not specify either. Except for missing outcome data, for which numerical data were available in 57% (134/235), TTE relevant study characteristics, e.g., informative censoring and proportional hazards, were sporadically addressed in trial publications. Direct comparison to the reporting of their including reviews indicated that limitations and variability of trial reporting translates to the review level as well.
Conclusions: Reporting of trials included in TTE meta-analyses of recent systematic reviews appears variable and limited, similar to their including reviews.
Patient, public and/or healthcare consumer involvement: Not applicable.

Background: In 2017, Cochrane Sweden was launched, and a collaboration between the Medical Faculty, the Medical Degree (MD) Program and the Faculty Library was initiated based on a need for guidance and relevant learning activities for students, PhD students and researchers in systematic review methodology. The MD Program had defined scientific scholarship and evidence-based medicine (EBM) as core qualifications. The library and MD Program already worked together, and involving Cochrane Sweden became essential.
Objectives: To collaboratively develop relevant learning activities, support and guidance for carrying out systematic reviews using Cochrane tools and methodology for our students, PhD students’ teachers, researchers and clinicians.
Methods: Relevant areas were identified where Cochrane tools could be implemented as stand-alone tools or integrated with other learning activities for our user groups. Identifying methods for support and methodological guidance were key elements in the implementation. Cochrane Interactive Learning (CIL), Covidence and RevMan Web (RMW) were consecutively implemented to meet the objectives. [Table 1- Overview of Cochrane tools implemented at Lund University, Faculty of Medicine]
Results: The process of implementing, evaluating and improving the use of Cochrane tools for our user groups extends over many years. Cochrane tools have been implemented in the PhD program, MD program and the Master programs in Medical Science, Psychology and Public Health. The usage statistics and feedback on Cochrane tools show that they are appreciated and help users’ understanding of review method and workflow. The number of Cochrane Reviews, other systematic reviews and reviews where students and researchers collaborate are increasing. The library’s systematic review service has been expanded to include tools and methodological guidance in the review process. The “Lund model” has also become an inspiring example for other MD programs.
Conclusions: Our collaboration and integrative approach have resulted in an infrastructure of Cochrane tools, people and competencies supporting systematic reviews and EBM. The synergies between the collaborating parts made the implementation of Cochrane tools possible.
Patient, public and/or healthcare consumer involvement: Our students, teachers, clinicians and researchers are given a solid foundation in EBM methodology, which has a strong potential to have a positive impact on patient care.

Table 1 Overview Cochrane Tools Lund University.pdf

Background: The COVID-19 pandemic presented a major disruption worldwide. Given the extensive, rapidly evolving evidence, high-quality evidence syntheses that provide context-sensitive and up-to-date data were urgently needed as a basis for evidence-informed clinical and public health decision-making. CEOsys (‘COVID-19 evidence ecosystem’) was established as a national German network for living systematic reviews (LSRs) and living guidelines.
Objectives: To describe the infrastructure, processes and impact of CEOsys and to discuss challenges faced and lessons learned when organizing and maintaining the network, which may provide guidance for future networks in view of pandemics and health-related crises.
Methods: We present an experience report from representatives of the former CEOsys network (09/2020-12/2021; full list of collaborators: www.covid-evidenz.de), a project within the German Network of University Medicine (NUM). After describing its development, we outline CEOsys work processes and present our outputs before highlighting challenges encountered, adaptations made and lessons learned.
Results: CEOsys gathered 18 university hospitals and 8 (non-)university partners. Processes included i.a. prioritization, conducting LSRs, supporting evidence-based guidelines and knowledge translation, backed by setting up a technical infrastructure and capacity-building. We published 12 COVID-19-related Cochrane reviews, supported three living guidelines (inpatient/outpatient treatment, school-based infection prevention) and developed 10 methodological approaches and 10 dissemination formats. Challenges included CEOsys’ late initiation, prioritization, the continuously evolving knowledge on COVID-19 and revision of outcomes for LSRs or establishing a ‘living’ technical infrastructure. Based on lessons learned, the cooperation between clinicians and methodologists within CEOsys and the network’s international collaborative partners (e.g., Cochrane International) represented major advantages. Mediated by the Institute for Medical Knowledge Management, CEOsys greatly benefitted from collaborations with guideline groups. Future networks should, for example, involve key stakeholders early on, aim for (inter-)national collaborations, balance traditional publishing of results and living documents and evaluate their impact on decision-makers and the public.
Conclusions: CEOsys achieved rapid, high-quality and up-to-date evidence syntheses, collaboratively produced across multiple sites and on a broad range of issues. Our experiences are now taken up by the follow-up project PREPARED (‘PREparedness and PAndemic REsponse in Germany’), which uses the infrastructure and processes established to design a blueprint for a more sustainable evidence ecosystem.

Background: Rapid systematic reviews (RRs) are a useful tool to provide healthcare decision-makers with the most up-to-date scientific evidence. Yet there remains much uncertainty about the most effective and efficient methods for carrying out RRs. In fact, there is even uncertainty about which are the main methodological concerns for RRs.
Objectives: To elicit a consensus from RR researchers on what they consider the most important methodological questions when conducting RRs in time-efficient ways.
Methods: People with experience in conducting and using evidence syntheses (researchers, knowledge users, policymakers, and healthcare providers) were invited to participate in an eDelphi study. A core group of experts generated an initial list of questions based on the available literature. Participants were then asked to rate and rank the importance of the methodological questions. Three survey rounds were performed to achieve a consensus. Items deemed of low importance were removed at each round. Items rated to be of high importance by ≥75% of participants were included in the final list.
Results: Each survey round elicited feedback from 52 to 70 experts. Of participants, 78% self-identified as researchers, 13% as healthcare practitioners, and 9% as policymakers. More than 30% of participants reported 15+ years of experience in evidence synthesis. The first survey round started with 29 items, and new items were suggested. In the second round, 26 items remained on the list. In the final round, only 7 Items were believed to be of high importance by ≥75% of participants. Three items concerned the search strategy, two focused on study selection, and two focused on quality/bias assessment. Conclusion: Seven items were consistently identified as highly important methodological concerns related to RRs. A consensus meeting will be held to discuss additional items and to generate a summary document containing the final priority list. The results will be used to inform further research in this area, contributing to the development of reliable and rigorous RRs. Patient, public or healthcare consumer involvement: Given the emphasis on the methodological aspects of RRs, patients were not included. The eDelphi does include interested parties, such as guideline and policy developers and end-users.

Background: The interrupted time series (ITS) design is commonly used to evaluate large-scale policy change or public health interventions when randomisation is infeasible. In ITS studies, measurements are collected at regular intervals before and after an interruption. The pre-interruption period is used to estimate an underlying time trend that, when projected into the post-interruption period, creates a counterfactual for what would have occurred without the interruption. The impact of the interruption can then be quantified using a variety of metrics such as immediate and long-term effects. Several statistical methods are available for the analysis and meta-analysis of ITS studies. However, there has been no empirical evaluation of the impact of using different statistical methods to analyse ITS studies and meta-analyse their results.
Objectives: To empirically compare meta-analysis results obtained from different meta-analysis and ITS analysis methods when applied to real-world ITS data.
Methods: ITS datasets were sourced from published meta-analyses and reanalysed using two ITS estimation methods. The level- and slope-change effect estimates were calculated and combined using fixed-effect and four random-effects meta-analysis methods. We compared the meta-analytic effect estimates, 95% confidence intervals, p-values and estimates of heterogeneity across the statistical method combinations.
Results: Of an eligible 40 reviews, data from 17 meta-analyses (including 283 ITS studies) were obtained and analysed. We found that the meta-analysis method choice did not systematically impact the meta-analytic effect estimates, standard errors and between-study variances, irrespective of the ITS analysis method. However, the meta-analytic confidence intervals and p-values were impacted by the meta-analysis confidence interval method, and the ITS analysis method used may modify this impact.
Conclusions: The effect estimates, standard errors and between-study variance estimates were minimally impacted by ITS analysis and meta-analysis method choice. However, confidence intervals and p-values varied depending on the statistical method used, which may impact the interpretation of a meta-analysis. In conjunction with evidence from numerical simulation, this study provides insights into which methods to use in different scenarios, which may assist researchers in evidence synthesis of public health or policy interventions.
Patient, public and/or healthcare consumer involvement: No patients/consumers were involved in the design/reporting of this study.

Background: The success of the global evidence-based healthcare (EBHC) agenda depends on individuals and organisations working together within a functioning evidence ecosystem. Collaborative evidence networks like JBI, Cochrane, Campbell and others, although differing in form and function, were all established with a common goal: to collaboratively produce, summarise and disseminate evidence to inform policy and practice to improve lives. Over the past 30 years, with the rise of digital technology, increasing recognition of the importance of evidence-based decision-making and the evolution of network structures, these networks have become central to the way evidence is synthesised, translated and implemented, yet little has been written about their strategic development or management.
Objectives: To explore the strategic functions and form of the JBI Collaboration to understand the role of a collaborative evidence network in promoting and supporting EBHC globally.
Methods: The network functions approach (NFA) was employed to map the form (governance, management and administrative structures) and core functions of the JBI Collaboration.
Results: The results of the NFA mapping established the ways the JBI Collaboration enables development, exchange and dissemination of knowledge; the purposeful building of social capital; mobilisation of resources; and amplification and advocacy of members’ work to increase the capacity and effectiveness of members in achieving their unified purpose. It demonstrated how participation in the network enables members to contribute to and draw on collective expertise; provides access to platforms and processes that broaden the dissemination and impact of members’ work; provides access to scientific resources, training and opportunities for individual/institutional capacity building; and provides practical knowledge of how to solve complex health problems using JBI’s model of EBHC.
Conclusions: Collaborative evidence networks play a crucial role in bringing together diverse stakeholders from across the evidence ecosystem to drive the EBHC agenda forward. These networks combine formal and informal structures to facilitate collaboration to address complex global health issues. There would be value in cognate networks using the NFA to identify the areas of overlap, working together to systematically reduce duplication of effort and strengthen our understanding of how we can collaborate across networks to achieve impact in EBHC globally.

Background: Because of the growing need to provide evidence syntheses under time constraints, recent research has explored rapid review methods, which often employ single-reviewer literature screening. However, the single-reviewer screening process is error-prone; on average, 13% of relevant studies are missed. To date, it is unclear whether certain types of studies or publications have a higher risk of being falsely excluded than others.
Objectives: The aim of our methods study with an observational design was to identify risk factors on a study-, abstract-, journal-, and reviewer-level for eligible studies to be falsely excluded during single-reviewer abstract screening.
Methods: We used a database of 1,000 inclusion and exclusion decisions pertaining to 80 eligible references from a crowd-based randomized controlled trial assessing the accuracy of single-reviewer abstract screening. We gathered a list of potential risk factors for studies falsely excluded during single-reviewer abstract screening. One investigator collected potential risk factor variables independently, and a second investigator checked the data. We built a random forest model using R version 4.2.2 to identify variables that could predict the risk of false exclusion of studies. We split up the dataset into a training and a validation set. We trained the random forest classifier with the training set and used it to predict classes of “single false exclusion” for the validation set. We calculated Mean Decrease Accuracy and Mean Decrease Gini to rank the importance of the collected variables.
Results: The random forest model ranked study design and the Scimago Journal & Country Rank (SJR) of the publishing journal as the variables with the highest importance for false exclusions. Cross-sectional studies, controlled-before-after studies, and publications in journals with an SJR ≤ 2.29 had a higher risk of being falsely excluded. Other variables, such as the screener being a native speaker, self-rated topic expertise, and previous review screening experience, were ranked as the least important. The model achieved a good area under the curve at 0.71.
Conclusions: Our results indicate that study design and a low SJR of the journal may be important risk factors for studies to be falsely excluded during single-reviewer abstract screening. Investigators need to be cognizant of these risk factors when conducting reviews that include non-randomized evidence. Patient or healthcare consumer involvement: Rapid reviews generally involve patients and healthcare providers in the review process in order to focus on patient-relevant health outcomes.

Background: Interrupted time series (ITS) studies are frequently used to examine the impact of population-level interventions. Systematic reviews with meta-analyses including ITS designs may inform public health and policy decision-making. Reanalysis of ITS may be required for inclusion in meta-analysis. Although publications of ITS rarely provide raw data for reanalysis, graphs are often included, from which time series data can be digitally extracted. However, the accuracy of effect estimates calculated from data digitally extracted from ITS graphs is currently unknown.
Objectives: To assess the accuracy of effect estimates calculated from digitally extracted data from ITS graphs.
Methods: Forty-three ITS with available datasets and time series graphs were included. Time series data from each graph were extracted independently by four researchers using digital data extraction software. Data extraction errors were analysed. Segmented linear regression models were fitted to the extracted and provided datasets, from which estimates of immediate level and slope change (and associated statistics) were calculated and compared across the datasets. These estimates were then standardised for comparison across the datasets. For a particular dataset, standardisation was achieved by dividing the level and slope change estimates by the range of the outcome of the provided time series data (i.e., the maximum observed value of the outcome minus the minimum observed value); this yielded standardised effect estimates that could range from 0 to 1.
Results: Although there were some data extraction errors of time points, primarily due to complications in the original graphs, they did not translate into important differences in the standardised level and slope change effect estimates (and associated statistics) (Figures 1 and 2).
Conclusions: Using digital data extraction to obtain data from ITS graphs should be considered in reviews including ITS. Including these studies in meta-analyses, even with slight inaccuracy, is likely to outweigh the loss of information from noninclusion.
Patient, public and/or healthcare consumer involvement: There was no involvement with patient, public and/or healthcare consumers.

Figure_1_box_plot_level.png
Figure_2_box_plot_slope.png

Background: Evidence Synthesis Ireland (ESI), which includes Cochrane Ireland, aims to build evidence synthesis knowledge, awareness and capacity among the public, healthcare institutions, policymakers, clinicians and researchers on the island of Ireland. One way we do this is through the ESI Fellowship Scheme, launched in January 2019.
Objectives: The scheme builds capacity in Fellows through developing the practical skills of how to plan, design, conduct and report an evidence synthesis. Fellows receive mentoring from evidence synthesis experts and contribute to reviews that have a strong policy and/or practice impact. Process: The scheme places Fellows virtually with experienced review teams nationally and internationally to gain hands-on evidence synthesis experience in a high-quality environment with one-to-one mentorship. Fellowships range from 6 months to 2 years and are unpaid, allowing those working full-time to apply. Benefits include free attendance at ESI workshops, co-authorship, €1000 travel bursary, access to Covidence, and networking/collaboration opportunities. Eligible applicants are clinicians/trainees, researchers, academics and/or postgraduate residents of Ireland/Northern Ireland. Fellows are selected through a competitive process and are expected to commit to 8 hours/week to the review in addition to theoretical training.
Results: Fellows are placed on a range of review types: 50% on Cochrane reviews (reviews of effectiveness, qualitative evidence synthesis, rapid reviews, living reviews and diagnostic test accuracy) and 50% on other reviews (umbrella, overviews, scoping reviews, prognostic reviews, evidence and gap maps and mixed methods). To date, 68 Fellows have been placed. Fellows are mostly female (64%), over half have never completed a systematic review and 35% are early career. Fellows are academics (30%), healthcare professionals (35%), postdoctoral researchers (24%), clinician-academics (6%), policymakers (6%) and others (9%). Fellows report capacity built-in evidence synthesis competencies and illustrate this capacity through leading further reviews, collaborating with mentors and teaching evidence synthesis.
Conclusions: ESI provides a unique opportunity for researchers to develop their evidence synthesis skills and contribute to evidence-based healthcare in Ireland. This model could be replicated by others to strengthen capacity in producing robust evidence. Patient relevance: The scheme is relevant to patients as it contributes to strengthening the capacity of researchers/clinicians to produce robust evidence.

Background: Policy and practice decisions must be evidence-based, but ensuring that research evidence is readily available and used by decision-makers is challenging. This was exacerbated during the COVID-19 pandemic, when knowledge mobilisation strategies to bridge the gap between evidence synthesis and informing decisions were critical. The Wales COVID-19 Evidence Centre (WCEC) was set up to ensure that health and social care policy and practice decision-makers had timely access to relevant COVID-19 research evidence.
Objectives: To describe WCEC knowledge mobilisation methods for rapid evidence reviews, reflect on the impact, and share learning.
Methods: Knowledge mobilisation methods were flexible to meet decision-makers’ (our stakeholders’) needs. We collaborated with stakeholders throughout to ensure we addressed their requirements and identified policy and practice implications, evidence impact and develop knowledge mobilisation plans. We identified barriers and facilitators to improving our methods over time and ensure all outputs are widely accessible.
Results: Coproduction with stakeholders (priority-setting, review conduct, planning pathway-to-impact and knowledge mobilisation) built trust in our methods and evidence reliability. Reviews informed Welsh health and care decision-making during the pandemic and recovery periods, addressing current health and care pressures. From April 2021 to December 2022, 20 of our reviews informed Welsh Government guidance, moving to COVID alert level 0, schools reopening, a Public Health Wales campaign to raise vaccine uptake, and led to Welsh Government commissioning of carbon dioxide monitors in schools. A review of strategies to support learning and wellbeing among 16- to 19-year-old learners was a key piece of evidence included in the Welsh Government’s ‘Renew and Reform: Post-16 and Transitions Plan’, and reviews on inequalities that were exacerbated during the pandemic (LGBTQ+, women and girls, people living with disabilities) informed a Welsh Government action plan. Recently, a review of the effectiveness of diagnostics centres is guiding the design and implementation of new community diagnostic centres in Wales (January 2023).
Conclusions: Coproduction throughout supported knowledge mobilisation and led to the use of rapid reviews by scientific advisors and policy and practice decision-makers. Patient, public, and/or healthcare consumer involvement: The WCEC had a public partnership group of 8 members who contributed to all our processes.

Background: Conventional systematic review (SR) methods are time-consuming and highly resource intensive. Artificial intelligence (AI) algorithms such as machine learning and deep learning can help reviewers complete these tasks in less time and with fewer resources. PICO Portal (PP) is an AI-assisted SR platform that prioritizes articles for screening using several algorithms including both decision tree and deep learning models.
Objectives: To assess the AI-assisted screening in PICO Portal.
Methods: Our data set comprised eight completed SRs, each using two independent screeners, with a total of 56,728 records (range: 4,204 to 14,185) on a range of topics from social to biomedical sciences. For each SR, we simulated the screening using batches of 100 articles to train and build predictions for eligibility, re-ranking successive articles, and comparing the predicted eligibility with the actual results from the SRs. We plotted the proportions of title/abstract and full-text included records that were captured by the AI screening at the title and abstract level for each project and calculated a weighted average of this efficiency (by project size). We meta-analyzed the sensitivity and specificity of the predictions versus the reviewers’ final decisions using Stata ‘metadta’.
Results: We estimate that if the active learning AI predictions had been used, reviewers would have needed to screen only 20-50% of title/abstracts to capture 95% of eligible records (Figure 1). After screening 10%, 25%, 50%, and 70% of title/abstract records, the average project would have captured approximately 60%, 85%, 95%, and 99% of the records included in the title/abstract stage (Figure 2). Sensitivity was better than specificity (95% vs. 68%) (Figures 3 and 4).
Conclusions: Based on our analysis, we estimate that 40-60% of screening effort can be saved using PICO Portal, an AI-assisted, web-based, SR platform. Future research should examine the impact of missing the final 5% of records on review conclusions and assess the resource-benefit ratio. Patient relevance and involvement: Our findings and future recommendations are from the researcher and funder perspective. Our conclusions directly impact the amount of time reviewers need to complete an SR. This work did not involve any stakeholders, patients, or consumers.

Figure 1a & 1b.png
Figure 2.png
FIgure 3.png
Figure 4.png

Background: In health technology assessments (HTAs), matching-adjusted indirect comparisons (MAICs) are used when head-to-head randomised studies comparing a drug (therapy) in question and a comparator, e.g., standard care in the treatment of a disease, are not available. MAICs use individual patient data (IPD) from trials of one treatment to match baseline summary statistics reported from trials of another treatment. MAICs use an approach similar to propensity score weighting, whereby treatment outcomes are compared across balanced trial populations. Although the Grading of Recommendations Assessment, Development and Evaluation (GRADE) and Confidence in Network Meta-Analysis (CINeMA) approaches to rate the quality of treatment effect estimates from network meta-analysis (NMA) have been suggested, it seems that an approach for MAICs is missing.
Objectives: a) To evaluate the prevalence of MAIC use in submissions to the National Institute for Health and Care Excellence (NICE); and b) to explore how to rate the quality/certainty of the evidence in MAIC using the currently available approaches.
Methods: Scoping searches of NICE website (without time restrictions) were conducted, and these will be supplemented with searches in Medline, Embase and Central. Prevalence data will be synthesised quantitatively. The existing GRADE and CINeMA approaches will be compared accounting for similarities and differences in the MAIC and Bucher methods.
Results: Preliminary findings suggest that MAICs are predominantly being used for reimbursement decisions in oncology. Worryingly, a large proportion of submissions to NICE rely on unanchored comparisons whereby a common comparator arm is missing, and these types of MAICs make much stronger assumptions and are widely regarded as unfeasible. The work is still ongoing; we will have more findings by September 2023 to present at the Colloquium.
Conclusions: Currently there is no guidance on how to rate the certainty of evidence of effect estimates obtained from MAICs. Although MAIC and Bucher methods share certain similarities, there are still distinct differences between the two. We believe a new approach is warranted, and determining certainty of evidence in MAICs may help decision-makers in making informed recommendations.

Background: Shortly after the launch of Cochrane Rehabilitation in December 2016, the World Health Organization (WHO) launched “Rehabilitation 2030: a call for action” with the aim to strengthen and upscale rehabilitation services in health systems worldwide in response to the growing needs. Cochrane Rehabilitation collaborated with the WHO Rehabilitation Programme in several projects on rehabilitation interventions, in the framework of Rehabilitation 2030 and the COVID-19 pandemic.
Objectives: The aim was to provide methodological support and Cochrane evidence at the request of the WHO, to inform clinical recommendations and guidance on rehabilitation management.
Methods: Different methods have been used according to the needs of the specific projects. For the Package of Interventions for Rehabilitation (PIR), developed for 20 selected health conditions, we performed an overview of all the CSRs according to the inclusion criteria defined with WHO. The CSRs identified during the screening process were summarized using an evidence map. We grouped outcomes and comparisons of included CSRs indicating the effect and the quality of evidence to provide a comprehensive view of what is known. Regarding the COVID-19 pandemic, we created an interactive living evidence map, based on priorities for research defined in collaboration with the WHO rehabilitation programme: limitations of functioning (disability) of rehabilitation interest, the phases of the disease, and research questions at the rehabilitation micro, meso and macro level. Furthermore, indirect evidence on symptoms of post COVID-19 condition was extracted and synthesized from CSRs on the same symptoms in other health conditions.
Results: Table 1 reports the results on the three main projects (PIR, ‘Evidence on’ COVID-19 and ‘Evidence relevant to’ symptoms of post COVID-19 condition) in collaboration between the WHO and Cochrane Rehabilitation.
Conclusions: In its collaboration with the WHO, Cochrane Rehabilitation has developed different strategies and methodologies to respond rapidly to emerging questions regarding rehabilitation interventions. The aim is to inform clinicians as well as policymakers to allow evidence-based decision-making. These approaches could benefit other similar fields in healthcare in the future and inform scientists on further research needs.
Patient, public and/or healthcare consumer involvement: Not applicable.

CR WHO Collaboration Table 1.pdf

Background: During the COVID-19 pandemic, a large increase in rapid reviews (RRs) was identified. This is mainly because the design appears as an alternative to provide timely and rapid responses to emerging issues. RRs follow the method of systematic reviews, but they simplify or omit some steps to shorten the length of time in which the review is conducted.
Objectives: To identify and describe which steps were performed and which were omitted in published RRs, according to the list of recommendations suggested by the Cochrane Methods Group.
Methods: We searched databases (MEDLINE, Embase, Epistemonikos, and CINAHL) to identify RRs of interventions for COVID-19. Two investigators independently screened and selected reviews that self-reported as RRs and that at least delivered a meta-analysis. We will extract the data related to the methodology used by each RR and compare them with the recommended steps by classifying them as “reported” or “omitted.” We will analyze compliance for each study and for the totality of the included studies. To chart compliance, we will classify it as “very high” when compliance ranges from 76% to 100% (100% corresponds to reporting for the eight recommendations), “moderate” when between 51% and 75%, “low” when between 26% and 50%, and “very low” when between 0% and 25%.
Results: This is a work in progress. The results will describe the performance of the method in a certain group of RRs. We will present a summary table of the Cochrane Methods recommendations for RRs, identifying whether there was reporting or omission of the step for each recommendation.
Conclusions: RRs are being used for decision-making both for everyday health and for public health policies, so it is essential that they are performed with a clear and specific method. We hope that the results of this analysis can inform whether an adequate method is being followed when carrying out RRs.
Patient, public and/or healthcare consumer involvement: This study does not involve direct participation of patients or consumers.

Background: Absolute risk reduction or risk difference (RD) is a key effect measure required for decision-making and its confidence interval (CI) is the basis for imprecision judgments. Many methodology groups (e.g., Cochrane and GRADE) recommend obtaining RD from linear transformation of a risk ratio (RR) that is usually derived from a meta-analysis. This transformation uses an assumed baseline risk (BR) and follows the equation RD= RR X (RR-1). The 95% CI of RD is derived from the same equation using the 95% CI of RR.
Objectives: In this proposal, we demonstrate several limitations to this traditional approach using a simulated case study and offer an alternative approach.
Methods: We simulated a case study using a published systematic review [beta blockers vs. placebo on all-cause mortality in symptomatic heart failure with preserved ejection fraction, RR, 0.79 (0.66-0.96)]. The first analysis was based on the traditional linear transformation of RR into RD using BR that we simulated from 0% to 100%. The second analysis is based on the proposed new approach which is a microsimulation in which RR is drawn from a Lognormal distribution with mean and standard error of the identified RR and BR is drawn from a Beta distribution which shape parameters were derived from a population-based study [Olmsted County, Minnesota, Mortality at 1 year= 0.29 (0.27-0.31)]. 10,000 simulations (draws) generated RD histogram with median, 2.5 and 97.5 percentiles (Open-source code in R).
Results: The traditional approach (Figure 1) demonstrates these limitations: 1) RD CI does not incorporate uncertainty in BR and derives all its uncertainty from the treatment effect, 2) CI widens linearly as BR increases, making RD estimates imprecise in higher-risk populations (counterintuitive), and very precise in low-risk populations (potentially misleadingly precise as it ignores uncertainty in BR). The proposed approach (Figure 2), produces a joint distribution of RD that incorporates uncertainty in BR to resolve the limitations of the first approach.
Conclusions: Simple transformation of RR into RD has many limitations and does not incorporate uncertainty in BR. An alternative proposed approach incorporates uncertainty in BR that was derived from a population-based study. Patient/public/consumer involvement: None.

cochrane RD figures.pdf

Background and objectives: The overall aim of the training school (TS) is to introduce to the participants the concept of Evidence Based Research (EBR) and support their ability to use existing evidence synthesis to justify, design and place results of the new study in the context of existing knowledge in order to avoid redundant research.
Methods: The TS is targeted at Early Career Investigators and senior Clinical Health Researchers who are interested in how to perform EBR. Following several working meetings of the group that has knowledge of existing courses and training in Systematic reviews, Evidence-Based Medicine/Practice/Healthcare, etc., from various organisations such as Cochrane, JBI, GRADE, etc., and by analysis of the content of their curricula and by an iterative discussion in the group, pilot EBR TS were designed, which were tested in Tartu in 2019 and then evolved to synchronous on-line teaching and in the end were developed as an asynchronous on-line course on the Moodle and Canvas platforms. The evaluation form for participants was designed and applied, and feedback from teachers and facilitators of TS plus feedback from EVidence Based RESearch - EVBRES COST Action Programme Management Group and Core Group were received and analyzed with every new version of TS.
Results: The curriculum and programme of TS were designed and tested. We taught the TS seven times to over 150 successful participants who finished the whole TS. TS is divided into 14 modules. Each module has its own specific interactive content. The overall rating of TS by our participants was 4.34 on a scale one (poor) to five (Excellent).
Conclusions: The development of the EBR TS is a significant step in increasing awareness around EBR among researchers, raising the importance of the development of trustworthy and important evidence synthesis and primary research in the context of existing evidence and thus preventing research waste. Consumer involvement: Several colleagues originally inexperienced in EBR or Evidence-Based Practice education were involved in the design of the TS, so the TS development was co-informed by their needs in EBR knowledge and skills.

Background: The eCOVID19 Recommendations Map and Gateway to Contextualization (RecMap) is a living platform presenting 7280 guideline recommendations. Funded by the Canadian Institute of Health Research (CIHR), RecMap involves the collaboration of over 40 partner organisations globally and has launched a knowledge mobilisation (KM) project to benefit wider stakeholders in different countries.
Objectives: To increase the awareness, use, and engagement of the RecMap among guideline developers in China and to identify barriers encountered in the KM activities.
Methods: We used a co-design framework to guide our activities, where the RecMap central team, China team leads, voice of local stakeholders and public, and the context of China COVID-19 policies were considered in planning and iterating the activities. These activities include group discussion, local stakeholder consultation, guideline adolopment (contextualise and adopt recommendations from existing guidelines), plain language recommendation (PLR) translation, webinars, and social media release. We ensured an inclusive participation of age and gender, guideline developers from both clinical and research background, and public involvement.
Results: We collaborated with China GRADE Centres, Cochrane China Network Affiliates, and Cochrane Campbell Global Ageing Partnership. We conducted 10 discussions and consultations with experts and held webinars with 1.4 million views. We found that local policies and language are key barriers for introducing and implementing RecMap recommendations in China. The restricted and unpredictable China COVID-19 policies in the past months have hindered our guideline adolopment progress, so it is important to test the feasibility of KM activity with local stakeholders at an early stage to avoid waste of time and resources. To encourage more Chinese to visit the English RecMap website, we prioritised translation of 40 PLRs into Chinese. After identifying mistakes in PLRs translated by volunteers, we established a workflow with professional translator, methodology editor, clinical editor, and citizen editor to co-produce Chinese PLRs to ensure quality and readability.
Conclusions: The rapid global health challenges require us to establish a collaborative platform and streamline the collaboration process for each stakeholder spending less time and fewer resources on negotiating the mechanisms to achieve more efficient, effective, and equitable partnership outcomes. Public involvement: Citizen editor was involved in PLR development.

Objectives: The Cochrane Rapid Review Methods Group (RRMG) developed interim guidance to support authors in conducting rapid reviews (RRs). The objective of this mixed-methods study was to assess the adherence to and usability of this guidance. We also explored why some Cochrane authors of COVID-19–related reviews preferred doing a full systematic review (SR) over an RR (see published protocol: https://osf.io/3a8zb).
Methods: We identified all reviews citing the Interim Cochrane RRMG guidance up to February 17, 2022, and performed an exploratory adherence analysis. We interviewed 20 RR authors to assess the comprehensibility of recommendations and reasons for any deviations. Further, we surveyed nine authors of COVID-19–related full SR.
Results: We analyzed 128 RRs (111 non-Cochrane, 17 Cochrane) citing the Interim Cochrane RRMG guidance. Several recommendations were not followed by a large proportion of RR authors, whereas in some cases, full SR methodology was used instead of recommended abbreviations. The recommendations that were not followed by the most-analyzed RRs were the stepwise approach to study design inclusion (97%), limiting the number of outcomes (88%), and peer review of at least one search strategy (88%). The rationale for using a standardized title and abstract form was not obvious to some RR authors, and 39%/45% preferred dual independent screening of abstracts/full texts over screening only excludes dually. The most reported reasons for deviating from the guidance were time constraints, unclarities in the recommended approach, or inapplicability to the specific RR. Overall, the guidance was viewed as user-friendly. However, without pre-existing experience of SR conduct, applying the guidance was perceived as difficult. The main reasons for conducting a full COVID-19–related SR over an RR despite the time pressure during the pandemic were late availability of the guidance, preset mandate to conduct an SR, lack of methodological clarity, and inapplicability to the evidence base.
Conclusions: Clarifications are warranted throughout the Interim Cochrane RRMG guidance to ensure that users with various experience levels can understand and apply its recommendations accordingly.
Patient, public and/or healthcare consumer involvement: We interviewed RRMG guidance users for feedback on its comprehensibility and usability to improve the interim guidance.

Background: Classical randomized controlled trials often meet difficulties in enrollment and compliance due to subjects’ preference for the randomization scheme. Zelen’s design can largely resolve these issues through randomizing subjects before informing stage.
Objectives: This study systematically reviewed the progress of Zelen’s design and its modifications in clinical research and clarified its methodological elements, advantages, and limitations.
Methods: A systematic literature search was conducted for Zelen’s design from the inception of databases (PubMed, Cochrane library, CNKI, WanFang, and Vip database) to February 2023. The data were extracted, including sample size, population characters, length of follow-up, Zelen’s model, group shifting, and statistical methods.
Results: One hundred twelve trials were included. The range of sample size was from 15 to 23,024. The longest follow-up length was 17.2 years. The dominant disease in this design was mental disorders (21, 18.75%), followed by osteoarthrosis diseases (16, 14.29%), cancer (11, 9.82%), cardiovascular diseases (10, 8.93%), and others. Regarding types of consent, more than half of the trials used a double-consent (57, 50.89%), 32 (28.57%) used a single-consent, and others did not report the consent procedure. Twelve trials (10.71%) used a modified Zelen’s design nested within an observational study. This design used a two-stage informed consent. Stage 1, patients were invited to participate in a cohort study; Stage 2, patients randomized in the experimental group were informed of the allocation result and inquired whether they would like to follow the treatment. Five trials used the McNulty-Zelen design, which could be applied in cluster randomized controlled trials and overcome the potential bias of the Hawthorne effect, e.g., trials involving educational interventions. Informed consent was obtained from a third party on behalf of the participants. Intention-to-treat analysis was the main population used in Zelen’s design.
Conclusions: Zelen’s design has a broad application foreground in clinical trials. It also could adapt to research needs by combining with various observational studies. Patient or healthcare consumer involvement: No patients or consumers participated in research activities.

Background: Cochrane has embarked on an important change programme to transform the way that we produce high-quality, independent and timely evidence. By 31 March 2024, the editorial process for all submitted protocols, reviews and updates will be managed centrally by the Central Editorial Service.
Objectives: To explain the rationale for and workings of the Central Editorial Service for both prospective authors and members of Cochrane entities.
Description: The Central Editorial Service currently handles the editorial process for most drafts submitted for publication in the Cochrane Database of Systematic Reviews (CDSR). This will progress to all content from 31 March 2024. The separation of the process of producing Cochrane Reviews from the editorial process to assess their suitability for publication in the Cochrane Library represents a major change for Cochrane. This talk will explain the rationale for the Central Editorial Service, and the difference between the Central Editorial Service and a Cochrane Review Group, which remains an understandable source of confusion for many Cochrane contributors. The presentation will explain the editorial process from submission, editorial assessment and peer review to the final editorial decision We will also signpost to other Colloquium workshops that authors may find helpful in preparing their Cochrane Reviews.

Colloquium 2023 Abstract - Central Editorial Service Oral Presentation.pdf

Background: It is necessary to invest in strategies that allow groups and organizations in charge of developing evidence synthesis to inform clinical or health policy decisions and effectively incorporate new emergent evidence in the development of knowledge transfer (KT) products such as clinical practice guidelines (CPG), health technology assessment (HTA), and structured evidence summaries.
Objectives: To develop and evaluate a capacity-building strategy for health sector organizations to incorporate the living evidence (LE) model in the development of KT products.
Methods: The project was developed in three phases: P1. Definition of a guidance framework; P2. Training in LE strategy and supporting tools; P3. Applying the framework and acquiring knowledge in the development of real-life LE synthesis following the learning-by-doing methodology. Study population: Technical staff working on CPG, developing organizations, HTA agencies, and tertiary hospitals running institutional HTA. The Epistemonikos-L.OVE platform and tools were used as the technological enablers for the LE process. Each component of the capacity-building strategy was evaluated through online anonymous surveys. A final set of in-depth interviews were conducted to evaluate the whole capacity-building strategy used.
Results: Phase 1: Based on methodological articles and brainstorming meetings, we developed a preliminary framework. After its testing among potential users and expert consultation, the final version of the framework was used in the next phases of the project. For phases 2 and 3, we enrolled 21 team members from a total of seven organizations. Six training workshops in the LE methodology and Epistemonikos tools were carried out. Participating organizations’ teams developed a total of seven living evidence syntheses following the framework and using Epistemonikos tools. They maintained the evidence surveillance and related monitoring tasks for at least 8 months, which were reported on a monthly basis and made available at the project website (https://livingevidenceframework.com/en/platform/). Eighty-five percent of participants took part of the online survey and 68% of in-depth interviews to evaluate the capacity-building strategy.
Conclusions: We addressed three basic needs to build a sustainable capacity among groups and organizations for the production and use of LE synthesis: the need for having guidance; for training; and for being supported during the process.
Patient, public and/or healthcare consumer involvement:

Background: During COVID-19, the public was put in danger by erroneous beliefs derived from poor-quality research amplified through social media. In the Cochrane Infectious Diseases Group (CIDG), we needed to be innovative with methods to address COVID-19 misinformation. Here, we will present this evolving response, leading to an approach to appraise research that caused unsubstantiated hypotheses. In this process, we adapted Cochrane methods to appraise comparative serological studies.
Objectives: 1. To present the CIDG strategy to address the COVID-19 misinformation pandemic. 2. To outline novel methods for critically appraising comparative serological studies.
Methods: The CIDG identified important areas of misinformation where our methods could provide an authoritative voice. This started with Cochrane Reviews of trials to help end the use of unapproved drugs for COVID-19 (chloroquine and ivermectin). Later, we identified a potentially misinformed hypothesis derived from laboratory research (“microclots” cause the post-COVID-19 syndrome [PCS]) leading to public demand for the treatment plasmapheresis. Because there were no randomized controlled trials, we rigorously appraised these laboratory studies using Cochrane principles in collaboration with laboratory scientists and thrombotic disorder specialists.
Results: Cochrane Reviews of trials of chloroquine and ivermectin challenged fake and poorly conducted science in the public narrative; some trials were excluded from the ivermectin review based on a failure to fulfill the expected ethical and scientific criteria. These reviews were highly cited and used as the gold standard to state there was no evidence these treatments worked. Conducting a Cochrane Review of plasmapheresis for PCS was problematic, as trials had not been conducted and would be expensive. So we used novel methods to appraise the comparative serological studies that generated the hypothesis that microclots were associated with PCS. We identified bias across three domains: collection and handling of samples; experimental methodology; and reporting and interpretation of results.
Conclusions: Sometimes policy needs systematic reviews that evaluate the rationale for a treatment to help justify whether a trial is worth doing. These can be laboratory studies with sera or animal studies. Cochrane principles and methods can be adapted to meet this need, but flexibility within Cochrane formats and editorial policies could facilitate this.

Background: Systematic reviews (SRs) and meta-analyses (MAs) of prediction model studies range in breadth and can aim to identify, appraise and summarise the evidence about existing models and their predictive performance. They differ in many important ways from SRs of intervention studies. Emphasis is on predictive performance of models obtained from internal or external validation, rather than effect sizes. Although TRIPOD provides guidance for reporting primary prediction model studies and PRISMA 2020 is available for SRs of interventions, there is currently no tailored guideline for reporting SRs of prediction model studies.
Objectives: To develop a tailored checklist for the transparent reporting of SRs and MAs of diagnostic and prognostic prediction model studies.
Methods: We formed an executive committee responsible for developing TRIPOD-SRMA. Reporting items from existing guidelines such as TRIPOD, PRISMA 2020 and others were used to produce the initial checklist. A modified Delphi approach was then used to elicit views from a wider group of experts with experience of SRs and primary prediction model studies. In the first round, 86 individuals were invited to participate, of whom 43 participants responded and formed the Delphi panel. Online surveys were used to inform the consensus process on which items to include and gather feedback to refine the checklist. After two Delphi rounds and consideration of all feedback, TRIPOD-SRMA was finalised and approved by all members of the executive committee.
Results: The new TRIPOD-SRMA checklist will be presented. It consists of 26 reporting items within six sections. Some reporting items remain unchanged or minimally modified from PRISMA 2020 as there are common elements to all SRs. However, 11 items have been tailored or included specifically to address SRs of prediction model studies. A checklist for abstracts is also included.
Conclusions: TRIPOD-SRMA and the accompanying checklist for abstracts provide the first reporting guideline for SRs and MAs of prediction model studies. Use of TRIPOD-SRMA by journals and authors will help to improve transparency of SRs of prediction model studies and ensure readers can more easily evaluate the evidence for existing prediction models within a clinical field.
Patient, public and/or healthcare consumer involvement: None.

Background: For ethical, logistical and cost-related reasons, studies examining interventions to prevent occurrence and transmission of infectious diseases often use atypical experimental designs. Analysis methods for these study designs may be established, but inclusion of data from these studies in systematic reviews often raises unique challenges, which methodological experts at Cochrane Infectious Diseases must overcome by determining appropriate analysis approaches.
Objectives: To present examples of atypical experimental study designs and outline how Cochrane Infectious Diseases authors have included data from these study designs in systematic reviews.
Methods: We qualitatively reviewed examples of the inclusion of atypical experimental study designs in Cochrane Infectious Diseases reviews in recent years. We extracted information on the key features of these study designs and the analysis approach used by review authors to synthesise data from these studies. A descriptive account of the main issues that were encountered, and lessons learnt are presented.
Results: We identified three key examples of inclusion of data from atypical experimental study designs in Cochrane Infectious Diseases reviews: namely, the inclusion of 1) interrupted time series studies in a review of space spraying for preventing malaria transmission; 2) experimental hut studies in a review of piperonyl butoxide combined with pyrethroids in insecticide‐treated nets to prevent malaria; and 3) a cluster-randomised, test-negative study in a review of Wolbachia-carrying Aedes mosquitoes for preventing dengue infection. Review authors used various approaches to synthesise data from these studies.
Conclusions: Even when studies with atypical experimental designs have been carefully planned, conducted and analysed, it is not always the case that study authors report effect estimates and standard errors that can be entered directly into RevMan. Review authors must occasionally calculate effect estimates that take special features of the data (for example, clustering or seasonality) into consideration. Where effect estimates and standard errors are reported by the study authors, literature presenting the development of the atypical experimental study design and accompanying analysis methods should be reviewed carefully. Collaboration between Cochrane statisticians is key to ensuring robustness of methods and results reported from reviews that include studies with atypical experimental designs.
Patient, public and/or healthcare consumer involvement: None.

Background: The use of network meta-analysis (NMA) in systematic reviews (SRs) of effects is increasing. Despite skilled researchers and biostatisticians in Sub-Saharan Africa (SSA), with access to formal training in evidence synthesis, there is limited training in NMA.
Objectives: To describe our experience with enhancing capacity for NMA in SSA.
Methods: We developed two NMA online courses. The 8-week Primer in NMA SRs helped participants to find, appraise, interpret, and consider the use of NMA while the 5-week Global NMA Masterclass facilitated conduct of NMA using Stata and R. A multidisciplinary team collaborated to develop the content and format with text-based lessons, self-assessments, embedded lectures, asynchronous discussions, and weekly synchronous Q&A sessions. We integrated HIV/TB and COVID clinical examples. Participants provided feedback and facilitators reflected on implementation.
Results: Forty-three participants (post-graduate students, clinicians, and decision-makers from various SSA countries) attended. They reported that the training was very relevant to their work. The self-directed, online learning approach enabled participants to learn at their convenience. However, some participants lacked motivation and commitment. As NMA is complex, requiring dedicated learning schedules, participants reported that they needed more time, and some would have preferred in-person learning. Using examples relevant to SSA and everyday problems encountered by clinicians made the learning more authentic.
Conclusions: These were the first courses related to NMA offered in SSA. Harnessing the benefits of online learning while being cognizant of the challenges and striving to improve on the offering is key for future training. There is great potential to conduct NMA across SSA. Patient, public, and/or healthcare consumer involvement: No direct involvement in the development and offering of our courses. However, given the significant interest in the examples used in the NMA workshop (HIV/TB and COVID), and their impact on patients’ lives and wellbeing, we would welcome public partnership in drawing attention to our findings.

Relevance and importance to patients: The aim of the Message Lab is to help improve how evidence-based public health messages are communicated, so that the messages and underlying research are informative and correctly understood by patients and the public.
Background: Well-designed public health messages can make people aware of what they can do (or not do) for their health and help them to make informed choices. On the other hand, poorly designed messages can confuse people, lead to poorly informed decisions, and diminish trust in health authorities and research.
Objectives: To set up a platform to facilitate the design and evaluation of public health messages.
Methods: Our starting point is evidence-based guidance for effective health communication, including the use of simple and familiar wording, clear visual design, and careful testing in target audiences. We will prioritise questions together with an international advisory group, focusing on important uncertainties about how best to communicate information about the effects of health interventions. These include, for example, how to communicate uncertainty about effects, the use of positive versus negative framing for different types of decisions, and the use of visual displays of intervention effects. We identified recruitment platforms that can be used for online randomized trials and platforms for user testing. We assessed these based on several criteria. We then tested selected platforms in a randomized trial comparing three ways of communicating the overall uncertainty of effects and comparing including the margin of error (confidence interval) with not including it.
Results: In this presentation, we will discuss the advantages and disadvantages of different user testing and recruitment platforms for online randomized trials, and we will present the findings of our trial of different ways of communicating the uncertainty of effects.
Conclusions: The Message Lab offers an efficient way for researchers to test different methods of communicating public health messages and for health authorities to design public health messages that are effectively communicated.
Patient, public and/or healthcare consumer involvement:

Message Lab Cochrane final.pdf

Background: The Evidence Pipeline seeks to transform study identification for Cochrane and other evidence synthesis producers (see Figure). Research enters the Evidence Pipeline and goes through tailored workflows involving crowdsourcing and machine learning, working together to produce accurate, reliable metadata about studies. The Evidence Pipeline has been applied at both the review level and at the repository level. At the review level, a workflow called Screen4Me helps to reduce author screening burden through assessment of title-abstract records; at the repository level, studies relevant for Cochrane’s Central Register of Controlled Trials (CENTRAL) or the Cochrane COVID-19 Study Register (CCSR) are identified using Crowd and machine, helping to keep these important resources up to date.
Methods: This presentation will provide a detailed look at the main component parts of the Evidence Pipeline applied at the specific review level (via S4M) and at the data repository level (CENTRAL and CCSR). It will also describe our evaluation work on each component part and on the system as a whole in comparison to traditional methods.
Results: At the time of writing, the Evidence Pipeline has been used (via the Screen4Me workflow) in over 100 Cochrane reviews. The mean reduction in screening workload for author teams is 72% with a mean time to screening task completion of ten days per review. At the repository level, 99% of RCTs in CENTRAL have been identified by the Evidence Pipeline.
Conclusions: The Cochrane Evidence Pipeline aims to expedite the study identification process for reviews and other evidence synthesis outputs, offering significant workload reduction for author teams within the current review production paradigm, but is also enabling a new paradigm to emerge—one that leverages economies of effort and scale to help create comprehensive, curated repos of studies for the EBHC communities who need it. Patient, public, and/or healthcare consumer involvement: The Evidence Pipeline uses Cochrane Crowd which is Cochrane’s citizen science platform. Anyone with an interest in health can join Cochrane Crowd. This collective effort has played a significant role in enabling the Cochrane Evidence Pipeline.

Picture 1.jpg

Background:
Background: For decades the p value-based interpretation and reporting of results dominated the publications. Nowadays, the scientific community agrees that this binary approach is not enough. A systemic reform is necessary for moving from this binary approach to some other non-binary options. Some of them, that were discussed in the last Cochrane Handbook version (2019), are the point estimate, the confidence interval, and the minimal important difference, suggesting some narrative statements. However, it is not clear how Cochrane and non-Cochrane editors and authors agree with this approach and, even more important, if all retrieve the same conclusions when they look at some specific results.
Objectives: To evaluate how Cochrane and non-Cochrane editors and authors interpret results and which preferred reporting style is for them.
Methods: We are conducting an online survey among healthcare professionals and consumers. These stakeholders have to choose the binary or non-binary option that better expresses the results for the following scenario: After exhaustive literature searches, a systematic review identified only two pivotal randomized controlled trials (RCTs) that evaluated the effectiveness of drug X versus placebo (P) in patients with a rare genetic disease. The risk of bias for all domains was low in both RCTs (assessed using the Cochrane RoB-2 tool), and there was no methodological, clinical, or statistical heterogeneity between studies. The meta-analysis yielded the following results: Mortality risk: X 26%(10/39) and P 45%(18/40) Risk difference: X 19% lower (95% CI 40% lower to 1% higher) Risk Ratio: 0.57 (95% CI 0.30 a 1.08) P = 0.0721 *95% CI: 95% Confidence Interval
Results: will be shown at the colloquium.
Conclusions: It will be interesting to see which preferred reporting statements are for the Cochrane and non-Cochrane communities, and to compare the preferences between authors and editors. Besides, we will know the level of agreement or disagreement in each group, which is critical for the efficiency of the editorial process.
Patient, public and/or healthcare consumer involvement: Proper interpretation of study results is critical for the healthcare process

Background: Systematic reviews (SRs) of randomised controlled trials (RCTs) often provide the highest certainty evidence for decision-making. However, if RCTs are not available to inform specific health questions, other study designs should be considered. This includes nonrandomised studies of interventions (NRSIs), which are frequently used in the field of vaccination.
Objectives: The key objectives are to illustrate the limitations and opportunities of including NRSIs in SRs using an exemplary review on vaccination against COVID-19 in children.
Methods: The exemplary review (https://doi.org/10.1016/S2352-4642(23)00078-0) followed Cochrane methodological standards. Drawing on the experience gained in the review process, practical challenges from study selection to interpretation of results will be illustrated. Further, limitations and opportunities of using NRSIs for informed decision making will be explored.
Results: The following key challenges in conducting our review with NRSIs were observed: (1) diversity of study designs (e.g., traditional cohort, ecological or test-negative case-control studies) in the field of vaccination, introducing clinical and methodological heterogeneity. (2) Additional risks of bias beyond the ones known from RCTs that originate from several factors pre-, at-, and postintervention level, particularly relating to methods used for selecting the study population, assignment of intervention groups and implementation of the intervention. (3) NRSIs often present different measures of adjusted effects (e.g., risk ratios, odds ratios or incidence risk ratios), challenging the synthesis of vaccine effectiveness estimates. (4) The evidence derived from NRSIs in a fast-changing pandemic situation requires very careful interpretation in relation to applicability and transferability (e.g., baseline immunity of study population and general population or investigated virus variants).
Conclusions: NRSIs are usually at a higher risk of bias and more heterogeneous in design than RCTs. Especially in fast-changing epidemiological landscapes, as observed during the COVID-19 pandemic, available RCT data were quickly outdated, whereas some NRSIs timely addressed changes in epidemiology (e.g., shift of variants). When considering concomitant limitations, our experience showed that NRSIs in this exemplary SR could be successfully used for timely information of health policy and practice. Consumer involvement: The paediatric working group of the Standing Committee on Immunisation (STIKO) in Germany was involved from prioritisation of the review question to interpretation of the findings.

Background: Developing clinically relevant, trustworthy, evidence-based resources to connect research to practice and assist with informing clinical decision-making is central to the improvement of healthcare outcomes. Development of such resources requires a multifaceted, coordinated, collaborative partnership approach. JBI has established a unique framework to engage and collaborate with key stakeholders across 32 specialty field areas to develop rigorous, up-to-date, clinically relevant evidence-based resources to inform clinical practice. Objective: To describe and discuss the JBI framework for collaborative partnership for practice improvement within the core activities of the JBI Transfer Science program.
Results: Underpinned by a rigorous methodological approach, more than 5,000 evidence-based resources across 32 specialty fields have been developed using a collaborative partnership framework. The core components and key characteristics of the partnership approach (including a scientific methodology group, expert reference groups, field collaborators and scientific writers) will be presented alongside discussion of the practicalities, enablers and barriers of implementing this framework within specialty field groups. The individual and shared role and function of the group will be explored with supportive examples demonstrating impact on practice. Conclusion: Incorporating a strategic framework that focuses on collaborative partnerships, in which a shared common goal between the groups is to improve evidence-based practice, can assist with reducing the research-practice gap and improving care delivery.

Background: The COVID-19 pandemic presented an unprecedented need for trusted evidence. We have an established program to co-create knowledge translation (KT) tools for parents and families that merge the best available research evidence with parent experiences to support decision-making for their children’s health. We received funding from the Canadian Institutes of Health Research to conduct qualitative research to identify parents’ experiences and information needs related to the pandemic and to co-create six KT tools.
Objectives: Describe the process of co-creating evidence-based KT tools for parents and discuss highlights and challenges.
Methods: We conducted qualitative interviews and focus groups with parents to understand their experiences and information needs related to caring for their children during the pandemic. We involved our established parent advisory group throughout the process to guide methods, pilot test processes, assist with recruitment, review KT tool prototypes, and participate in usability testing.
Results: The following themes were identified among parents who had managed a child with COVID-19: dealing with uncertainty, anxiety, stress and stigma, and getting through it. Themes related to the pandemic more generally were the following: loss of skill (children), changes in physical health, children’s and parents’ mental health, and accessing trustworthy information. After discussing themes with the parent advisory group, three topic areas were selected for KT tool development: managing a child with COVID-19, socialization during and beyond the pandemic, and COVID-19 vaccination. Six tools were developed: animated video and interactive infographic for each topic (https://www.echokt.ca/covid/).
Conclusions: A highlight and strength was working directly with parents. We involved our established parent advisory group as a partner on the funding application and throughout the project, and we believe this was critical to our success. Challenges included the number of KT tools (2 tools on each of 3 topics) developed using multiple methods (interviews, focus groups, living evidence synthesis) within a relatively short timeframe; changing evidence, information and protocols over time and across jurisdictions; reliance on external sources for evidence and information; appropriate messaging (informative vs. directive); and acknowledging and affirming parents’ varied experiences. Patient, public, and/or healthcare consumer involvement: Parents contributed to methods, pilot testing, recruitment, KT tool development, and usability testing.

Background: Cochrane was founded in 1993 and has grown into a global network producing gold standard evidence syntheses. Over those years we have been doing things in the same way and during that time the field has evolved. To maintain our place as one of the world’s leaders in producing trusted evidence, we are making some big changes in 2023. Reviews will be shorter. Users of evidence are best served by a more focused article. This also makes writing and updating reviews simpler as you do not need to make changes in multiple places. The main body of the review will contain sections that are most useful to users of Cochrane evidence, with a clearer distinction between the main article and supplementary materials. With a shorter review, we are moving the structure to where it is needed most: the data. This change is powered by moving to study centric data management in RevMan, which comes with a host of benefits for authors. By improving the structure of data associated with Cochrane reviews, we can increase their impact by making it easier for policy makers, guideline developers, and others to reuse our evidence.
Objectives: This presentation will provide an update on changes to the review format and data management for Cochrane reviews. It will cover: 1. Why has Cochrane changed the review format and how data are managed in RevMan? 2. What does it mean for review authors, editors, peer reviewers, and users of Cochrane evidence? 3. How the changes will unlock the potential for innovations in how we tailor our evidence for key stakeholders.
Conclusions: The standardized review article with more structured data will result in a better experience for authors, with the reassurance that fewer words do not equate to lower standards. The process of producing Cochrane reviews will be more efficient, facilitating living reviews and unlocking the potential for innovations in how we tailor our evidence for key stakeholders. Patient, public, and/or healthcare consumer involvement: User experience testing and consultations informed developments.

Background: The global evidence community has long recognised that collaboration is key to producing trustworthy, pragmatic evidence. The COVID-19 pandemic highlighted the need for partnerships in developing rapid evidence-informed responses, streamlining global efforts, reducing research waste and ensuring the best-available evidence is accessible, transparent and understood. There is, however, growing concern about making partnerships equitable for—and beneficial to—all partners. Although collaboration has increased, vested interests, bureaucracy and inability to adapt remain limiting factors. The campaign for World Evidence-Based Healthcare (EBHC) Day 2022 focused on ‘partnerships for purpose’ within and across the global evidence ecosystem, calling on the global evidence community to share their experiences through blog submissions around the formation, development, maintenance, evaluation and outcomes of partnerships in EBHC.
Objectives: To explore the challenges, facilitators and outcomes of global health partnerships promoting and supporting evidence-based healthcare.
Methods: This is an inductive qualitative study of the 41 blogs published for the World EBHC Day 2022 Campaign. Analysis was conducted using reflexive thematic analysis according to Braun and Clarke.
Results: Numerous themes were identified across three research foci, including shared challenges experienced across partnerships relating to the lack of equity, diversity and inclusion policies, practice or understanding; navigating different operational systems and working practices; and programmatic misalignment. Results indicated that key facilitators for successful partnerships were dedicated resources for partnership management; commitment to information sharing, transparency and open communication; a focus on equity, diversity and inclusion; ensuring a common purpose and aligned values; and working to actively build trust, respect, relationships and social capital. Reported outcomes from partnerships included increased efficiencies and reduced duplication of effort; leveraging shared resources and expertise; addressing knowledge needs/gaps; increased funding opportunities; professional development; and expanded networks, increased reach and global impact.
Conclusions: Since COVID-19, we have had to come to terms with a tectonic shift in how our partnerships and relationships will work in its aftermath. To this end, the findings from this study and the 41 blogs published for World EBHC Day provide valuable lessons for navigating new ways of working together. Patient, public, health consumer involvement: Several blogs were written by patient advocates.

Background: Long-awaited COVID-19 vaccines are effective in preventing severe COVID-19 disease. However, vaccine hesitancy, identified as one of the top 10 threats to global health by the WHO, has proven to be an important issue in the COVID-19 pandemic.
Objectives: Involve important stakeholder in the review process and investigate the effectiveness of communication strategies to increase COVID-19 vaccine intention.
Methods: Within this project funded by the Germany Research Foundation, we invited stakeholders to an online meeting to develop a relevant research question. The discussion was based on the findings of our prior scoping review, which maps current evidence on interventions to increase COVID-19 vaccine intention. Together with relevant stakeholders from policy and research, we prioritised types of interventions and target populations and developed the PICO for a subsequent systematic review. We will discuss the implementation of the intervention strategies, identified in the systematic review, in another stakeholder meeting by the end of the project.
Results: As the initial stakeholder meeting was so successful, most stakeholders agreed to further participate in the project and be involved in the upcoming systematic review. Thus, we had biweekly meetings until the end of the project with experts from the Robert Koch Institute and the NRW Center for Health. We conducted the planned systematic review of randomised controlled trials and included studies investigating communication-based interventions to increase COVID-19 vaccine uptake, for any population. The policymakers were particularly interested in communication strategies, as these are easy to implement. Outcome measures, considered most relevant by the stakeholder, were COVID-19 vaccine uptake, vaccine confidence and intention. We performed meta-analyses using Review Manager Desktop version 5.4. Our findings included 27 studies. Evidence from our meta-analyses shows that for COVID-19 vaccine uptake there may be a slight increase in vaccine uptake when education and information strategies or social norm strategies are applied, compared to no intervention.
Conclusions: Our findings will have implications for policy decision-making in prospective vaccination promotion, not only regarding COVID-19. Patient, public, and/or healthcare consumer involvement: Involving stakeholders in the review process ensured that relevant research questions are answered and facilitated translation and dissemination of the results.

Background: There has been an increasing number of systematic reviews and meta-analyses in dermatology over the past decade, however the quality of these studies is unknown.
Objectives: To determine the change in the methodological and reporting quality of systematic reviews and meta-analyses in dermatology over the past decade.
Methods: This was a cross-sectional study of systematic reviews and meta-analyses in dermatology published in the 10 highest ranked dermatology journals in 2010 and 2019. Included studies were identified by searching MEDLINE, Embase, CDSR, ACP Journal Club, DARE, CCA, CCRCT, CMR, HTA, and NHS EED. Methodological quality was assessed in duplicate by two independent investigators with the ROBIS tool and, for studies of interventions, the AMSTAR-2 tools. Reporting quality was assessed with the PRISMA 2009 and PRISMA-A 2013 statements.
Results: We identified 27 systematic reviews and meta-analyses published in 2010 and 127 published in 2019. There was no evidence of a difference in the proportion of systematic reviews and meta-analyses at high/unclear risk of bias with ROBIS (Fisher’s exact test = 1.00) or with critically low methodological quality using AMSTAR-2 (Fisher’s exact test = 0.456), between 2010 and 2019. The difference in proportion of systematic reviews and meta-analyses at high/unclear overall risk of bias with ROBIS was -1.2% (95% CI -17.3%-14.9%) in 2010 (81.4%) than 2019 (82.6%) (Figures 1-2). There was evidence of a difference in proportion of PRISMA (t(26)=2.7, p=0.01), and very strong evidence of a difference in proportion of PRISMA-A (t(26)=4.2, p<0.001) checklist items adequately reported between 2010 and 2019. The difference in mean proportion of PRISMA checklist items adequately reported was 3.6 items more (95% CI 1.8 – 5.4 items more) in 2019 (mean=10.7 items, SD=2.4 items) than 2010 (mean=7.1 items, SD=2.9 items), and of PRISMA-A checklist items adequately reported was 1.1 items more (95% CI 0.2–2.0 items more) in 2019 (mean=5.6 items, SD=1.5 items) than 2010 (mean=4.4 items, SD=1.7 items)
Conclusions: There was no improvement in the methodological quality of systematic reviews and meta-analyses in dermatology between 2010 and 2019, but strong evidence of an improvement in the reporting quality.
Patient, public and/or healthcare consumer involvement: None.

Background: A recent framework from the US Evidence-based Practice Centers advises that when evidence from systematic reviews is deemed insufficient for decision-making, electronic health record (EHR) data from health systems should be considered to fill the knowledge gap. A 2021 systematic review demonstrated that triptans improved pain and function for acute migraine attacks. However, the systematic review identified insufficient evidence about patients with increased risk of cardiovascular events who were often excluded from clinical trials.
Objectives: To evaluate the safety of triptan treatments in migraine patients with cardiovascular disease or multiple cardiovascular risk factors using EHR data from a large tri-state health system.
Methods: This target trial emulated a randomized clinical trial that hypothetically assigned patients to triptans or no triptan treatments. Eligible patients were adults (aged ≥18 years) with at least a 1-year history of migraines, confirmed cardiovascular or cerebrovascular disease, or with at least two cardiovascular risk factors. Patients in the triptan group were matched with those in the no-triptan group in a 1:1 ratio. The primary outcome was major adverse cardiovascular events (MACE) at 60 days of starting treatments.
Results: A total of 3,719 patients who received triptan were matched to 3,719 patients who did not receive triptans (Table 1). Figure 1 shows the cumulative incidence curves for MACE. The absolute risk of MACE outcomes was low (Table 2). At 60 days, 154 patients (4.14%) in the triptan group reported a MACE outcome compared to 37 patients (0.99%) in the no triptan group (RR: 4.16; 95% CI: 2.96 – 5.84). Patients treated with triptans also had significantly higher risk of nonfatal myocardial infarction, nonfatal stroke, transient cerebral ischemia, and heart failure. Five patients (0.13%) in the triptan group died versus seven (0.19%) in the non-triptan group. Certainty in the evidence was judged to be moderate.
Conclusions: This target trial emulation showed that triptans were associated with increased risk of MACE outcomes, raising the certainty of evidence in this outcome from insufficient, as was judged in a systematic review, to moderate. Patient, public, and/or healthcare consumer involvement: None.

table 1.pdf
Table 2.pdf
Figure1.pdf

Background: A diverse range of evidence synthesis approaches have been developed to meet end users’ (such as policy makers) knowledge needs. Examples include systematic reviews, rapid reviews, scoping reviews, realist reviews, network meta-analysis, meta-narrative reviews, and meta-synthesis. For researchers and those commissioning reviews, choosing the most appropriate method may be confusing. “Right Review” is a web-based decision support tool that guides users through a series of simple questions for recommending evidence synthesis methods suitable for their research question. Currently, the tool separates quantitative reviews and qualitative evidence synthesis.
Objectives: To update the Right Review tool to establish a common set of questions for the synthesis of both quantitative and qualitative studies.
Methods: A two-round modified online international Delphi was conducted (2021) with researchers, policy makers, health care providers, patients, and decision makers. Along with demographic questions, participants were asked to rate the importance and or clarity of the tool guiding questions, definitions of evidence synthesis types, and the tool output structure. A consensus definition of at least 70% agreement for each item was decided a priori. Any items not reaching consensus after round 2 were discussed by the international project steering committee (PSC).
Results: Twenty-four experts from 9 countries completed round 1 and 12 round 2. A total of 46 items were presented in round 1, with 21 reaching consensus. Twenty-seven items were presented in round 2, with eight reaching consensus. Items not reaching consensus, including 8 tool guiding questions, 9 review definitions (predominantly related to qualitative evidence), and 2 tool output items, were brought forward for discussion by the PSC. Three items were removed entirely, and the remainder revised and edited and/or combined with existing items.
Conclusions: The Delphi process informs the update of this tool which is free, practical decision support tool that helps reviewers choose an appropriate method.

Background: Opioid agonist therapy (OAT), typically involving buprenorphine or methadone as pharmacotherapy, is an effective treatment for opioid use disorder (OUD) and is a key response to growing opioid-related harms in Canada. OAT delivery in Canada varies between provinces/territories, including differences in prescribing requirements and primary care involvement. For provinces/territories that aim to improve OAT delivery, knowledge of other jurisdictions’ policies, expected outcomes, and how they have been influenced by political, sociocultural, and health system factors, can inform clinical and policy decision-making.
Objectives: To compare OAT prescribing policies across Canadian provinces/territories and understand how and why they occur.
Methods: We conducted a comparative policy analysis, collecting data from policies, guidelines, reports, education/training materials, and published research articles on provincial health systems and drug policies. To describe and compare OAT across all Canadian provinces, we constructed timelines of OAT policies for each jurisdiction from 2014. Experts in each province (clinicians; policymakers) reviewed these to ensure accuracy and relevance. We classified these policies based on provider involvement; financial, personnel, and educational resources for providers; treatment settings; and costs to patients. We also examined any cross-provincial interactions and resource sharing.
Results: In May 2018, Health Canada removed a federal requirement to obtain an exemption to prescribe methadone, which could increase patients’ access to this treatment. After examining provincial policy changes before and after this decision, we identified patterns in OAT prescribing regulations: western jurisdictions (e.g., British Columbia, Alberta, Saskatchewan) tended towards more restrictive prescribing requirements and oversight than their eastern counterparts (e.g., Quebec, New Brunswick, Nova Scotia). We also observed similar geographical patterns with recommendations for providers to use training/guidelines from either British Columbia (west) or Ontario (east).
Conclusions: Our findings suggest that OUD treatment policies may be influenced by geographical, political, and/or professional networks. What is considered evidence or knowledge for decision-making may be shaped by these factors and local policymaking needs. Patient involvement: By engaging people with lived/living experience and decision-makers, further exploration of these factors and their effect on OUD treatment policies can better inform the development of contextually relevant and appropriate policies to improve OUD care.

Background: Improving health literacy and increasing the use of reliable health information may reduce vulnerability to unreliable health-related information. Health professionals may help in translating the knowledge to nonprofessionals, but they often find it challenging.
Objectives: To show our experiences and pathway of knowledge translation (KT) actions for consumers.
Methods: We applied for funding from the Social Responsibility of Science. Project started in the November 2022 and will last for 2 years. The project is based on Cochrane systematic reviews (SRs) and plain language summaries (PLSs). We started collaboration with health professionals to prioritise SR’s topics which are deemed important to healthcare practice (e.g., physiotherapists) and to patients’ groups with whom we already cooperate (e.g., cystic fibrosis patients). Prioritization was based on subjective assessments on a scale from one to ten of the utility of the topic for healthcare practice completed by two healthcare professionals independently. SRs which got the highest assessments and were deemed to be current were grouped into themes by health condition. Results from SRs will be shared using plain language summaries and additionally expressed in the form of infographics and animations. All necessary professional terms will be explained with comic drawings or short (1-3 minute) webinars. The aim for all is to make an easily readable and understandable “information pill” from each theme. All our experiences and expertise, gathered from our work as Cochrane Poland, will be structured to prepare the guide for Polish professionals to help in everyday KT.
Results: Project is connected with translations of 120 PLSs per year and preparation of additional products (infographics, animations). We expect to have at least three topic themes expressed in plain language and described in graphic forms for presentation during the Cochrane Colloquium.
Conclusions: Promotion of SRs among health professionals in Poland still needs effort, but translation of studies results’ into easily accessible language for patients and lay audience is even more challenging and emerging.
Patient, public and/or healthcare consumer involvement: Our project mainly aims to bring professional knowledge from Cochrane SRs close to patients’ everyday lives.

Background: Systematic reviews (SRs) synthesise the best evidence of effectiveness and safety on Chinese herbal medicine (CHM). Decision-making should be supported by the high-quality evidence of prudently conducted SRs, but the trustworthiness of conclusions may be limited by poor methodological rigour.
Methods: This survey aimed to examine the methodological quality of a representative sample of SRs on CHM published during January 2018 to March 2020. We conducted literature search in the Cochrane Database of Systematic Reviews, MEDLINE via Ovid, and EMBASE via Ovid. Eligible SRs must be in Chinese or English with at least one meta-analysis on the treatment effect of any CHM documented in the 2015 Chinese Pharmacopoeia. Two reviewers extracted the bibliographical characteristics of SRs and appraised their methodological quality using AMSTAR 2 (Assessing the Methodological Quality of Systematic Reviews 2). The associations between bibliographical characteristics and methodological quality were investigated using Kruskal-Wallis tests and Spearman’s rank correlation coefficients.
Results: We sampled and appraised 148 SRs. Overall, one (0.7%) was of high methodological quality; zero (0%), four (2.7%), and one-hundred forty-three (96.6%) SRs were of moderate, low, and critically low quality. Only 13 SRs (8.8%) provided a pre-defined protocol; none (0%) provided justifications for including particular primary study designs; six (4.1%) conducted a comprehensive literature search; two (1.4%) provided a list of excluded studies; nine (6.1%) undertook meta-analysis with appropriate methods; and seven (4.7%) reported funding sources of included primary studies. Cochrane reviews had higher overall quality than non-Cochrane reviews (P ˂ 0.001). SRs with European funding support were less likely to have critically low quality when compared with their counterparts (P = 0.020). SRs conducted by more authors (rs = 0.23; P = 0.006) and published in higher impact factor journals (rs = 0.20; P = 0.044) were associated with higher methodological quality.
Conclusions: Our results indicated that the methodological quality of SRs on CHM is low. Future authors should enhance the methodological quality through registering a priori protocols, justifying selection of study designs, conducting comprehensive literature search, providing a list of excluded studies with rationales, using appropriate method for meta-analyses, and reporting funding sources among primary studies.

Background: Framework synthesis (FS) is a method for synthesis of qualitative research in systematic reviews (SRs), which is being increasingly utilised to synthesise data from diverse sources to explore complex healthcare issues. While published guidance on conducting mixed-methods SRs and qualitative FS is available, there is little consensus on conducting mixed-methods FS.
Objectives: To conduct a targeted literature review (TLR) to evaluate the current use and methodology of FS in mixed-methods literature reviews.
Methods: This TLR was conducted according to a prespecified protocol. In February 2023, we searched MEDLINE, Embase and Cochrane Database of Systematic Reviews (via Ovid) for mixed-methods reviews using FS. Records were reviewed by a single individual at the title/abstract and full-text review stages. Information was extracted on rationale for FS use, framework development, extraction, synthesis and integration of quantitative and qualitative data, quality assessment (QA), outputs and reported limitations of FS.
Results: We initially identified 240 unique records, of which 123 were excluded at title/abstract review and 68 excluded at full-text review, leaving 49 peer-reviewed publications for analysis. All but four reviews were SRs, with two rapid reviews, one scoping review and one SR of SRs. Primary quantitative, qualitative or mixed-methods studies were eligible for inclusion; secondary research or opinion were included in less than 25% of reviews. Two reviews only extracted author summary or interpretation of quantitative findings. The level of detail on how FS was conducted varied, with some detailing the process step by step and others only citing the method. Very few described how quantitative and qualitative data were integrated. Where reported, methods included convergent synthesis, whereby quantitative data are transformed into qualitative statements through narrative interpretation (“qualitising”), and the Pillar Integration Process, whereby data are listed by type and then matched to corresponding qualitative or quantitative data before further synthesis. Most reviews used QA tools designed to appraise multiple study designs; however, some did not perform formal QA owing to heterogeneity. Few reviews cited qualitative reporting guidelines.
Conclusions: The volume of mixed-methods literature reviews using FS being published is increasing. With limited consensus on best practice, further guidance and improved transparency in reporting is required.

Background: In 2014, the Systematic Review (SR) Toolbox was developed to gather tools that could support researchers undertaking an SR. As the breadth of evidence synthesis methodologies have expanded greatly since this time, we updated the SR Toolbox in 2022 to reflect these developments. We also analysed the software tools and guidance within the SR Toolbox at the time to identify gaps that could potentially be filled by new tools and guidance in future.
Objectives: To describe the process and present the results of updating the SR Toolbox and analysing its content.
Methods: In February 2022, we manually extracted all guidance documents and software tools within the SR Toolbox to that point on a Microsoft Excel spreadsheet. The spreadsheet categorised records by types of evidence syntheses records as described by Sutton et al. (2019) and by the stage of process in undertaking evidence syntheses (e.g., searching, screening, and data extraction). A single reviewer extracted and analysed these records, with a second reviewer checking a proportion for accuracy. By using this spreadsheet and Microsoft Access, we updated the SR Toolbox and performed a simple quantitative analysis on the included records.
Results: On 13 May 2022, we launched the updated SR Toolbox. At the time, 235 software tools and 112 guidance documents were included. Most guidance documents (N = 78) and software tools (N = 223) were applicable to systematic reviews of interventions. However, there were fewer included tools and documents that related to reviews of reviews (N = 66 software tools and N = 22 guidance documents) and qualitative evidence syntheses (N = 19). Most documents provided guidance on critical appraisal and quality assessment (N = 70), with software mainly focusing on searching (N = 84) and data synthesis (N = 82). However, there were very few tools or guidance documents that related specifically to stakeholder engagement (N = 5).
Conclusions: The SR Toolbox can support those looking to undertake different types of evidence syntheses. However, there are currently gaps in the guidance and software tools available, which may warrant further research.
Patient, public and/or healthcare consumer involvement: None.

Background: Despite progress in the health of newborns and children, most countries in sub-Saharan Africa have not met the Sustainable Development Goals for under-five mortality. Consequences of COVID-19 are being felt on health systems and services, further hampering healthcare delivery to children alongside growing poverty, hunger and inequity. Clinical practice guidelines (CPGs) are needed to bridge the gap between research evidence and practice directly impacting patient care, health system funding and access to health services.
Objectives: The Global Evidence, Local Adaptation (GELA) project is working with ministries of health in South Africa, Nigeria and Malawi to maximise the impact of research through increasing decision-makers’ and researchers’ capacity to use global research to develop locally relevant CPGs for newborn and child health.
Methods: From 2023 to 2025, we are implementing a multi-faceted, multidisciplinary research and capacity strengthening programme, adaptation methodology and digital platforms to support authoring delivery for contextually rich CPGs. This is delivered through partnerships with national CPG groups, including policy makers, epidemiologists and civil society representatives.
Results: A 3-year project was successfully initiated in all countries. Steering Group members and Guideline Development Panelists were identified and are participating in various project-related activities. Recently, the teams completed a landscape analysis of CPGs in newborn and child health and concluded the priority-setting process identifying three topics for new CPGs for newborn and child health per country. Monitoring and evaluation activities are using a variety of approaches including integrated knowledge translation, observation of CPG groups and evaluation of capacity needs for decision-makers.
Conclusions: Overall, the project is on course to achieve its targets with priority topics identified and CPG groups being convened. Our success is enabled through a project team of African and international leaders in the field of evidence-based healthcare and CPG methods who have partnered with national ministries in Malawi, Nigeria and South Africa, the World Health Organization headquarters (WHO HQ) and the Afro regional office and civil society groups. Ongoing evaluation will help us learn which aspects work well to reduce waste and save resources for our settings and may be scalable to other countries like ours.
Patient, public and/or healthcare consumer involvement: Yes.

Background: In our previous study on the reporting quality of abstracts according to PRISMA-A, we found that abstracts of Cochrane reviews (CRs) were better reported than non-Cochrane reviews, differed significantly in length (median length: 507 words vs. 249 words), and appeared to be getting longer in recent years. However, the considered sample of CRs was too small to draw robust conclusions. Overall, there is a general lack of studies that address the quality of reporting or trends in the length of the abstracts of CRs.
Objectives: To evaluate the reporting of abstracts of CRs according to PRISMA-A and to investigate the possible influence of the abstract´s length.
Methods: This was a retrospective, observational study based on all CRs indexed in Medline (via PubMed) until November 18th, 2022. Trend analyses were performed on the number of publications and length of abstracts over the years for all 15,188 abstracts. Then, PRISMA-A adherence was assessed by two independent reviewers for a random sample of a planned sample size of 440 abstracts. Results were stratified by the number of words and year of publication.
Results: Overall, the median number of words in the abstract was 469 (IQR: 389-686 words), steadily increasing from 353 words in 2000 to 807 words in 2021, with less than 1% of the abstracts being less than or equal to 300 words (in 2000: 30.7%). Analyses on PRISMA-A adherence show a mean score of 6.1 fully reported items. Stratified by year, PRISMA-A adherence increases with increasing word count in 2000-2010 and 2011-2015, whereas there is no difference in PRISMA-A adherence by the abstract length in 2016-2022.
Conclusions: Over the years, abstracts of CRs have become longer and longer, running up to 1,000 words. This conflicts with the Cochrane Handbook, which recommends a maximum length of 400 words before being adapted to MECIR in 2019, which has recommended a length of less than 700 words since 2012 but allows a length of up to 1,000 words. It is questionable whether such long abstracts meet the goal of presenting all essential contents of the article in an informative, accurate, appealing, and concise form. Patient, public, and/or healthcare consumer involvement: No involvement.

Background: Risk of bias (RoB) assessment is an integral part of the systematic review methodology. New version of the Cochrane RoB tool for randomized trials (RoB 2) was published in 2019. It has been reported that the Cochrane RoB tool from 2011 was used inadequately in Cochrane and non-Cochrane reviews. There are insufficient data about the use of RoB 2 tool.
Objectives: This study aimed to analyze the frequency of usage of RoB 2 tool and the adequacy of reporting RoB 2 assessments in non-Cochrane reviews published in 2020.
Methods: We conducted a meta-research study. We included non-Cochrane systematic reviews of interventions published in 2020. For the reviews that have used RoB 2 tool, we analyzed how the findings of the RoB 2 assessment were reported.
Results: Among 3880 analyzed reviews, the authors most frequently reported use of the Cochrane 2011 RoB tool (N=2228; 57.4%), followed by Cochrane RoB 2 tool (N=267; 6.9%). From 267 reviews that reported using RoB 2 tool, 213 (79.8%) actually used RoB 2 tool. Among 213 reviews that used RoB 2 tool, 180 (85.4%) used all domains specified by the RoB 2 tool. In 26 (12.2%) reviews, erroneous statements were used to indicate RoB 2 assessment. Only 20 (9.4%) reviews presented a complete RoB 2 assessment through a detailed table with answers to all signaling questions. The judgment of risk of bias through RoB 2 tool was not justified by a comment in 158 (74.2%) of reviews. Only in 33 (14.5%) of reviews, the judgment in all domains was justified in the accompanying comment. Conclusion: The majority of non-Cochrane reviews published in 2020 still used RoB 2011 tool and many authors reported the results of RoB 2 assessment inadequately. Interventions for improving RoB 2 tool use in non-Cochrane reviews are warranted. Relevance and importance to patients: Efforts to improve research methodology and how studies are conducted can lead to better conducted and written studies, and better evidence that can be used for managing patients.

Background: When investigators analyze data from observational studies, they make numerous potentially justifiable, but still subjective, analytic decisions on which direction, magnitude, and statistical significance of findings can be contingent. This allows investigators to test many alternative analytic specifications and selectively report results for the specification that yields the most statistically significant or interesting results. Objective: We present a novel approach to interpret the results of observational studies in the context of the variation expected because of analytic flexibility. We apply this new method to the effect of unprocessed red meat on all-cause mortality.
Methods: We reviewed all observational studies addressing the effect of red meat and all-cause mortality from a recently published systematic review and documented variations in analytic choices across studies. We used data from the National Health and Nutrition Examination Survey (NHANES) 2007 to 2014 linked with National Death Index and applied specification curve analysis—a novel analytic method that involves defining and implementing all plausible and justifiable analytic specifications—to investigate the effect of unprocessed red meat on all-cause mortality. Our choice of analytic specifications was informed by analytic methods used in published studies addressing the same or similar questions.
Results: We applied specification curve analysis to NHANES, including 10,661 participants. In total, we performed 1,208 unique analyses, 48 (3.97%) of which produced statistically significant results, 40 (83.33%) of which indicated red meat reduced all-cause mortality, and 8 (16.67%) of which indicated red meat to increase mortality. The specification curve analysis produced a median hazard ratio of 0.94 [IQR: 0.83 to 1.05].
Conclusions: Our results suggest that inconsistency in the reported results in the literature may be explained by differences in analytic methods. We encourage evidence users to interpret the results of observational studies in the context of variation expected due to analytic flexibility. Patient, public involvement: We did not involve consumers in this investigation.

VoE Figure.pdf

Background: Large-scale systematic reviews are becoming more common owing to the rapid growth in publications on certain topics. They raise unique challenges for managing and coordinating the review process for which there is limited guidance. We will present the challenges we encountered and how we addressed them during two large Cochrane systematic reviews: Quality improvement strategies for diabetes care (n=553 trials) and Audit and feedback (n=290 trials).
Objectives: To address the logistical challenges that are common in the coordination of large systematic reviews.
Methods: We will compare the approaches to coordinating two large systematic reviews. We will focus on common logistical challenges such as recruiting and maintaining a team with enough availability to complete the project in a timely manner; delegating work and ensuring consistency between reviewers on a large team; and managing and housing a very large dataset. We will also compare tools for managing large reviews such as the systematic review programs DistillerSR and Covidence for usability and customisability as well as online repositories for external data storage.
Results: The main challenge in coordinating large systematic reviews is dedicating enough resources to ensure the review can be completed before another update is required. Having a large team can be useful for keeping momentum going in screening and extraction; however, it is vital to ensure that reviewers are being consistent by using clear and specific instructions and having a training period for each reviewer. Smaller teams are better for ensuring consistency in reporting, but screening and extraction takes longer. The use of technology such as DistillerSR and Covidence facilitates the organisation of screening a large set of studies. DistillerSR offers more customisability but may be cost-prohibitive. Covidence links nicely with RevMan Web but may not be as user-friendly. An online data repository such as Zenodo is useful for streamlining the main review and preventing any issues with the Cochrane CENTRAL system when it comes time for publication; however, there is always a risk of repositories no longer being in service in the future.

Background: Hospitals are usually the entry point of new technologies. Facing the challenge posed by the growing costs of cancer drugs, there is a need for healthcare providers to articulate systematic and timely formulary decision-making processes considering not only the efficacy and safety of these treatments, but also their economic impact and efficiency.
Objectives: To develop a method for the rapid assessment of high-cost cancer drug indications (HCCDI) for formulary decision-making in a hospital setting.
Methods: We identified and structured decision criteria from previous formulary decisions regarding HCCDI, and a literature review of decision frameworks in oncology and the Chilean health reimbursement system. We generated a criterion/indication combination score for two samples of HCCDI (solid tumors and hematologic oncology) by measuring their performance and translating it to a common 0-100 scale. From the consensus among clinicians, managers, pharmacists, and health economics researchers of the institution, we assigned weights to each criterion to calculate the aggregate scores. Finally, we validated each step of the process with the decision-makers.
Results: The identified criteria were (1) absolute and relative difference in overall survival and (2) progression-free survival of the area under the Kaplan-Meier curves; (3) difference in the proportion of patients with severe adverse events; (4) improvement of quality of life; (5) incremental expected costs; and (6) efficiency. The inputs for the assessment of each HCCDI were high-quality studies identified through rapid reviews of electronic databases and health technology assessment reports. Aggregation of criteria 1-4 was named “benefit index”. The quotient of the benefit index and the HCCDI costs was called “efficiency index”. The costs were then used to calculate the indication’s budget impact.
Conclusions: We developed a formulary decision-making tool for the rapid assessment of HCCDI validated by decision-makers. The method has allowed us to assess 79 indications for solid tumors and 21 for hematologic oncology since 2019, facilitating the reproducibility of the decision criteria. Improvements in the principles regarding HCCDI formulary decisions contribute to a more egalitarian use of resources and patients’ access to healthcare. Patient, public, and/or healthcare consumer involvement: No patients, public, or healthcare consumers were involved in this project.