Every country on earth is now facing a resource crisis that will not be resolved simply by focusing more on effectiveness; a new paradigm is required - value based healthcare (VBHC). The EU definition of VBHC is now adopted globally, except in the USA which is not committed to universal healthcare. Their the term ‘value’ is the relationship between outcomes and costs for the patients treated which would be classified as efficiency in all other countries, but Archie did not use the term Efficiency in that narrow way. Archie wrote that he was concerned about ‘the optimum use of equipment and personnel ...to cover all these activities I have used the word efficiency’. He was clearly thinking about value, the first person to do so in modern healthcare. The EU definition has four dimensions - personal, allocative, technical and social, and all four are excellently introduced by Archie. Personal value resonates through his description of care in the prisoner of war camp. Allocative value is highlighted in his text on ‘inequalities between diseases... the division of the medical budget amongst all the medical activities’ (p76). Technical value is described in emphasising the need for the ‘optimum use’ of resources for not only problems of treatment but also ‘..screening, diagnosis, place of treatment and length of stay, and, if necessary, rehabilitation’ (p2) and social value in describing the ‘local pride’ that people had for their ‘cottage hospital’ and how this influenced the flow of resources (p34). Effectiveness and Efficiency therefore introduced VBHC fifty years ago! As health services around the world come to understand that they cannot close the gap between demand and resources by ensuring that only effective interventions are offered, there is a huge opportunity for the Collaboration to lead the VBHC paradigm as it led the Evidence Based Healthcare Paradigm. This does not mean doing cost-benefit analyses. It means using the experience and judgement in all the groups to promote stewardship and provide advice on the best use of resources, on value, not just on effectiveness.

Background: We conducted systematic reviews and meta-analyses of over 250 randomized trials of interventions to prevent obesity in children aged 5 to 18 years. Although we found that interventions targeting diet, activity, and both combined may have small beneficial average effects on BMI, considerable heterogeneity arose from different participant and intervention characteristics. To explore how intervention characteristics, and their synergistic effect, impacted on the intervention effectiveness in preventing obesity, we aimed to reanalyse the results of the studies. To achieve this, we developed a novel analytic framework that addresses key intervention characteristics that are likely to explain differential effects. As children are the direct target of most of the interventions in our review, we decided to ask them what features they think an intervention should have to work best at achieving its goal.
Objectives: To describe the beneficial role that children and young people played in the development and implementation of our analytic framework.
Methods: A literature review informed our preliminary analytic framework, which we discussed with our project advisory group (also including two young people aged 15+). We then organised two workshops with children and young people (in one instance also with their parents) in which we discussed our framework aims and asked the participants for advice on what intervention characteristics may impact its effectiveness. The coding of the trials according to the finalised framework is being undertaken in part by young people themselves.
Results: By asking our panels of children and young people the simple question “What should we do to prevent childhood obesity?”, it emerged that interventions they thought would be best received were fun, casual, enjoyable, interesting, and interactive. Additionally, our panel suggested that the credibility of the person delivering the intervention and the opportunity to choose activities (e.g., their favourite sport) are also important features.
Conclusions: Working with children and young people in both the planning and conduct of our work has been a scientifically useful and personally stimulating experience. Patient, public, and/or healthcare consumer involvement: Children and young people were involved in the development of the analytic framework.

Background: Compared with clinical practice guidelines, the implementation context for health-system guidance (HSG) is more multi-layered and complicated because of the complexity of health systems, the diversity of evidence, and the highly context-sensitive and multifactorial policymaking process. Current guideline implementation frameworks and tools might not be suitable for relatively complex HSG implementation. Therefore, a comprehensive and well-organized theoretical framework to support the HSG implementation was developed.
Objectives: Building on a theoretical framework we developed, this project aims to develop an operationalized tool for supporting global HSG implementation at national or subnational levels.
Methods: First, a modified Delphi approach with two steps was applied, including the following: (1) generating items based on the theoretical framework and (2) using a modified Delphi method to select and refine items to be incorporated in the tool. Second, a formative evaluation approach with four steps was applied, including the following: (1) operationalizing the HSG implementation tool based on the Delphi-derived components (i.e., domains and individual items in each domain); (2) theoretically ‘testing’ the application of the tool with several HSG examples; (3) examining the face validity and usability through semi-structured interviews; and (4) finalizing the HSG implementation tool based on the above findings.
Results: The main output of this study is an operationalized HSG implementation tool that can support HSG implementation at the national and/or subnational level. Also, we will learn about whether, how, and why the HSG implementers will use this tool and what could be done to make this tool more usable. The results will be presented at the conference.
Conclusions: The operationalized tool will help HSG users identify potential factors that facilitate or hinder the HSG implementation process, and strategies can leverage facilitators and address barriers during their preparation for HSG implementation; on the other hand, the tool will help HSG developers integrate the implementation considerations during the development process of HSG. Patient, public, and/or healthcare consumer involvement: As the target population of HSG recommendations, the healthcare consumer representative will be invited to participate in the modified Delphi and formative evaluation processes to provide the related insights that contribute to the tool development.

Background: There is strong evidence that machine learning can substantially reduce the burden of manual systematic review (SR) screening. However, outside of Cochrane and some academic groups, the adoption and acceptability of tools is still weak.
Objectives: To assess the accuracy of the Cochrane Randomised Controlled Trial (RCT) Classifier to accelerate systematic review screening in a research consultancy setting.
Methods: Our Review and Evidence Synthesis (RES) team are developing a semiautomated workflow (“RESbot”), which pieces together compatible tools to accelerate SR production. We have tested the Cochrane RCT Classifier as a potential candidate to include in the workflow to support screening. From December 2022 to February 2023, the classifier was tested on three SRs of RCTs covering interventions for renal denervation (Review A), postpartum depression (Review B), and schizophrenia (Review C). For each review, the search results were screened manually by two independent reviewers. The same results were run through the classifier, using both the “sensitive” and “precision” version. Classifier results were cross-checked against reviewer decisions.
Results: For Review A, the search retrieved 2,795 records. Manual screening found 24 eligible trials. Loading the search results through the classifier reduced the volume to 1,504 (sensitive) and 701 (precision). The precision set contained 23 of the included trials and the sensitive set contained all 24. For Review B, 2,153 records were retrieved by the search. Manual screening found 22 eligible trials. The classifier reduced the volume to 1,594 (sensitive) and 1,265 (precision). Both sets contained all 22 trials. For Review C, 1,172 records were retrieved by the search. Manual screening found 20 eligible trials. The classifier reduced the volume to 823 (sensitive) and 548 (precision). Both sets contained all 20 trials. The reduction in screening burden across the tests ranged from 25.9% to 46.1% using the sensitive and 41.2% to 74.8% using the precision setting.
Conclusions: The Cochrane RCT Classifier performed well in our tests, with no trials missed across the three reviews using the sensitivity setting and only one trial missed using the precision setting (in Review A). Our findings support the wider adoption of this classifier to accelerate review production.

Background Meta-analysis has grown increasingly complex in recent years, with a growing range of alternative and sometimes conflicting methods for combining studies, weighting studies, estimating heterogeneity, calculating confidence intervals, and so on. It is increasingly difficult to decide what methods to use, and it is difficult for stakeholders to understand the methods and be confident of their validity. Objectives To challenge the growing complexity of meta-analysis by seeking to return to the first principles for meta-analysis and considering why we perform meta-analyses at all. Concepts It will be argued that there are two key purposes for meta-analysis: 1. To summarise the evidence in identified studies of an intervention. 2. To investigate a deeper “scientific truth” about the intervention. The talk will make the case that the growing complexity in meta-analysis stems from a confusion of the two stated purposes of meta-analysis: by making the flawed assumption that a simple summary of evidence extracted from publications can tell us about the truth underlying the included studies. The talk will demonstrate that a simple “assumption-free” weighted average approach to meta-analysis is always valid if we are aiming only to summarise the available evidence. This also applies when considering heterogeneity. A simple approach has limitations, particularly with few studies or data, but shifting to more complex methods does not solve the problems. Conversely, identifying the “truth” about an intervention requires strong assumptions about the studies that are likely to be flawed. Alternatively, it requires a more complex analysis approach, with better data, such as an individual participant data meta-analysis. Conclusions We need to be clearer on what the purpose of any meta-analysis is and be honest about its limitations. Simple, conventional approaches may be best when we are restricted to summarising evidence in publications. Greater clarity will ensure that patients and stakeholders have a better understanding of what any meta-analysis is actually telling them and what it can’t tell them, without unhelpful methodological complications. This should support greater understanding of the strengths and limitations of meta-analyses. Patient, public, and/or healthcare consumer involvement: No involvement.

Background: Public health and social measures (PHSM) refer to nonpharmaceutical interventions implemented by individuals, communities and governments at all levels. During the COVID-19 pandemic, PHSM have been applied globally at an unprecedented scale and time span. PHSM have protected both lives and livelihoods, but they have also had negative health, social and economic consequences for individuals and societies, particularly for those living in vulnerable conditions. Producing robust evidence on the effectiveness and negative impacts of PHSM and understanding factors influencing adherence to measures is crucial to equitable and context-specific decisions and strengthening emergency preparedness and response. However, the complexity of PHSM and ethical, legal and feasibility limitations related to PHSM research in an emergency context, especially randomized controlled trials, have hampered systematic evaluation of PHSM.
Objectives: The World Health Organization (WHO) launched a global initiative in 2021 to build robust data and research evidence to better understand the effectiveness of PHSM and improve precision in future PHSM decisions and policies.
Methods: The initiative is taking a comprehensive approach to strengthen the evidence base on PHSM spanning from conceptual work to promoting primary research production and evidence use in policy.
Results: The initiative’s research pillar promotes the production of trustworthy and actionable evidence on PHSM through (i) a conceptual framework on PHSM fostering a common and comprehensive understanding of PHSM and their impacts, (ii) a series of global evidence syntheses on the effectiveness and impact of PHSM, factors influencing the adherence to measures as well as social protection policies for protecting livelihoods, (iii) a global research agenda for setting priorities for research production for the next decade, (iv) a suite of study protocols to collect high-quality, comparable and timely data on the effectiveness of PHSM in various settings during an emergency, and (v) a research database serving as a one-stop repository of primary studies and living systematic reviews on PHSM.
Conclusions: The WHO PHSM initiative aims to produce and catalyse evidence that is comparable, high quality, equity focused and policy relevant to support decision-makers to make evidence-informed decisions about PHSM to better protect communities during future emergencies.

Background: Hospital-led, multicomponent organisational strategies that aim to enhance recovery after surgery are associated with improved clinical outcomes. Less is known about aspects of interventions or their delivery which can lead to better patient outcomes.
Objectives: To establish which aspects of multicomponent interventions to enhance recovery after surgery and reduce length of stay in hospital are associated with better outcomes for older adults admitted to hospital for planned procedures.
Methods: Working in partnership with expert clinical advisors, older adults, and carers with experience of hospital stays for planned procedures, we undertook a mixed methods synthesis. We first conducted a quantitative review (of comparative studies reporting length of stay and patient reported outcome or experience measures) and a qualitative review (of the experiences of patients, carers and staff of interventions received/delivered). We then developed a logic model based on patient perspectives represented in the qualitative review and discussions with clinical stakeholders and our patient and carer groups. The logic model was used as the focus of a Qualitative Comparative Analysis (QCA) to integrate the findings of the quantitative and qualitative reviews.
Results: A complex balance of intervention components trigger successful interventions: these represent approaches that allow patients to understand their treatment (e.g., through providing information in different formats), to ask questions about their treatment (through spending additional time with patients), and to build supportive relationships (through having a consistent point of care) and strategies that facilitate patients to monitor their own progress (goal-focused) and challenge themselves in recovery (through early mobilisation). Limitations: We did not consider the efficacy of different clinical procedures (largely unseen or unknown to most patients) that may also influence outcomes; evidence for some of the components came from weak studies; most studies screened patients with more complex needs out of the intervention; poor or absent reporting meant that we had to discard or combine some elements identified within the logic model.
Conclusions: Grounding our synthesis in the experiences of patients, carers, and clinicians enabled us to generate meaningful implications for policy and practice that go beyond measuring success in terms of reducing time spent in hospital.

Background: Low- and middle-income countries are disproportionately affected by high rates of under-5-year-old mortality, often caused by preventable conditions. High-quality, evidence-based clinical practice guidelines (CPGs) may play a key role in improving clinical management and quality of care and, thus, in reducing child mortality. However, limited availability or poor accessibility, inadequate reporting and low-quality methodologies of available CPGs may undermine their utility and expected impact in improving quality of care and outcomes.
Objectives: To conduct a scoping review to identify CPGs for newborn and child health published in South Africa, Nigeria, and Malawi and to assess their quality.
Methods: We searched key websites (June-July 2022) for published national and subnational de novo or adapted CPGs addressing newborn and child health in the three countries. Pairs of reviewers screened for eligibility, extracted information from included CPGs (scope, condition or topic, target population and end users, developers, methods), and appraised their quality using the Appraisal of Guidelines for Research & Evaluation (AGREE II) instrument. Data were analysed descriptively in Microsoft Excel and STATA-17.
Results: We included 40 CPGs. Most provided guidance on communicable diseases (19/40), whilst eight on non-communicable diseases. The majority of CPGs did not report on methods for assessing the certainty of evidence (7/40), of which one CPG reported using GRADE. Overall, CPGs scored well on clarity of presentation (median 81%, IQR 67-94), and scored poorly on AGREE II domains rigour of development (median 11%, IQR 4-32) and editorial independence (median 6%, IQR 0-27). Similar trends in AGREE II domain scores were observed across all countries.
Conclusions: We highlight gaps in methodological and reporting quality of CPGs for newborn and child health across the three countries, which may impact our credibility of recommendations. Gaps in guidance were identified as the topics of CPGs often did not align with country-specific disease burdens. CPGs specifically addressing malnutrition, neonatal disorders, and trauma related-mental health disorders need to be considered. We further highlight a lack of regional and national CPG repositories which may impact on access to and use of CPGs. Patient, public, and/or healthcare consumer involvement: No direct patient involvement in the conduct of our research.

Background: Vast gaps in healthcare access exist within the United States, particularly across racial and ethnic groups. Promoting health equity has become a priority of the US government and major health organizations; however, a national consensus on shared definitions of these terms does not yet exist. In this study, we will compile, evaluate, and synthesize “racial health equity” (RHE) definitions via a landscape review. Inclusion criteria will encompass systematic reviews, primary literature, narrative reviews, and healthcare guides that include RHE definitions. Defining RHE is a first step to centering and advancing racial health equity.
Objectives: To evaluate RHE terminology and definitions from key health organizations, primary articles, narrative reviews, and systematic reviews and to compile and synthesize findings to evaluate research gaps and make recommendations for future research.
Methods: We developed a three-part search strategy to encompass RHE definitions. First, definitions of RHE were compiled from major health organizations with relevance to high-income countries. Second, for systematic reviews, we used a comprehensive search strategy in MEDLINE, Psychinfo, the Cochrane Library, and Campbell systematic reviews since 2020. Third, for primary research articles and narrative reviews, we searched MEDLINE, CINAHL, Psychinfo, and the Campbell Collaboration library, with no date restriction. Both systematic search strategies were developed with a research librarian. Study title/abstracts and full texts will be screened by two reviewers and disputes will be resolved by consensus.
Results: Health equity definitions largely differed across healthcare organizations, although some definitions were shared and cited the same original source. Few major healthcare organizations reported explicit definitions of RHE, although it was commonly listed as a priority. Preliminary search results of RHE across primary research articles yielded ~2,500 articles. Search results of systematic reviews identified >7,500 citations.
Conclusions: Defining racial health equity is a first step toward establishing actionable interventions to further research and reduce health inequities. This landscape review is part of a larger research project, funded by the Robert Wood Johnson Foundation, investigating racial health equity in systematic reviews.
Patient, public and/or healthcare consumer involvement: This study and the parent project were reviewed by multiple diverse stakeholders in healthcare.

Background: In 2013-2014, our Cochrane systematic review relating to physical rehabilitation after stroke was produced with involvement of stroke survivors, carers and health professionals (“stakeholders”) who made key decisions about the review scope, structure and presentation of results. In 2022-2023, we conducted a major update of this review, again involving key stakeholders to ensure continued relevance and impact of this review. Objective: To describe and reflect on the stakeholder involvement in this Cochrane review of physical rehabilitation after stroke. We will:
• Describe involvement using the ACTIVE framework (who, how, when, level of involvement) and any impact/changes;
• Reflect on what did/did not work well.
Methods: We recruited a group of stakeholders, comprising people with lived experience of receiving or providing physical rehabilitation after stroke in the UK, aiming for a group with a range of different experiences. We held a series of pre-planned meetings, each with a specific goal in relation to informing the review. We held two international webinars to gain international perspectives. Stakeholders provided reflections and feedback.
Results: The stakeholder group comprised three stroke survivors, four carers and seven physiotherapists. To date, we have held four online meetings, supplemented with additional sub-group meetings, recorded presentations and email communication. Two further meetings are planned. A total of 124 people (including patients, carers, professionals and reviewers), from three continents, attended the international webinars. A complex new framework to describe physical rehabilitation following stroke was co-produced. This framework informed the grouping of studies within the review analyses. The group highlighted the need for additional comparisons and subgroup analyses to make the results meaningful and relevant. Wording of key results and implications for practice was co-produced. Barriers and facilitators to successful involvement were identified. For example, online meetings created challenges to equitable participation; use of recorded presentations were an efficient and successful way of keeping stakeholders up-to-date and informed.
Conclusions: Stakeholder involvement influenced the review, ensuring analyses focused on questions that were meaningful and relevant and that results were clearly communicated to end users of research. This example contributes to a growing body of evidence about methods of involving stakeholders in Cochrane reviews.

Data Sharing Making data gathered during systematic reviews (SRs) publicly available could reduce unnecessary duplication, support analyses for secondary research questions, and facilitate methodological research. However, only a third of recent SRs have data availability statements, and among those, only 13% have downloadable data from a website/repository. What Is SRDR+? The Systematic Review Data Repository Plus (SRDR+) is a free, online platform for screening and extracting, archiving, and sharing data gathered during SRs. Since inception, it has had 10,083 user accounts from 117 countries. Data have been made publicly available (open access) for 225 SRs containing 19,898 studies on numerous topics. The US Agency for Healthcare Research and Quality (AHRQ) funds SRDR+. Key Features of SRDR+ SRDR+ is an open-source relational database that allows machine learning–informed screening, flexible data extraction and risk of bias assessment, data adjudication, and data sharing. Newly enhanced SRDR+ features include (1) a data comparison tool to help adjudicate duplicate independently extracted data and (2) a single location for handling citations for abstract screening through full-text screening, data extraction, and data export. SRDR+ can also share data that were not initially extracted into SRDR+ (i.e., data can be imported from other platforms). “FHIR-ed up” Data Sharing Fast Healthcare Interoperability Resources (FHIR) is a standard for electronic exchange of health data. The Evidence-Based Medicine on FHIR (EBMonFHIR) initiative is developing standards for sharing SR data, such as citations, study design, population/intervention/exposure/outcome definitions, results, risk of bias, and certainty of evidence. We are enabling SRDR+ to fully comply with FHIR to facilitate data sharing and allow interoperability of SR data with other platforms, including for guideline development (e.g., MAGICapp) and study registries (e.g., ClinicalTrials.gov). SRDR+ FHIR-enabled resources will be available via a representational state transfer (RESTful) application programming interface (API), which facilitates backwards compatibility, and will be human readable through the Fast Evidence Interoperability Resources (FEvIR) platform. Plan for the Colloquium Presentation We will provide an overview of SRDR+ and demonstrate progress in making SRDR+ fully FHIR enabled. Patient Involvement Although no patients were involved in this abstract, various SRDR+ projects have involved patient collaborators and contain data for patient-important outcomes.

Background: A key question for meta-analysis is to reliably assess whether treatment effects vary across different participant subgroups. Traditionally, these interactions have been estimated using approaches known to induce aggregation bias, so we previously recommended a within-trial approach to provide unbiased estimates for binary or ordered-categorical patient-level treatment-covariate interactions. However, patients, clinicians, and policymakers need reliable estimates of treatment effects within specific covariate subgroups, on relative and absolute scales, in order to target treatments appropriately, which estimation of an interaction effect itself does not provide. Objective and
Methods: Our objective is to describe further developments to the “within-trial” framework by providing practical methods to (1) estimate within-trial interactions across two or more subgroups; (2) estimate subgroup-specific (“floating”) treatment effects that are compatible with the within-trial interactions and make maximum use of available data; and (3) clearly present these data using novel implementation of forest plots. We describe the steps involved and show how the methods can be applied using detailed aggregate (“summary”) data either extracted from trial publications or obtained directly from trial authors. We demonstrate the within-trial framework by applying it to two examples taken from previously published meta-analyses in which detailed aggregate data were available.
Results: In our first example, a meta-analysis of the effects of interleukin-6 antagonists on outcomes for patients hospitalised with COVID-19, we show how the method can be used for a binary covariate. Our second example, a meta-analysis of the effects of postoperative radiotherapy on survival of patients with non-small cell lung cancer, allows us to demonstrate the framework for a categorical covariate with three levels. We demonstrate how to implement these methods using a newly developed command (metafloat) in Stata.
Conclusions: Our within-trial framework allows straightforward estimation of a range of within-trial treatment-covariate interactions, compatible subgroup-specific treatment effects, and corresponding forest plots to clearly and effectively demonstrate how treatment effects differ across patient subgroups. Patient, public, and/or healthcare consumer involvement: One patient and one member of the public are involved in the dissemination of the results of this study, including advising on how best to present the methodology to various audiences.

Background: The COVID-19 pandemic has created significant challenges for healthcare, especially the increase in health-acquired infections (HAIs), which has put significant pressure on the development of social and economic policy in China. At present, the number of HAIs-related research and policy reports is gradually increasing and gaining attention from different stakeholders. At the same time, the current scope of HAIs research, translation, and implementation in China is unclear and cannot further effectively support evidence-informed decision-making.
Objectives: To evaluate the current evidence ecosystem of HAIs prevention and control in China and identify key issues and challenges to improve it in the future.
Methods: We established a national and international multidisciplinary working group. There were two stages for this evaluation. First, we systematically searched seven main databases (PubMed, Web of Science [WoS], Embase, Cochrane Library, China National Knowledge Infrastructure [CNKI], Virtual Interactive Platform [VIP], and Chinese Biomedical Database [CBM]) for all HAIs-relevant documents such as clinical trials, models, reviews, guidelines, policies, and implementation systems from their individual inception to Feb 20, 2023. Reviewers working in pairs and independently selected documents according to the inclusion criteria and extract information. For included documents, according to their type, their evidence quality was assessed. Second, referring to the MAGIC (Making Grade the Irresistible Choice) evidence ecosystem, reviewers evaluated the completeness of the HAIs prevention and control ecosystem in China through the processes of evidence production, synthesis, dissemination, implementation, and evaluation.
Results: Our search identified 28,126 documents. The detailed results will be shown at the conference.
Conclusions: Relevant research and documents on HAIs are growing rapidly. However, repeated research in the same field leads to serious research waste and an incomplete evidence ecosystem. It is suggested that policymakers, stakeholders, and researchers should refer to the current evidence ecosystem before making decisions to fill the research gap.
Patient, public and/or healthcare consumer involvement: None.

Background: The conduct of systematic reviews and other evidence syntheses often involves narrowly focused questions addressing proximal and individual behavioral causes for health outcomes, without considering that distal social determinants of health deeply influence health behaviors and health outcomes. Addressing pervasive racial health inequities, often driven by social determinants and structural racism, requires disrupting research norms by centering marginalized groups. Evidence synthesis guidance does not routinely require incorporating racial health equity across review processes, and a national consensus on how to center racial health equity in evidence syntheses is currently lacking. A key aspect of rethinking methods in the context of racial health equity is revisiting the purpose, nature, and outcomes of engaging stakeholders—including patients, clinicians, community organizations, purchasers, payers, health systems, policy makers, and training institutions—in reviews.
Objectives: (1) To systematically review and characterize current practices in engaging stakeholders in evidence syntheses of interventions to improve racial health equity and (2) to compile and synthesize extant guidance for engaging diverse stakeholders in evidence syntheses.
Methods: We searched MEDLINE and the Cochrane and Campbell Collaboration databases, relevant synthesis and stakeholder organization web sites, and reference lists of salient papers. We included evidence syntheses addressing effects of interventions that aimed to improve racial health equity or methods/guidance documents that addressed approaches for engaging diverse stakeholders in syntheses. Two reviewers independently screened citations. One reviewer will conduct data extraction and quality assessment, with verification by a second reviewer. We will conduct descriptive analysis of key characteristics related to engagement.
Results: Our database and additional searches identified more than 7,500 citations. Screening and extraction are ongoing, but preliminary signals suggest that few reviews of racial health equity interventions explicitly report involving stakeholders. Guidance relevant to stakeholder engagement typically does not directly address engaging racially or ethnically diverse stakeholders but does speak to general equity and ethics considerations.
Conclusions: This review of stakeholder engagement specific to racial health equity, part of a larger Robert Wood Johnson Foundation-funded project, will help advance our understanding of current and needed guidance for involving racially/ethnically diverse stakeholders. Patient, public, and/or healthcare consumer involvement: Protocol review by technical experts.

Background: Approximately 40% of people with dementia experience depression. Whilst psychological interventions are effective for the treatment of depression in dementia, access remains low. A behavioural activation self-help intervention, with support to the person with dementia to use the intervention provided by an informal caregiver, who themselves receives guidance from an occupational therapist (or other trained healthcare professional), may represent a solution. Objective: To develop a behavioral activation intervention targeting depression among people with dementia for the Swedish context together with key stakeholders.
Methods: Semistructured interviews and focus groups were held with people with dementia (n=8), informal caregivers (n=19), healthcare professionals (n=18), and nongovernmental organisations (n=7) and analysed using manifest content analysis. A Public Advisory Group, consisting of informal caregivers of people with dementia (n=4), worked alongside the research team to support the interpretation and sense-making of research findings and codesign of the intervention.
Results: An overarching theme, ‘Tailoring and flexibility’, resulted from interviews and focus groups. Stakeholders expressed a need to adapt intervention material to increase relevancy and representativeness by (1) adding multiple case stories to illustrate different life situations, age groups, and ethnic backgrounds and to increase relevance to Swedish society and culture; (2) designing new illustrations, as proposed illustrations were perceived as old-fashioned and reinforced ageing and dementia stereotypes; and (3) reducing text to minimize treatment burden. Stakeholders also expressed a need for flexibility concerning intervention delivery and expressed a need for choice concerning (1) location of guidance sessions to enable face-to-face sessions to be delivered in a familiar, safe and convenient environment (e.g., home or familiar community setting); (2) mode of guidance (e.g., face to face, telephone, online); and (3) amount of guidance (e.g., frequency and number of guidance sessions).
Conclusions: Results informed the development of a tailored intervention, optimised to improve acceptability, feasibility and relevancy for people with dementia and depression. A planned feasibility study will further examine feasibility and acceptability of the intervention. Public involvement: Our Public Advisory Group has worked closely together with the research team to support the interpretation and sense-making of research findings and codesign the intervention to increase acceptability and relevancy of the intervention.

Background: Clinical practice guidelines are a critical tool for guiding physicians in clinical practice. Guidelines have been evaluated from different perspectives using various tools. However, the existing evaluation tools have several limitations. First, these tools do not address some key elements of guideline quality, such as guideline applicability, transparency of the processes and methods used for development, and prospective registration. Second, some of the evaluation tools have not been adequately assessed for reliability and validity. Third, most evaluation tools have a limited scope such as methodological quality, reporting quality, or implementation. Thus, a comprehensive evaluation of a guideline is time-consuming because it requires the use of multiple tools encompassing different dimensions. In addition, there may be an overlap of items across different evaluation tools. Fourth, interpreting the results of multiple tools in combination and comparing them across different guidelines is challenging.
Objectives: To overcome these barriers and to improve the quality of Chinese guidelines, we formed a working group to develop a unified, comprehensive, and practical evaluation tool named STAR. Methods and
Results: A scoping review was developed to formulate the initial items related to the three dimensions of scientificity, transparency, and applicability; two rounds of Delphi expert survey resulted in a total of 39 items grouped into in 11 domains. Based on a hierarchical analysis of the results of the importance survey, each domain and each item were given weights that reflect their relative importance. Finally, a consensus on the final STAR rating tool was reached in an expert consensus meeting. The tool was tested and found to be reliable, valid, and easy to use.
Conclusions: STAR may be the first tool that uses scientific approach to assign different weights to the domains or items of guideline evaluation, and it is applicable to registered guidelines that have voluntarily applied to the research center for ranking and providing relevant supporting materials.

The STAR Tool.pdf

Background: There is an increased recognition of the value of coproduction and working in partnership with stakeholders to produce trusted and equity-sensitive evidence syntheses. Stakeholders are individuals or groups who are impacted by health research including patients, caregivers, the public, policymakers, healthcare providers, and program managers.
Objectives: (1) To summarise the equity considerations included in existing frameworks and guidance for conducting evidence syntheses in partnership with stakeholders; (2) To map evidence on the impact of stakeholder engagement in conducting evidence syntheses across equity characteristics; and (3) To develop a logic model for equitable partnerships with stakeholders when conducting evidence syntheses.
Methods: This study was conducted as part of the ongoing work of the Multi-Stakeholder Engagement (MuSE) Consortium, an international group of over 120 members representing stakeholders, such as patients, healthcare providers, policymakers, research funders, and guideline developers from various organizations (e.g., WHO, GRADE, AHRQ, RAND, JBI). We conducted a scoping review and systematic evidence map. We searched six bibliographic databases (2010-2022) and grey literature for publications describing guidance or evaluations of stakeholder engagement during an evidence synthesis. We extracted data in duplicate and identified equity considerations using the PROGRESS+ equity framework. We engaged stakeholders in the development of our logic model and preliminary recommendations.
Results: This presentation will include preliminary recommendations for equitable partnerships with stakeholders when conducting evidence syntheses, including: (1) Identification and recruitment of diverse stakeholders, (2) Power-sharing and equitable decision-making during review conduct, (3) When and how to engage stakeholders to promote equity-sensitive syntheses, and (4) Potential impacts and outcomes of stakeholder engagement in evidence syntheses on health equity. Conclusion: The findings from this project will inform the upcoming MuSE Consortium guidance for stakeholder engagement in evidence syntheses and will be integrated into training delivered by the Cochrane Equity Thematic Group. The results will be particularly relevant and important for patients, as the guidance will be coproduced with their leadership and will promote their engagement in conducting reviews to ensure that evidence used for health decision-making includes equity considerations. Patient, public, and/or healthcare consumer involvement: The MuSE Consortium includes stakeholder coleads representing patients, the public and consumers.

Background: Involvement of end users in the production of reviews has mostly been limited to consultative review of researcher-authored protocols with involvement in outcome identification far less common. To address this gap, for a recent Cochrane review on suicide and self-harm prevention interventions in education settings, we collaborated with young people (YP) to codesign review outcomes that were relevant to them.
Objectives: The key objectives were to understand how our codesigned outcomes compared with outcomes that are usually included in self-harm intervention trials and the extent to which YP have been involved in the design and conduct of the trials included in the review.
Methods: Five participatory codesign workshops were completed with 32 YP to codesign outcomes and reflect on how these outcomes compared with those measured in self-harm intervention and prevention interventions. The codesigned outcomes included in the review were better for more coping skills and safer environments (i.e., more acceptance and understanding at home/school). We did not further refine these into more narrowly defined outcomes through the researcher/clinician lens but took a scoping approach and included all constructs relevant to them.
Results: YP offered notably different views about what the most important review outcomes should be, emphasizing the broad diversity of experience and recovery. Importantly, the outcomes that they codesigned highlighted significant idiosyncrasy in what YP perceive as helpful and pointed to important targets of intervention. They also highlighted the importance of moving away from clinical, deficit-based and individually focused outcomes to more holistic and strengths-based outcomes. In the 51 trials included in the review, a large range of constructs relevant to the codesigned outcomes were used. However, very few trials measured the impact of improvements in these constructs on self-harm. Involvement of YP in designing these trials varied significantly but was very limited in most studies. Conclusion: The findings clearly elucidate the lack of alignment between what recovery from self-harm means (in terms of outcomes) for young people between YPs and researchers. Thus, involvement of YP in the ‘upstream’ stage of research is essential to ensure that review outcomes measure what is actually relevant to them.

Background: The Norwegian Institute of Public Health’s (NIPH) machine learning (ML) implementation team within the Cluster for Reviews and Health Technology Assessments sprung from the need to produce more health technology assessments and systematic reviews, faster, during the COVID-19 pandemic. We introduced and scaled up ML among nonspecialists through using off-the-shelf software, beginning with a 6-month team whose mandate was to explore potential benefits of ML. We then continued with a team to build acceptance and use of the tools through teaching and peer support.
Objectives: To present our experiences with starting up and implementing an ML team for evidence synthesis. More specifically, we will present key elements of our team’s strategy, including: - Embedding process and performance evaluations into existing commissioned reviews (i.e., how we “tested” ML functions within ongoing work) - Creating tailored training materials and providing peer support - Onboarding new team members and keeping up to date - Managing workflow changes - Fostering acceptance and implementation of ML
Conclusions: We will conclude by discussing lessons learned and the role of a nonspecialist, implementation-focused ML network or working group in providing training to evidence synthesis organizations.

Background: A CI in an updated meta-analysis may not have the expected coverage if the investigator does not account for whether the earlier meta-analysis failed to reject the null hypothesis.
Objectives: To discuss the method of likelihood ratio meta-analysis (LRMA) in relation to Cochrane’s usual method of meta-analysis.
Methods: A likelihood ratio–based approach is used in a meta-analysis to pool data from separate studies to quantitatively assess where the total evidence points. A log-likelihood ratio function is used for the measure of association in each study. Those functions are summed to obtain a combined function from which a total effect estimate, and ‘intrinsic’ confidence interval, can be retrieved. Results can be presented in a familiar forest plot format. An LRMA is demonstrated using an example of the influence of a concomitant conventional synthetic disease-modifying antirheumatic drug (csDMARD) on adherence to biologic DMARDs (bDMARD) in rheumatoid arthritis. The example includes population-based cohort studies of adult patients with rheumatoid arthritis between January 2007 and March 2014 in five Canadian provinces, and the US IBM MarketScan database. The outcome of discontinuation of bDMARD therapy is compared between patients who are concomitant versus non-concomitant users of csDMARDs.
Results: The study population comprises 20 221 new users of bDMARDs: those using adalimumab (7609), etanercept (9809), abatacept (1024) or infliximab (1779). Overall, concomitant use of csDMARD therapy is not statistically significantly associated with reduced discontinuation of bDMARD treatment in a random effects LRMA in these patients (hazard ratio=0.90, 95% intrinsic CI=0.79-1.02). The association is statistically significant when using usual random effects meta-analysis (95% CI=0.81-0.99). In the hypothetical scenario where the IBM MarketScan data were added after the original analysis, the 95% intrinsic CI remains unchanged at 0.79-1.02, but the 95% CI becomes uninterpretable.
Conclusions: LRMA yields the same point estimate as a usual meta-analysis but with a 95% intrinsic CI that is wider than the traditional 95% CI. The intrinsic CI is more readily interpretable. Further, with LRMA, there is no need to account for previous statistical significance in an updated analysis.
Patient, public and/or healthcare consumer involvement: None.

Background: Specifying questions and criteria at the level of each synthesis can enhance the transparency of reviews and help ensure that reviews address questions of importance to decision-makers. Version 6 of the Cochrane Handbook for Systematic Reviews of Interventions introduced the concept of ‘PICO for each synthesis’ to bring greater focus on the need to plan and report details of the synthesis questions addressed within a systematic review. The results of a systematic review are ultimately determined by these synthesis questions and the decisions authors take in deciding which studies are eligible to answer each question. Without changing the review eligibility criteria, a synthesis can be structured to address different questions (e.g., broader or narrower) simply by grouping interventions, outcomes, or populations differently. Yet this level of specification is uncommon and can be challenging in broad systematic reviews such as those commissioned to address policy questions.
Objectives: To describe our approach to question specification in a broad systematic review commissioned to inform government decisions about health insurance rebates, including:
• Use of an analytic framework and staged consultation to pre-specify a proposed structure for synthesis, agree on the final PICO for each synthesis, and triage studies for analysis.
• The benefits of this approach for decision-making, workflow, and relevance of the synthesis.
Methods: This was 1 of 16 reviews each examining the effects of a natural therapy in any population (condition, injury, risk factor) on any health outcome. Following guidance in Chapter 3 of the Cochrane handbook, we prespecified a framework with proposed population and outcome groupings for synthesis (https://doi.org/10.1186/s13643-022-02015-1). We compiled an aggregate list of populations and outcomes from included studies organised by the framework. Populations and outcomes were prioritised by the commissioners, enabling us to finalise the synthesis PICOs and triage studies for meta-analysis.
Results: The synthesis examined effects on six outcomes (pain, fatigue, sleep, mental health, quality of life, physical function) from 234 studies categorised into 20 population groups. For patients, the approach underpins transparent synthesis of evidence commissioned to inform decisions that affect their health. Patient, public, and/or healthcare consumer involvement: through the commissioner (www.nhmrc.gov.au/health-advice/all-topics/complementary-medicines/natural-therapies-review).

Background: Evidence synthesis guidance does not routinely require incorporating racial health equity across synthesis processes, and a national consensus on how to center racial health equity in evidence syntheses is currently lacking. Diverse stakeholder voices—from patients, clinicians, community organizations, researchers including systematic reviewers, purchasers, payers, research funders, health systems, policymakers, and training institutions—can inform new guidance on the processes, methods, analyses, and dissemination of systematic reviews that center racial health equity.
Objectives: To obtain diverse stakeholder perspectives on (1) definitions of racial health equity; (2) optimal scope, team composition, methods, interventions, dissemination, and translation approaches that facilitate centering racial health equity in systematic reviews; and (3) measures of success and priority areas for centering racial health equity in systematic reviews.
Methods: We will conduct 25 to 35 one-on-one interviews with stakeholders representing varied perspectives. We will use both deductive and inductive approaches to qualitative analysis linking codes to our research questions and formulating cross-cutting themes across multiple interviews and stakeholder groups.
Results: Preliminary results point to key nuances in how racial health equity is defined. Stakeholders support centering racial health equity in systematic reviews, particularly when wide racial disparities are documented. While reviews should continue to focus on effectiveness of interventions, stakeholders also point to the importance of systematically assessing the context and implications of disparities in etiology, access to care, and uptake of interventions. Stakeholders suggest broadening evidence inclusion to qualitative data to have a mixed synthesis review process that provides additional context to racial health inequities. Stakeholders also suggested expanding systematic review methodology to require analytic frameworks to frame questions in ways that consider racial and ethnic centricity. In terms of stakeholder involvement, stakeholders emphasized including those with lived experience in review teams from the beginning of a review and compensating them for their contributions.
Conclusions: This qualitative exploration of stakeholder perspectives, funded by the Robert Wood Johnson Foundation, will help contextualize our understanding of current practices, gaps in methodological guidance, and future research directions for centering racial health equity.
Patient, public and/or healthcare consumer involvement: Included among interviewed stakeholders.

Background: Patient and public involvement and engagement (PPIE) is an expectation when conducting research, including Health Technology Assessments (HTA), and is increasingly expected as a key component of evidence syntheses. When determining outcome measures for a proposed HTA evaluating nonsurgical interventions for treating pelvic organ prolapse (POP) in women, we wanted to ensure our research was of most relevance to women with the condition. Here, we describe the methods we used to collaborate with women with POP.
Objectives: To describe the PPIE methods that informed the scope of our evidence synthesis HTA proposal.
Methods: In November 2021, we held a workshop with women with POP to discuss our research question, its relevance to women with the condition, and what interventions and outcome measures they would like to see included in our evidence synthesis. Following this, we designed a survey using Qualtrics, which was distributed via PPIE networks and patient groups in the UK in May 2022. From this, we derived an initial ranking of outcomes and time-points. We brought these results to an online workshop with women with POP hosted on Zoom in December 2022. Here, we gave a background to the proposed project and used Mentimeter to derive a final ranking of outcome measures and follow-up times of most importance to attendees.
Results: Following our process, four outcome measures of most relevance to women with POP were chosen for the proposed project. Long-term follow-up (at 12 months or more) was identified as the most important time-point at which to assess change in these outcomes. Our discussions within the workshops also identified other areas through which the proposed project could be made more relevant to women with POP.
Conclusions: The involvement of women with POP in our proposal has greatly enhanced the relevancy of our research to patients and the public. People undertaking Cochrane reviews may wish to consider adopting or adapting this approach to help ensure their research is of most relevance to their population of interest. Patient, public, and/or healthcare consumer involvement: PPIE was conducted throughout our work.

Background: There has been a growing interest in using Health Technology Assessment (HTA) as a tool for supporting decision making in health policies. However, the complexity of HTA methods and a lack of local expertise have limited its usage in many countries. The World Health Organization has taken measures in recent years to encourage countries to conduct and use HTA, including through resolutions from the Eastern Mediterranean (EM) Region Regional Committee, which in 2019 called for increased investment in strengthening national institutional capacities for evidence-informed policy making through the use of HTAs. Given the limitations in national technical capacities, there is a focus on adapting HTA results from other settings to the national context of countries.
Objectives: To systematically review the tools developed for HTA transferability and evaluate their strengths and limitations.
Methods: The systematic review was conducted between 1995 and 2021 and included studies that introduced tools, methods, and frameworks for transferability of HTA results across jurisdictions. Databases, such as MEDLINE, EMBASE, Cochrane Library, Epistemonikos, Web of Sciences, EconLit, Economic Working Papers Database, Health Economic Evaluation Database, the NHS Economic Evaluation Database, Scopus, and Google Scholar, were searched, along with relevant bibliographies. The data were extracted and assessed by at least two reviewers and synthesized using both tabulation and narrative approaches.
Results: A total of 10,375 documents were evaluated, resulting in 17 studies that met the inclusion criteria. These 17 studies consisted of 13 newly developed tools (methods) that were appraised based on 12 critical elements, including ease of use, rapid screening criteria, and factors affecting transferability. The majority of the models were checklists, with only a few deemed suitable for full HTA. Three models have been validated through published studies, but there is no evidence of utilization in the countries of the EM region.
Conclusions: Although the existing tools provide valuable resources for evaluating transferability, there remains a need for a more comprehensive tool to support decision-makers in low-resource settings considering country context and capacity. Patient, public, and/or healthcare consumer involvement: not applicable.

Abstract on sys rev on HTA transferability tool WHO EMRO. docx.pdf

Background: The advancement of Artificial Intelligence (AI) assisted technologies leave many wondering about the current capabilities, limitations, and opportunities for integration into scientific endeavors. Large language models (LLM) – such as ChatGPT, designed by OpenAI – have recently gained widespread attention with their ability to respond to various prompts in a natural-sounding way. Systematic reviews (SRs) utilize secondary data and often require many months and tens to hundreds of thousands of dollars to complete, making them attractive grounds for developing AI-assistive technologies.
Objectives: To evaluate the responses of ChatGPT to SR tasks to assess applicability, correctness, and potential usefulness to reviewers.
Methods: We gave ChatGPT a set of SR tasks during a live webinar attended by over 400 people and evaluated the responses, considering our own expertise and attendees’ comments.
Results: When tasked with developing an SR question and eligibility criteria, ChatGPT appropriately formulated a structured question based on the requested PICO elements and produced criteria related to PICO, study design, language, and publication dates. ChatGPT failed to write a PubMed search strategy for a given SR question: while the output matched the structural components of a search, it was unusable due to fabricated pseudo-MeSH terms and inappropriate and incorrect filters. When tasked with screening 27 titles for a given question, ChatGPT correctly identified 8/10 eligible articles while also including one ineligible article. When tasked with summarizing multiple abstracts, ChatGPT accurately identified relevant results from 2/3, but misidentified background information from the third as results. In all tasks, ChatGPT’s response required interpretation and expertise to assess validity. Many attendees expressed hesitancy and distrust of ChatGPT for completing SR tasks.
Conclusions: While ChatGPT and LLMs show some promise for aiding in SR tasks, the technology is in its infancy and needs development for such applications. In addition, great caution should be taken by all, particularly non-content experts, to critically evaluate any content produced by LLMs due to much of the output appearing, at a high level, to be valid, while some was erroneous and in need of active vetting. Consumer involvement: The webinar was public; anyone could attend and comment.

Background: The United Nations Sustainable Development Goals were adopted in 2015 to replace the Millennium Development Goals. As we approach the midway point towards the 2030 target date, it is timely to reflect on the contribution of those in the evidence synthesis community to achieving these important global objectives. The Global Commission on Evidence concluded that both coverage and quality of evidence syntheses was uneven, that many of the reviews were out of date, and many did not include studies from low- or middle-income countries.
Objectives: This study sought to determine alignment of published JBI systematic and scoping reviews to the targets of Sustainable Development Goal 3 (Global Health and Well-being) and to understand author awareness of the goals and their relationship to their work.
Methods: A desktop audit of systematic and scoping reviews published in the journal JBI Evidence Synthesis was undertaken followed by a short author survey.
Results: The results of this pilot study indicate that the SDGs are not currently a significant consideration for most authors, but there is a desire to learn more and integrate them into their prioritisation processes. Fundamental to maximising the potential of evidence syntheses and the achievement of the targets set out in the SDGs (particularly SDG-3) are issues relating to policy, partnership, and prioritisation. The results of this pilot clearly highlight the need for the global evidence community to work together in partnership to ensure that policy and practice relevant reviews are being prioritised.
Conclusions: Although the SDGs are not the only driver to influence the prioritisation of evidence syntheses, they are an important consideration and part of our collective social responsibility. Cooperation and coordination of effort to prioritise evidence syntheses would assist in avoiding research waste, duplication of effort, and the conduct of irrelevant reviews. It is time to think about resetting and recommitting our mutual agendas to transform the evidence ecosystem, maximise the limited resources available to us, and truly make a global impact on health outcomes. Patient, public, and/or healthcare consumer involvement: N/A.

Background: Up to 20% of children and adolescents worldwide experience mental health problems. Because young people find it easier to talk to their peers than to adult family members, teachers, or professionals, training them in how they can support each other is very important.
Objectives: The project aimed to develop materials to train children and youth in how to support their peers in the context of mental health problems. Because (very) low certainty evidence about the association between peer support and mental health problems was anticipated, the secondary objective was to develop a formal consensus method to collect good practice input from a broad group of stakeholders.
Methods: We systematically collected evidence about mental health risk or protective factors related to how children/adolescents interact with their peers (Q1), and about educational programs aimed at social skills and mental health (problems) (Q2). Next, a modified Delphi consensus method was developed and statements describing the potential content and characteristics of interventions, based on the identified evidence, were sent out to 23 stakeholders, including content experts and mental health peer workers. All statements for which consensus could not be reached were discussed in two face-to-face meetings.
Results: For Q1, we included 160 observational studies with very low certainty evidence about factors, such as peer support, peer influence, peer rejection/acceptance, friendship quality/intensity, and bullying. For Q2, we included 9 systematic reviews resulting in evidence conclusions of moderate to low certainty on the effectiveness of peer support programs. This resulted in 146 Delphi statements; for 77% of the statements, consensus was reached after a first Delphi round, the face-to-face discussions, and a second Delphi round. Forty-one statements were not withheld. The accepted statements were used to formulate the content and characteristics of peer support interventions.
Conclusions: Because the evidence about the content for our peer support program was of very low certainty, the input of various stakeholders, from academic experts to peer workers, was indispensable. To receive input from all profiles (including end users, to assure relevance for practice) and to obtain formal consensus, we successfully used a modified Delphi consensus method.

Background: Meta-analysis typically estimates a quantity that differs from the implicitly intended estimand; typically, standard approaches estimate the average effect of a treatment for a population of imperfect studies, rather than the true scientific effect that would be measured in a population of hypothetical perfect studies. We advocate for an alternative approach, called response-surface meta-analysis, which models the relationship between study design quality and effect size in order to estimate the effect in the hypothetical ideal study.
Objectives: We demonstrate how we can extend meta-regression to perform response-surface meta-analysis, as well as how resulting estimates can differ from estimates obtained by traditional methods.
Methods: We perform a response-surface meta-analysis by performing a meta-regression with design quality as a covariate and reporting the predicted mean and confidence interval at the ideal design quality. We apply this approach to a simulation study with a known true effect, as well as an empirical example published in the Cochrane Library. Using a univariate measure of design quality in the synthetic example and risk of bias as a proxy for quality in the empirical example, we compare estimates from response-surface meta-analysis with those of traditional approaches.
Results: In the simulation study, response-surface meta-analysis usually covers the true effect within one standard error. In contrast, we obtain overly biased results from traditional fixed-effects and random effects meta-analyses. In the empirical study, although the original analysis favors the intervention, response-surface meta-analysis does not provide strong evidence for the superiority of the intervention.
Conclusions: Response-surface meta-analysis reframes meta-analysis as an endeavor to estimate the true scientific effect that would be measured under a perfect study, rather than to summarize the existing population of imperfect studies. More work is needed to standardize the response-surface approach, such as determining a rating system for design quality. We hope this work will help to encourage clarity on the causal estimand of interest in meta-analysis. Patient, public, and/or healthcare consumer involvement: N/A.

Background: The U.S. Food and Drug Administration (FDA) and the European Medicines Agency (EMA) are responsible for scientific evaluation and approval of medicines. Approval of new drugs is based on the evaluation of individual clinical studies conducted by manufacturers who produce new compounds. Concerns have been raised regarding the rules governing approval of medicines, quality and transparency of evaluations.
Objectives: To critically review regulations for drug approvals in Europe and the U.S. and to ascertain whether the use of meta-analytical approaches may inform regulatory decisions. Mental disorders are used as a paradigmatic example in view of several medicines already available for these conditions.
Methods: We reviewed EMA and FDA regulations for approval of new drugs for mental disorders. Additionally, as case examples, regulatory trials for three recently approved drugs for mental disorders (esketamine, paliperidone 3-months long-acting and cariprazine) will be meta-analyzed to summarize evidence available at the time of approval.
Results: FDA and EMA require at least two “positive” trials showing superiority over placebo, or noninferiority over active comparators, to grant marketing authorization for new psychotropic medicines. The use of meta-analytical techniques is not a requirement. We will present the results of meta-analyses of regulatory trials of esketamine, paliperidone 3-months long-acting and cariprazine and of network meta-analyses including these medicines. Such examples will show that pooling available evidence for efficacy and tolerability outcomes may offer an added dimension to the evaluation of new medicines, especially for comparing new medicines with those already on the market.
Conclusions: Evaluation of new medicines should be complemented by regulatory meta-analyses of all relevant clinical studies performed at the time of approval, including network meta-analysis if feasible, and individual participant data (IPD). This would be especially relevant in areas where several medicines already exist, as it would inform regulatory authorities on comparative efficacy and acceptability of new drugs compared with those already available. Regulatory meta-analyses should ideally be conducted by independent organizations such as Cochrane. Cochrane may work in consultation with regulatory authorities in developing transparent and high-quality overviews of existing evidence to complement data from clinical studies during the regulatory approval process. Patient/public/healthcare-consumer involvement: Results will improve quality of care.

Background: ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. Although it has demonstrated a strong understanding and cognitive ability in numerous fields, its comprehension of topics related to systematic review and meta-analysis is currently not well-understood.
Objectives: To determine ChatGPT’s level of understanding of knowledge related to systematic review and meta-analysis.
Methods: We learned the methods for intervention systematic reviews and meta-analyses systematically from the Cochrane Handbook. Then, the core research team members discussed and identified 10 questions related to systematic reviews and meta-analyses and obtained relevant answers by asking ChatGPT. Three investigators independently assessed the accuracy of ChatGPT’s responses and rated them on a scale of 1-10, with a score of 10 indicating a perfect answer. Finally, the average accuracy of the three researchers for each question was calculated. The primary outcome measure was the total score of the 10 questions, which was calculated as (the mean score of question 1 + the mean score of question 2 + ... + the mean score of question 10) / 100 x 100%.
Results: Ten questions and answers related to systematic review and meta-analysis were generated (Figure). The three researchers scored the 10 questions 91, 86, and 86, respectively. The average score was 87.7. The highest average score was 9.7 for Q4, indicating that ChatGPT provided the most accurate response to that question. The lowest average score was 7.3 for Q9, indicating that ChatGPT’s response to that question seemed to deviate from the truth to a greater extent.
Conclusions: ChatGPT has a relatively high level of understanding of systematic reviews and meta-analysis knowledge, which is limited by the level of knowledge of the respondents, and a study with a large sample is needed to further validate the results. Patient, public, and/or healthcare consumer involvement: None.

Chatgpt Q & A.pdf

Background: Patient and public involvement (PPI) is essential to guarantee that systematic reviews are pertinent and significant to those impacted by a health condition and to those using systematic reviews to guide health policy or practice. Therefore, the BMJ issued a new policy requiring that all submitted manuscripts must include a PPI declaration in their Methods section under a subheading called Patient Involvement in 2014.
Objectives: This study aimed to systematically analyze the proportion, role, function, and reporting quality of PPI in systematic reviews and meta-analyses.
Methods: We will conduct a hand search for all systematic reviews and meta-analyses published in BMJ journals from 2015 to 2022. Two investigators will independently extract data from the “Patient Involvement” section of the systematic reviews. Our analysis will include the proportion, quantity, and role of PPI in systematic reviews. We will also assess, based on the 12 stages proposed by the Cochrane Involving People learning resource, the specific function of PPI in developing questions, planning methods, writing and publishing protocols, running searches, selecting studies, collecting data, assessing risk of bias, analyzing data, interpreting findings, writing reviews and publishing, and disseminating them. Lastly, using the ACTIVE framework and GRIPP2 checklist, we will assess the reporting quality of the PPI in systematic reviews.
Results: We initially included 270 systematic reviews and meta-analyses. Detailed results will be presented at the 2023 Cochrane Colloquium.
Conclusions: The conclusions will be presented at the 2023 Cochrane Colloquium. Patient, public, and/or healthcare consumer involvement: None.

Background: Over the last 20 years, there has been a significant rise in the number of systematic reviews and meta-analyses being published. These types of studies are highly valued as they are believed to offer the highest level of evidence if carried out correctly and can offer clear and trustworthy summaries for clinical decision-making. However, researchers of meta-analyses may encounter the problem of missing mean and standard deviation (SD), often resulting in only median, interquartile range, or range data being available. Unfortunately, these data cannot be directly used for meta-analysis. Some estimation and conversion methods have been proposed, but there is currently no user-friendly tool available based on multiple scenarios of missing SD.
Objectives: The objective of this study was to develop a useful tool that can find sample mean and SD for common scenarios.
Methods: Two researchers extensively reviewed relevant methodologic references such as textbooks and systematic reviews. Based on the work, they compiled a list of possible scenarios in which there were inadequate data for meta-analysis and sought out formulas for finding sample mean or SD in each case.
Results: The working group double-checked the scenarios and corresponding solutions, and the identified 10 primary scenarios can be categorized into two broad categories. The first category involves circumstances in which descriptive statistics of a single group are available (within-group circumstances). The second category involves situations in which effect estimates of two treatment groups are available. Then, the working group developed an Excel tool for estimating and converting the data in each scenario. The solutions in the tool are primarily based on the Cochrane Handbook and a methodology study.
Conclusions: Researchers need to understand the circumstances and solutions of missing data before conducting meta-analysis, and appropriate data processing could improve precision and quality of evidence. Conversely, excluding certain irregular outcome reporting in a meta-analysis may lead to imprecision and biased estimates. Therefore, familiarity with the different scenarios and the tools available to address them is crucial for researchers to conduct a high-quality meta-analysis.

Background: The eCOVID-19 Recommendations Map & Gateway to Contextualization (RecMap) is an online tool containing 7,200 COVID-19 recommendations and is the product of collaboration with more than 40 organizations around the world. Our knowledge mobilization (KM) project is currently supported by the Canadian Institute of Health Research (CIHR) to enhance the reach of the RecMap by engaging with 10 different groups of knowledge users.
Objectives: The project aims to increase the awareness, use, and engagement of the RecMap using a codesign strategy.
Methods: A committee of stakeholder representatives, staff, and investigators was established to develop our RecMap KM plan. The committee meetings allow members to build a shared understanding of goals, provide input on decisions, and foster mutual learning. Our target audience focuses on a range of RecMap beneficiaries and intermediaries across Canada (community leaders working with migrants and refugees, policymakers, media, public health professionals, and Indigenous Peoples) and globally (nondigital public in Cameroon, parents, professionals working with marginalized communities in Slovakia, Cochrane members in Africa, and guideline developers in China). Results (Anticipated Value): A framework, informed by a systematic review of codesign approaches, was developed by our committee (Figure 1) and guides the KM plan for engaging each target audience. The framework contains four components: an aim, principles, activities, and outcomes. Our team is currently conducting unidirectional activities through webinars, storytelling, and media press releases. We are learning about our audiences’ needs through interviews and workshops. The RecMap is also being integrated with other websites, and plain-language tools are being developed and translated. We anticipate that the RecMap user base will increase and that identified barriers and facilitators will lead to RecMap improvements.
Conclusions: A diverse network of knowledge users and a KM framework were established through a codesign approach, enabling the implementation of RecMap-related activities. Our results will inform future disease-specific recommendation maps that are tailored to the needs of potential users.
Patient, public and/or healthcare consumer involvement: To ensure an effective KM plan for our digital health tool, we have engaged stakeholder representatives from all 10 groups throughout the designing, planning, and implementation stages.

KM Framework (4).pdf

Background: In the last decade, various software tools have been developed for network meta-analysis (NMA), but presentation and interpretation of findings from large networks of interventions remain a challenging task.
Objectives: We developed a fully interactive online tool called ‘NMAstudio’, aimed to facilitate the production and visualization of key NMA results through a direct connection between a customizable network plot and all NMA outputs.
Methods: In NMAstudio, users can upload their data and interact with the network diagram by clicking one or more nodes-treatments or edges-comparisons. Based on their selection, different outputs and information are displayed: (a) boxplots of effect modifiers assisting the evaluation of transitivity; (b) pairwise or NMA forest plots and bidimensional plots if two outcomes are given; (c) league tables coloured by risk of bias or confidence ratings from the CINeMA framework; (d) incoherence tests; (e) comparison-adjusted funnel plots; (f) ranking plots; and (g) evolution of the network over time. Pop-up windows containing extra information and guidance are enabled. An access token can also be generated by the user and can be used to save their project and further share it publicly to foster reproducibility.
Results: We illustrate NMAstudio using a comprehensive network of 20 drugs for chronic plaque psoriasis, and we demonstrate how our tool simplifies the visualisation of complex networks and facilitates checking of assumptions as well as interpretation and sharing of results.
Conclusions: In summary, our web application provides a truly interactive, flexible and user-friendly tool to display, enhance and communicate the NMA findings.
Patient, public and/or healthcare consumer involvement: None.

Background: Network meta-analyses (NMAs) provide comparative evidence about the effects of all healthcare interventions for a given condition. However, NMA results can be undermined by flaws in design, conduct, analyses and reporting. Several tools are available for the critical appraisal of an NMA. However, none aim specifically to assess the risk of bias (RoB) in an NMA.
Objectives: To describe a new RoB NMA tool to assess the risk of bias in an NMA, developed using rigorous methodology.
Methods: As an international steering committee, we developed a protocol and made conceptual decisions about the tool’s structure. We developed the tool using a multistep approach: definition of the scope of the tool; conduct a methodological review to compile a preliminary list of concepts related to bias in NMAs [1]; web-based Delphi survey to collate expert opinion regarding the quality of NMAs [2]; an online survey to solicit decision-makers’ needs [2]; and refinement through piloting.
Results: The RoB NMA tool includes three domains: bias arising from the network characteristics/geometry, bias arising from potential effect modifiers, and bias arising from statistical synthesis. The three domains comprise a total of 16 statements which can identify potential limitations of the NMA, leading to a judgement about the risk of bias within each domain (low, high or unclear risk of bias). The tool should be used to identify potential limitations in the way an NMA was conducted that may lead to bias in its results or conclusions. The RoB NMA tool should be conducted as a supplement to the assessment of the risk of bias in the underlying evidence review, for example, using the ROBIS tool. Piloting of the RoB NMA tool is ongoing.
Conclusions: The development of the RoB NMA tool was evidence based and included piloting and refinement. We believe that the RoB NMA tool will allow objective rating of the RoB in an NMA.
Patient, public and/or healthcare consumer involvement: None.

Rferences 2023 colloquium.PNG

Background: Developing trustworthy guidelines requires considerable investment of resources and time. This makes priority-setting of topics and questions a key aspect of guideline development. Explicit and thorough prioritization engaging a range of appropriate stakeholders enhances the relevance, potential utility, and uptake of priority recommendations by end users.
Objectives: To identify priority topics, formulate questions, and select outcomes for a World Health Organization nutrition guideline.
Methods: The guideline steering group and external guideline methodologists partnered to coordinate the prioritisation. We aligned the approach with WHO methods, and good practice for priority-setting, including an explicit and comprehensive process involving stakeholder engagement, information management, consideration of values and context, and opportunities to review decisions. Draft guideline questions were informed by four commissioned scoping reviews, areas of uncertainty needing guidance identified by member countries or implementing organisations, and past WHO meetings, with inputs from the guideline development group (GDG) at an initial scoping meeting. Working groups refined draft questions into proposed PICO formats, which the GDG prioritised according to clear criteria (online survey). Informed by survey results, the most important questions were agreed upon by the GDG at a consensus meeting, after which the GDG scored outcomes online according to importance. All contributors declared interests.
Results: The 16 prioritised guideline questions comprised intervention effectiveness (treatment and prevention), prognostic, and diagnostic questions. Our approach took us iteratively through pre-prioritisation and prioritisation stages, as well as post-prioritisation, in which broad questions required further splitting into several focused sub-PICOs to unpack complexity (e.g., effect modification and baseline risk differences) and enable identification of relevant evidence by systematic reviews for the questions and evidence-to-decision criteria for recommendations. A systematic process enabled the consideration of social determinants of health, equity, human rights, and gender throughout the process, guided by PROGRESS-Plus stratifiers. Documenting the approach aided transparency, and inclusivity facilitated stakeholder buy-in. Conclusion: This priority-setting process was complex, time-intensive, and recursive but clearly valuable and necessary to direct efforts and resources to the most important needs for guidance, representing a step towards improving delivery of evidence-informed care and key outcomes. Patient, public, and/or healthcare consumer involvement: Stakeholders included guideline implementers and researchers.

Background: Organisations such as youth movements or companies regularly request trained volunteers from the Belgian Red Cross to provide group psychosocial support after the occurrence of potentially traumatic events.
Objectives: The Belgian Red Cross’ Social Intervention Service and the Centre for Evidence-Based Practice (CEBaP) have updated the manual that is used to train volunteers in providing psychosocial support according to the principles of evidence-based practice. This manual was revised given that it was previously developed based on expert opinion.
Methods: CEBaP researchers performed a systematic literature search of existing systematic reviews using The Cochrane Library, MEDLINE, Embase, and PsycNET to collect evidence on the effectiveness of acute, brief, and preventive psychosocial or psychological group interventions after potentially traumatic events. Evidence conclusions were translated into recommendations, which were discussed with an expert panel consisting of Red Cross volunteers (i.e., the end users), academic researchers, and practitioners.
Results: We identified eight observational and seven experimental studies providing evidence of low to very low certainty. Previous recommendations needed significant changes based on the evidence conclusions, the input from the expert panel, and existing evidence-based guidelines. These were, however, not in line with the Red Cross volunteers’ previous experiences with providing psychosocial support, which complicated the acceptance and implementation of the new recommendations.
Conclusions: This experience highlights the crucial aspect of building evidence users’ trust in evidence. Apart from involving end users in the expert panel, we believe that trust can be increased by engaging users early on and continuously during the process of evidence-based practice and by explaining the principles, strengths, and limitations of evidence-based methodology at the start of the process. In return, input and feedback from experienced users may provide valuable insights for evidence synthesizers.
Patient, public and/or healthcare consumer involvement: Although these interactions require a considerable time investment, they may benefit all parties, including researchers, evidence implementers, and the target audience.

Background: The increased number of outcomes and competing interventions assessed in a network meta-analysis (NMA) increases the difficulty of interpreting results and hence challenges their use in everyday clinical practice and policy.
Objectives: To facilitate evidence-based decision-making on a clinical topic with multiple outcomes per intervention through a graphical representation of NMA results, the rank-heat plot. We are currently developing the rankheatplot R package and R Shiny app and will disseminate the first version at the conference.
Methods: The rankheatplot R Shiny app can be used in any type of discipline and disease using an NMA of multiple interventions compared in different studies obtained from any type of review (i.e., systematic review, rapid review, overview of reviews). The rank-heat plot was first developed and published in 2016. It has been cited in more than 120 publications (Google Scholar) and has been used in multiple areas such as geriatrics, pediatrics, neurology, and oncology. It is already part of the viscomp R package, which presents multiple visualization approaches on component NMA. We developed an interactive web application at https://rankheatplot.com/ to produce the rank-heat plot when the study-level data for each outcome are available and without performing the NMA analysis outside the tool. The tool can be used for any type of data.
Results: The rank-heat plot allows the fast identification of the most likely best and worst interventions, with respect to their effectiveness and/or safety, in a given outcome. It can also identify interventions that have not been studied for an included outcome. The rankheatplot app currently performs analyses in a frequentist framework importing intervention outcome results from the netmeta R package. In the next version of the tool, we plan to incorporate the option of conducting the analysis in a Bayesian framework, as well as outputs based on clinically important effects (e.g., minimal clinically important difference).
Conclusions: The rank-heat plot summarizes results on all interventions and all outcomes, providing information regarding the ranking of interventions per outcome. The tool can be used to facilitate interpretation and decision-making based on NMA synthesis. No patients were involved in the development of this tool.

Background: In 2019, Cochrane adopted a new tool (RoB 2) for assessing risk of bias in randomised controlled trials. The tool incorporated new advances in bias research and differed markedly from its predecessor in how it was applied. Cochrane introduced a phased implementation of the RoB 2 tool, with mentored support from the Cochrane Methods Support Unit (MSU) to assist authors who were new to using it. The MSU team noticed some commonly occurring misconceptions and misapplications by authors. These could potentially lead to inappropriate assessments of bias which could affect certainty in evidence.
Objectives: To compile a list of 10 top tips to help authors apply the RoB 2 tool.
Methods: From June 2019 to December 2020, the MSU team assessed 144 reviews and protocols and made a record of the most common misapplications and misconceptions in using RoB 2. We made them into a short list of 10 useful hints and tips for any review authors, not just Cochrane authors, new to using RoB 2.
Results: The top tips cover all domains of the RoB 2 tool and are as follows: 1) State the effect of interest. 2) State the outcome to be assessed. 3) Pilot the tool to achieve consistency. 4) Apply the tool to a specific numerical result. 5) Answer all of the signalling questions. 6) Consider whether baseline imbalance reflects problems with the randomisation process rather than chance. 7) Do not assume that a lack of blinding increases risk of bias. 8) Participants changing interventions is not always a cause for concern. 9) Avoid arbitrary thresholds for missing data and consider the analysis used. 10) A ‘statistical analysis plan’ can be a protocol or a trial register entry—but check the date it was registered.
Conclusions: The list of top tips with their explanations can help authors to answer some of the less familiar concepts covered in the new RoB 2 tool as well as offer practical suggestions. We think this will help authors avoid some of the common pitfalls the MSU saw with authors new to using the RoB 2 tool.
Patient, public and/or healthcare consumer involvement: None.

Background: The Exeter Policy Research Programme Evidence Review Facility is commissioned to produce systematic reviews to inform health and social care policy. With the focus and direction of each review determined by the priorities of our government stakeholders, it is vital our processes incorporate perspectives from patients and members of the public.
Objectives: Discuss our experiences of incorporating the perspectives of patients and members of the public into systematic reviews commissioned to inform government policy. A public collaborator will co-present this presentation.
Methods: Patient and public involvement and engagement (PPIE) is embedded within our organisational structure and throughout the review process. A public collaborator is a core member of our team, and we have a standing PPIE group of 15 individuals with an interest in the production and use of evidence syntheses for decision-making. Supported by our PPIE facilitator, they are involved in early scoping and refinement of research requests and development of an involvement plan. Further involvement from individuals with specific topic–related lived experience is sought as projects develop. Training needs of patient and public collaborators are observed with bespoke training to address areas of uncertainty provided by the research team. Reporting of PPIE activities and their impact on the review process and presentation of findings are standard components of all our outputs.
Results: Delivering evidence syntheses to inform government policy involves lengthy negotiations to develop focused research questions, alongside changing research priorities and political sensitivity. In this context, incorporating meaningful PPIE throughout the review process can be challenging. The broad scope of topics and fast-paced nature of the work adds further complexity. Embedding PPIE within our organisational structure as well as having a flexible and responsive approach ensures we maximise opportunities for incorporating patient and public perspectives. Embedding PPIE within our organisation empowers patient and public representatives to have their say on issues which matter to them.
Conclusions: Patients and members of the public make an invaluable contribution towards our work. Although involvement within the process of conducting reviews for policymakers is not always straightforward, their insights help ensure that government policy reflects the needs of individuals using UK health and social care services.

Background: Sub-Saharan Africa (SSA) has the highest under-five mortality rate globally. Evidence-informed guidelines are key in supporting healthcare decision-making. The GELA (Global Evidence, Local Adaptation) project aims to enhance the capacity to use global research to develop locally relevant guidelines for newborn and child health in three SSA countries, either de novo or through adaptation of existing guidelines. To achieve this, we first carried out a priority-setting exercise.
Objectives: To identify priority newborn and child health topics in South Africa (SA), Malawi, and Nigeria that require guidelines.
Methods: We followed good practice for priority-setting, including stakeholder engagement, online surveys, and consensus meetings. We established national Steering Groups (SGs) to help prioritise topics and advise on the process. Members represented government, health professionals, academia, and NGOs; all declared their interests. A variety of stakeholders were engaged via online surveys to rate the importance of topics. Survey results informed national consensus meetings with SGs, in which final priorities were identified. Non-participant observations of SG consensus meetings were conducted to gain insights into processes, dynamics, and contexts.
Results: Initial priority topics were identified and added to online surveys, which were open for 3-4 weeks in 2022. Surveys were completed by 37, 23, and 78 people in SA, Malawi, and Nigeria, respectively; similar proportions of participants completed all sections (SA 66%, Malawi 61%, Nigeria 68%). The surveys identified 9, 10, and 8 topics in SA, Malawi, and Nigeria, respectively; these informed several SG consensus meetings. Through voting or discussion at these meetings and scoping guidelines, the top three priorities, per country, were identified. SG meeting observations revealed similar and different capacity gaps (e.g., guideline development process), stakeholder dynamics (e.g., certain dominant voices), and procedural challenges (e.g., voting resistance). These may have impacted the priority setting process and outcomes.
Conclusions: Dynamic and iterative stakeholder engagement enabled identification of priority topics for guideline development on newborn and child health. Contextualised priority-setting, though highly recursive and time-intensive, is essential, as shown by the limited overlap in topics prioritised. Patient and stakeholder involvement: Civil society organisations were invited to contribute to the priority setting survey.

Background: The extraction, analysis and presentation of results in scoping reviews can be challenging. Inconsistencies and inappropriateness in the analytical approaches undertaken during the analysis and presentation of the data within scoping reviews has been a recurrent issue. Additionally, scoping reviews can include a variety of evidence sources, such as peer-reviewed primary research, reviews, and grey literature, such as guidelines, organizational reports, policies, government documents and blogs. This variety of evidence levels and sources creates challenges when trying to extract, analyse and present the results in a cohesive manner.
Objectives: This presentation aims to clarify the process of extracting data from different sources of evidence, discuss what data can be extracted (and what should not) and how to analyse extracted data, including an explanation of basic qualitative content analysis, and offer suggestions for the presentation of results in scoping reviews.
Methods: Using best-practice examples and drawing on the expertise of the JBI Scoping Review Methodology group, and a member who is an editor of a journal that publishes scoping reviews, this discussion will expand on existing guidance.
Results: Examples of the data extraction process and specifically what can be extracted from the included evidence sources (and what should not) will be presented. The use of basic qualitative content analysis within scoping reviews, including when it should and should not be used, will be showcased. Furthermore, suggestions for the presentation of results in scoping reviews will be discussed.
Conclusions: The extraction, analysis and presentation of findings within a scoping review can be challenging because of the variety of evidence sources that scoping reviews can typically include and the absence of specific guidance for reviewers. This discussion will hopefully address these challenges and guide how to extract, analyse and present data within scoping reviews. It is hoped that scoping review authors will use this guidance to improve the quality and clarity of published scoping reviews and make conducting and reporting scoping reviews easier.
Patient, public and/or healthcare consumer involvement: None.

Background: JBI (formally known as the Joanna Briggs Institute) offers a suite of critical appraisal instruments that are freely available to evidence synthesisers. These instruments have been developed by JBI and collaborators and approved by the JBI Scientific Committee following extensive consultation. Following recent developments within the science of risk of bias assessment, it has been acknowledged that the existing suite of instruments are not aligned to these developments and conflate and confuse the process of critical appraisal with that of risk of bias assessment.
Objectives: Here, we introduce the revised critical appraisal tool for randomized controlled trials (RCTs) and detail the key changes made from the previous iteration.
Results:
Methods: The JBI Effectiveness Methodology Group (EMG) began the update procedure by cataloguing the questions asked in each JBI critical appraisal tool for study designs that employ quantitative data. These questions were ordered into constructs of validity (internal, statistical conclusion, comprehensiveness of reporting, external) following DELPHI-like methods. For questions that were related to the internal validity construct, they were further catalogued to a domain of bias through a series of mapping exercises. Finally, questions were then separated based on whether they were answered at the study, outcome, or result level. Findings: A strength of the JBI critical appraisal instruments has been their flexibility to facilitate assessments of risk of bias following different approaches. However, due to their presentation, using the tools following a domain-based approach was not intuitive to all users. By presenting the questions to the validity of construct they belong and the domain of bias they address, users of the tool are better placed to follow a domain-based approach using these new tools if they choose. The revised instrument also identifies which questions address internal validity and which questions should be addressed at different hierarchal levels.
Conclusions: The revision to the JBI critical appraisal instruments aims to provide greater flexibility to users of these tools. It is expected that this work will increase the usability and applicability of these instruments while maintaining consistency with modern advances in evidence synthesis.

Background and Objective: There is increased interest in nature-based interventions (NBIs) to support mental health. These interventions are complex, operating across sectors, with a variety of activities, populations and environments. Previous research has highlighted poor elucidation of how and why context influences the outcomes observed in NBIs. In the United Kingdom, a rise in social prescribing, where people are supported into community/voluntary organisations, has expanded without concomitant evidence-based guidance to underpin intervention development. This presented an imperative to work with stakeholders to coproduce a nature-based intervention development resource.
Methods: A realist review was conducted to develop initial programme theory and inform a set of stakeholder interviews. Interviews explored identified mechanisms in the context of real-world experience. Subsequently, a workshop series engaging a wider pool of providers was carried out to refine guidance documentation.
Results: The realist review [n=1,343 results identified through database searches (n=5), plus reference scanning of key topic reviews, and grey literature review] identified 10 key mechanisms through which health and well-being benefits are likely to accrue through NBIs (Being outside, Making a difference, Being in a group, Physical activity, Personal achievement, Having fun, Personal growth, Creativity, Caring, Relationship to Nature). Interviews (n=13) helped to define pathways linking these mechanisms to 10 specific intervention activities and four types of outcomes. Preliminary guidance based on this logic model was developed. In workshops (n=4) (44 participants), theoretical exercises allowed providers to consider implementation of the guidance for their particular context and requirements; the results were used to refine the guidance. Conclusion: Coproduction allowed us to integrate the best available evidence with contemporary experience, to produce an evidence-based handbook which has validity for providers and is based on robust programme theory. Relevance and Importance to Patients: The handbook has more than 2,100 downloads, from 60 countries. It has been accessed by activity providers, health professionals, professional institutes, government departments and local authorities. Ongoing commissioned work, testing the feasibility of nature-based activities reached through social prescribing, builds on our programme theory. The aim is to provide patients with access to interventions that are appropriate, effective and safe.

Background: The WHO notes that worldwide obesity has nearly tripled since 1975. In 2016, over 650 million adults were obese. Obesity is a major risk factor for cardiovascular diseases, musculoskeletal disorders, and some cancers. Clinical practice and public health guidelines on the prevention and treatment of obesity rely on the availability of up-to-date synthesised evidence. It is therefore important for Cochrane to produce timely and relevant reviews that can inform such guidelines.
Objectives:
• To identify reviews in the Cochrane Library (CL) related to obesity.
• To identify relevant guidelines and note the evidence gaps highlighted by these guidelines.
• To develop research questions from these evidence gaps.
• To map the list of CL reviews against these questions.
• To consult with stakeholders to prioritise questions for future reviews.
Methods: In Phase 1, we undertook systematic searches for Cochrane reviews published at any time and for relevant guidelines. We identified evidence gaps and formulated research questions for these gaps. We then mapped the existing Cochrane reviews against the formulated questions to identify areas where new reviews or updates of reviews are needed. In Phase 2, we developed an online questionnaire, identified stakeholders, and asked them to prioritise the questions for which Cochrane reviews are needed.
Results: Table 1 describes the outputs at each stage of the process. We identified 36 reviews and 7 protocols, as well as 22 eligible guidelines. We formulated 49 research questions that could be addressed by Cochrane intervention reviews. The Gap Map highlighted that none of the existing Cochrane reviews were both up-to-date and comprehensively answered the developed questions. We therefore included all the questions in our consultation with stakeholders. Thirty-six expert stakeholders participated in an online consultation exercise. Table 2 provides our ‘top ten’ review questions as prioritised by stakeholders.
Conclusions: This project identified priorities for reviews in the critically important area of obesity. We recommend that Cochrane authors address the prioritised review questions, in consultation with evidence users. Patient, public, and/or healthcare consumer involvement: We involved people with lived experience of obesity in our consultation and sought expertise from people from indigenous backgrounds.

Abstract Table 1.png
Abstract Table 2.png

Background: Inappropriate design, conduct and analysis of studies can lead to biased estimates of outcomes. Therefore, risk of bias (RoB) assessment of included studies is a crucial step in systematic reviews. Several tools exist to facilitate RoB assessments. Despite their widespread use, their usability is not well summarised.
Objectives: To review published findings of studies evaluating commonly used RoB tools.
Methods: We included RoB tools listed in the Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis (LATITUDES) Network. We conducted a forward citation search from the primary report for each RoB tool. Two review authors independently screened titles and abstracts and potentially relevant full texts. We included studies with the objective of evaluating the design and/or usability of an included tool. We summarise the structure and features of the different included tools, as well as similarities and differences between the tools. For each tool, we outline included study characteristics and methodology. Employing a deductive approach, one reviewer will use NViVo software to extract information meeting prespecified codes developed by the team (e.g., comments on usability of the tool, specific challenging items, and recommendations for future development). If other relevant information arises, we will move to an inductive approach and add new codes (which will be discussed within the team).
Results: This review is in progress. Results will be presented at the Colloquium.
Conclusions: Risk of bias assessment is an important part of the systematic review process. The findings of this review are expected to inform the development and updating of RoB tools, which in turn may positively affect the robustness of evidence production.
Patient, public and/or healthcare consumer involvement: None.

Background: Storyboarding in research can be used as a dynamic way to engage in the research process and as a form of data collection. Storyboarding can express a story for both participants and researchers to engage in, and it creates an opportunity between various stakeholders to form a shared language.
Objectives: We aimed to test the utility of the qualitative evidence synthesis (QES) analytical method ‘synthesis by storyboarding’ against the lived experiences of young women living with perinatal infections of HIV (YWLPIHIV).
Methods: As part of a larger study to explore treatment adherence for YWLPHIV, we conducted a QES up to November 2021, which included 47 studies, and synthesized 9 storylines (Hendricks et al. 2022). We analyzed the data using ‘synthesis by storyboarding’. As a second phase, we conducted a primary study. First, we interviewed YWLPHIV about their experiences of their treatment adherence, and then we asked them to edit the QES storyboard to reflect their own lived experiences.
Results: At first, most participants sat in silence, then some had highly emotive responses. We found that each YWLPHIV could relate to elements on the storyboard, including loss of parents, anxiety of disclosure, the emotional rollercoaster of adherence as a teenager, and desire to live without fear of judgement. Even though the storyboard was comprehensive, further detail could be added using their stories.
Conclusions: Storyboarding was found to be a useful analytical method in QES. YWLPHIV found the storyboards to be reflective of their own adherence journeys. This innovative visual synthesis created an opportunity for YWLPHIV to engage in review-level findings, make sense of their own stories, and contribute further insights to the identification of evidence gaps.
Patient, public and/or healthcare consumer involvement: Using storyboarding, the YWLPHIV felt included because they were able to engage in and comment on the review, something they thought impossible at first: “Ek het nooit gedink dat ek die hele ding kan lees nie!” (I never thought I would be able to read this whole thing!). We further used the storyboard as a 1mX1m cloth exhibit on which young women placed their own personal objects that shared their story of adherence for public engagement.

IMG-20220827-WA0120.jpg

Background: Scoping reviews (ScRs) aim to comprehensively map available evidence on a particular topic, and both primary studies and systematic reviews (SRs) may be eligible for inclusion. For well-developed topics, primary studies are usually included in the often numerous SRs available, and thus, restricting scoping to SRs may be sufficient, particularly if these include recent SRs. For less well-developed topics, a balance between including SRs and primary studies may be needed to ensure adequate representation of the available evidence, while maintaining efficiency. Objective: To describe a two-tiered search strategy for time-sensitive ScRs where different considerations necessitated tiered searching. The ScR topics included lesser-known interventions with scant evidence or new interventions and settings with an immature evidence base and limited synthesised evidence.
Methods: We developed and applied two-tiered searching in two time-sensitive ScRs for World Health Organization (WHO) nutrition guidelines, supported by an information specialist. Comprehensive a priori frameworks of all possibly eligible interventions were developed. The first-tier search was limited to SRs. Following data extraction, the frameworks were used for gap analyses identifying interventions or strategies not covered by the eligible SRs. These analyses informed the targeted second-tier searches to identify eligible primary studies addressing these gaps.
Results: After including eligible SRs, the frameworks enabled quick identification of eligible interventions or strategies not covered by SR evidence. The first-tier search resulted in 15 eligible SRs (from yield n=492) and the second-tier search in 12 eligible primary studies (from yield n=1,699) for one ScR. For the other ScR, the first-tier search resulted in 69 eligible SRs (from yield n=1,448) and the second-tier search in 24 eligible primary studies (from yield n=545). This tiered approach produced manageable yields and improved efficiency when compared with broad searching for SRs and primary studies across multiple interventions, which often results in prohibitively high yields.
Conclusions: This strategy enabled the efficient production of comprehensive ScRs informing the prioritisation of key guideline questions within time-sensitive policy windows.
Patient, public and/or healthcare consumer involvement: No direct involvement. The ScRs are informing WHO public health nutrition guidelines aimed at achieving the best health outcomes possible for the public.

Background: Healthcare expenditure is growing at an unsustainable rate in developed countries. A recent scoping review identified several alternative healthcare delivery models with the potential to improve health system sustainability.
Objectives: To (1) obtain input and consensus from an expert Delphi panel of policy, clinical, and consumer stakeholders about alternative models they considered most promising for increasing value in healthcare delivery in Australia, and (2) synthesise the evidence about the effects of three priority models.
Methods: The panel first reviewed a list of 84 models obtained through a preceding scoping review and contributed additional ideas in an open round. In a subsequent scoring round, the panel rated the priority of each model in terms of its potential to improve healthcare sustainability in Australia. Consensus was assumed when more than 50% of the panel rated a model as (1) very high or high priority or (2) not a priority or low priority.
Results: Eighty-two of 149 invited participants (55%) representing all Australian states/territories and wide expertise completed round one; 71 completed round two. Consensus on high priority was achieved for 59 alternative models; 14 were rated as (very) high priority by 70% of the panel. Top priorities included models for delivery and coordination of primary and/or secondary care for older adults living in aged care facilities, home versus hospital delivery of intravenous anti-cancer therapy for adults with cancer, and early discharge hospital at home versus in-hospital care for people with various conditions. These priorities have informed three new Cochrane reviews. No consensus was reached on 47 models, but no model was deemed low priority.
Conclusions: Input from an expert stakeholder panel identified healthcare delivery models not previously synthesised in systematic reviews that were a priority to investigate and led directly to new Cochrane reviews. Patient, public, and/or healthcare consumer involvement: Consumer advocacy organisation representatives contributed to the Delphi panel.

Background: RoB 2, the revised Cochrane tool for assessing risk of bias in randomized controlled trials (RCTs), aims to increase the reliability of assessments.
Objectives: To share our experience of using RoB 2 in a systematic review on remote alcohol and drug misuse therapies.
Methods: An experienced review team conducted 53 RoB 2 assessments of 45 RCTs for two substance use outcomes (several studies reported both outcomes).
Results: We experienced the following challenges: 1. Prespecifying the outcome: Our review included heterogeneous studies evaluating complex interventions with a range of outcomes. With no preferred outcome indicated by review commissioners, to prespecify an outcome we had to familiarise ourselves with the studies prior to quality appraisal, despite this not being considered best practice. Recommendation: Guidance should recognise that not all reviews include a homogeneous study set, with a standardised outcome measure. As an alternative to prespecification of outcomes, reviewers could prespecify ‘decision rules’, for example, to select the most reported outcome. 2. Assessing and reporting multiple outcomes: Studies with multiple eligible outcomes, required time-consuming and largely repetitive assessments. Although some aspects of RoB 2 are specific to each outcome included, others apply to the whole study. Recommendation: Adapt RoB 2 to allow for multiple assessments of a single study. 3. Interpreting statistics and trial methodology: RoB 2 demands sophisticated understanding of methodological aspects such as randomization procedures and handling missing data. Recommendation: More examples and scenarios should be included in the guidance. 4. Difficulty navigating the guidance: The application of RoB 2 to our set of complex studies necessitated frequent referral to the training videos and the full guidance, as we found the crib sheet lacked sufficient detail. Repeated rounds of piloting and development of a review-specific guidance document were also necessary to ensure consistency of application. Recommendation: Consolidate guidance into a single, multilayered resource, making it easier to delve deeper into explanations and examples when needed.
Conclusions: Application of RoB 2 is challenging and time-consuming even for an experienced review team. Broad and/or complex reviews may be particularly susceptible to these challenges. We propose user-friendly ways to revise the RoB 2 tool and guidance documents.
Patient, public and/or healthcare consumer involvement: None.

Background: The COVID-19 pandemic highlighted a need for timely, high-quality evidence outputs to inform health and social care decision-making. At the Wales COVID-19 Evidence Centre (2021-2023) we developed an inclusive approach to include stakeholder involvement, including public contributors, across all stages of our research processes (research question prioritisation; rapid evidence synthesis; rapid primary research; knowledge mobilisation) to support evidence-based decision-making. This enabled 19 reports to be referenced in Welsh Government COVID-19 relevant papers including infection control, education policy and tackling gender inequalities.
Objectives: As we transition to the Health and Care Research Wales Evidence Centre (March 2023), we describe our refined collaborative approach to ensure our work addresses the most important policy needs to achieve impact.
Methods: We are conducting a policy mapping exercise in collaboration with the Welsh Government ‘Science Evidence Advice’ and ‘Knowledge Analytical Service’ teams, involving a 45-minute online discussion with policy leads. We explore policy drivers, evidence needs, the type of evidence required (evidence synthesis or primary research) and timeframes. Potential research questions are then considered for adoption onto our work programme. They require a clear pathway to impact, equity and cost considerations and are themed with ‘A Healthier Wales policy’. Prioritisation includes discussions with the Welsh Government evidence teams; our coapplicant team with domain and methods expertise, including public contributors; and our expert steering committee, to avoid duplication with external research groups. This will be followed by a similar evidence-mapping exercise with Welsh NHS and Social Care leads.
Results: We have completed 25/30 meetings; 92 potential research questions have been identified and will undergo prioritisation as above. Most target the long-term policy aims of improved population health and wellbeing, better quality and accessible services, and higher value health and social care, with fewer questions focused on the health and social care workforce.
Conclusions: Our mapping exercise prioritises the needs of evidence users and ensures that evidence addresses priority questions that are important to them. Partnership with Welsh Government evidence providers and public contributors reduces unnecessary duplication and ensures maximum impact. Public involvement: Throughout our processes to ensure the evidence is relevant and important to patients.

Background: Both conventional and new treatment strategies often lack scientific evidence of effectiveness. Still, they demand finite resources. In the Netherlands, healthcare professionals, health insurers and patient advocates work together to systematically prioritise, execute and implement comparative effectiveness research (CER). After results come available, they are promptly interpreted and adopted in clinical guidelines. All Dutch hospitals have organisational infrastructure with which participating in CER and implementation of new knowledge is stimulated.
Description: With the growing demand for healthcare, increasing challenges in staff shortage and new treatment strategies popping up every day, the challenges we face to keep healthcare available and of high quality are tremendous. For the past 5 years, we have worked in a nationwide collaborative to ‘close the quality circle’. We experienced what is needed for different stakeholders to work together and develop tools to enable everyone to contribute. And even though we have made great progress, there is still a lot to learn. Therefore, we need the knowledge and experience from all participants who are working in policy making, patient involvement, healthcare research and management of healthcare organizations. In this presentation, we share the Dutch method of lowering barriers (example ‘Veldnorm’), stimulating participation (web-based monitoring of research) and collaborating with all parties who have a role in healthcare. What was effective in the Netherlands so far? What can we learn from others?
Patient, public and/or healthcare consumer involvement: The collaboration and way of working together in the Netherlands is necessary to keep our healthcare available to all. The Netherlands Patients Federation is one of the contributors to this collaboration. We closely work together with patient advocates in all aspects of prioritising and executing CER and implementing results.

Background: Overviews of reviews were originally conceived by Cochrane to offer a “friendly front-end” to their systematic reviews. Overviews are useful for decision-makers (healthcare providers, policy makers, patients) because they bring together systematic reviews on related topics (e.g., different interventions for a given condition) in a single product. The Cochrane Handbook provides methodological guidance; however, to date, there has been no reporting guideline for overviews. Consistent and transparent reporting is critical to support the production and use of trusted evidence.
Objectives: Describe the development of the Preferred Reporting Items for Overviews of Reviews (PRIOR) reporting guideline and discuss dissemination strategies.
Methods: We followed the approach recommended by the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network. An international team from eight countries and four continents developed PRIOR, including members with various roles (e.g., reviewers, methodologists, journal editors, public, representatives of organizations that produce reviews). Multiple stages occurred: establishing a team and protocol, establishing the need for a reporting guideline through evidence reviews of published overviews, systematic reviews of existing methodological guidance, modified Delphi to identify relevant items for reporting, and development of the guideline.
Results: PRIOR consists of a checklist (27 main items, 19 sub-items), explanation and elaboration document, and flow diagram. A manuscript describing PRIOR was published in the British Medical Journal (doi: https://doi.org/10.1136/bmj-2022-070849). PRIOR is also available through the EQUATOR Network (https://www.equator-network.org/reporting-guidelines/reporting-guideline-for-overviews-of-reviews-of-healthcare-interventions-development-of-the-prior-statement/).
Conclusions: PRIOR fills an important need for an evidence-based reporting guideline to support authors of overviews of reviews. Active efforts are needed to ensure effective dissemination and uptake. Dissemination plans include communication strategies and materials (e.g., infographics, tip sheets, blog shots), education modules (e.g., slide decks), presentations (e.g., webinars, workshops), and contacting editors of journals that regularly publish overviews of reviews to encourage endorsement and use by authors and peer-reviewers. Regular evaluations of the reporting quality of published overviews are important to assess the impact of PRIOR and to identify areas that warrant additional detail or clarity. Wide adoption of PRIOR from overview authors will help in the production and use of evidence by decision-makers. Patient/public involvement: Members of the public or patients were involved in the Delphi, attended consensus meetings, and were co-authors.

Background: Evidence syntheses provide the most reliable form of evidence for decision-makers. A formal assessment of the validity of included studies is an essential component of any systematic review. This requires a formal risk of bias (RoB)/quality assessment tool. Many tools are available, usually designed specifically for different study designs and/or topic areas.
Objectives: To introduce the LATITUDES Network, a library of assessment tools and instruments used to assess data validity in evidence syntheses.
Methods: We are establishing LATITUDES to increase the robustness of evidence syntheses by improving the process of validity (RoB/quality) assessment. Our goals are to increase the use of key tools, help people use tools more effectively, improve incorporation of the validity assessment into the review and disseminate best practice in validity assessment.
Results: LATITUDES (www.latitudes-network.org) aims to improve the overall quality of research syntheses by improving the process of validity assessment. The LATITUDES website will be launched formally at the Colloquium. We will provide a complementary initiative inspired by the EQUATOR Network (www.equator-network.org), an influential international resource to improve the reporting and transparency of health research studies. The LATITUDES network will index validity assessment tools developed for healthcare studies in an online library. Key RoB tools will be highlighted to help reviewers identify the optimum tool for their review. Project outputs and communication will focus around a freely available online platform featuring the library, key tools, toolkits, online training materials and links to courses and events.
Conclusions: LATITUDES aims to be the primary resource networking reviewers to key information when planning and conducting validity assessment for their review.
Patient, public and/or healthcare consumer involvement:

Background: The U.S. Agency for Healthcare Research and Quality (AHRQ)’s Evidence-based Practice Center (EPC) Program produces evidence reports to inform healthcare decision-makers such as funding agencies supporting new research. The National Institutes of Health (NIH)’s Pathways to Prevention (P2P) Program, established by the NIH Office of Disease Prevention (ODP), uses inclusive, transparent, unbiased, and rigorous processes in its workshops to engage U.S. federal agencies, researchers, stakeholders, and the public; identify scientific gaps in disease prevention; and develop a research agenda to move the field forward. Since 2013, the EPC Program has partnered with NIH-ODP by providing systematic reviews for P2P Workshops that support research recommendation development aimed at closing evidence gaps.
Objectives: (1) To describe EPC review and P2P workshop processes used by the AHRQ and NIH to identify and prioritize key research gaps and develop an actionable plan to address them; (2) to provide a summary of previous reviews and workshops including accomplishments resulting from these efforts; and (3) to share lessons learned from this longstanding AHRQ-NIH partnership.
Methods: A descriptive review of past systematic reviews and workshops since P2P program inception was conducted including portfolio analysis of funding opportunities and grants resulting from the AHRQ-NIH partnership.
Results: Eleven P2P workshops, all supported by EPC systematic reviews, were convened since program inception, covering a wide range of topics including opioids for chronic pain, health equity in preventive services, and youth suicide prevention. Accomplishments include 26 high-impact journal publications and several NIH funding opportunities yielding 20 grant awards supported by over $66M in funding aimed at closing evidence gaps identified by the AHRQ-NIH partnership. Recommendations from independent unbiased panels presiding over P2P workshops helped create action plans in further partnership with other federal agencies. Solicitation of public comments on draft evidence reports and panel recommendations allowed transparency and encouraged stakeholder engagement.
Conclusions: Synergies from close collaboration between evidence review producers and research funders can result in a robust strategy to identify crucial evidence gaps that hinder delivery of evidence-based care to patients. An inclusive process that engages unbiased experts, researchers, and the public helps develop research agenda that ultimately improves patient outcomes.

Background: Quality appraisal (QA) tools, e.g., RoB 2, are used to assess the likely risk of bias caused by methodological limitations of studies included in systematic reviews.
Objectives: To consider the utility of standard QA tools and other methodological issues that, whilst ostensibly not reflecting an individual study’s quality, undermine confidence in a body of primary evidence.
Methods: We conducted a systematic review of controlled studies evaluating structural adolescent contraceptive interventions in low- and middle-income countries. We planned a Qualitative Comparative Analysis (QCA) to explore differences between studies we categorised as ‘likely effective’ and ‘likely ineffective’. However, methodological variability undermined our confidence in this categorisation. Furthermore, these methodological issues were not captured by standard QA tools.
Results: We included 17 studies with heterogeneous study designs. Some methodological characteristics, e.g., whether outcomes were self-reported, were captured in standard QA tools. However, others, such as variable lengths of follow-up and different baseline levels of the outcome measure, could not be. Similarly, heterogeneity in the exact outcome measure used and who was included in the sample for that particular outcome all affected the comparability of effectiveness without being methodological limitations per se. Variation in these characteristics influenced our confidence in comparing and categorising the interventions’ effectiveness. These methodological concerns were spread throughout the set of studies, i.e., both the ‘likely effective’ and ‘likely ineffective’ sets.
Conclusions: QA tools do not capture all of the factors that can affect reviewers’ confidence in studies’ findings or the comparability of findings. This is particularly likely for topics lacking standardised measurements, or heterogeneous methods, which is not unusual in the fields of public health and development. Reviewers should be aware of the risks of over-relying on standard tools in determining their confidence in studies’ findings. Although we could have only included a narrow set of studies with comparable methodological characteristics, this would have prevented synthesis and limited our findings. Reviews have a role in highlighting methodological heterogeneity and calling for greater consensus on methods and outcomes among primary researchers. PPI: Two advisory groups were convened. One, comprising global decision-makers and evaluators, met twice. Another, involving Mozambican adolescents, met once.

Background: Involvement of key stakeholders and evidence users in the planning and development of systematic reviews (SRs) improve their overall quality and usefulness, facilitate uptake and reduce health inequities. Synergy among entities influencing or making decisions in health is important to avoid conflict, duplication of efforts and resource waste.
Objectives: To present an example of harmonization in decision-making on disease-modifying treatments (DMTs) for multiple sclerosis (MS) involving the Cochrane Review Group for MS and Rare Diseases of the CNS (CRGMS), the World Health Organization (WHO) Collaborating Centre on Evidence Synthesis and Guideline Production of Bologna, Italy (WHOCC) and the MS International Federation (MSIF), a nonprofit organisation.
Methods: The CRGMS produced four SRs, two with pairwise and two with network meta-analysis (NMA), including 30 DMTs for MS. The SRs served as the evidence base for clinical practice recommendations developed according to the GRADE methodology by two international multi-stakeholder guideline panels (MSIF Off-Label Treatments panel, MOLT and MSIF Essential Medicines Panel, MEMP) appointed by MSIF. A productive interaction between the CRGMS and both guideline panels at the SRs planning and development stage helped to identify outcomes that are meaningful for persons with MS and clinicians. Imprecision in the NMA estimates was assessed within a fully contextualised approach where thresholds between different magnitudes of effect were set by means of health state utility values, determined through scoping reviews and guideline panel judgement. The WHOCC contributed to the evidence base for domains on cost and resource use of the GRADE Evidence-to-Decision framework.
Results: An application for the inclusion of the first-ever DMTs for the treatment of MS in the 23rd WHO Model List of Essential Medicines (EML) was submitted by MSIF and the WHOCC, based on the evidence base that informed rigorously developed recommendations by two highly representative multi-stakeholder panels on the use of labelled and off-label DMTs.
Conclusions: Involvement of key stakeholders in the planning and development stages of Cochrane reviews of intervention, together with mutual collaboration among different actors in the ecosystem of health decision-making, may facilitate Cochrane becoming a knowledge translation-oriented organization.

Background: The Global Evidence, Local Adaptation (GELA) project aims to maximise the impact of research on poverty-related diseases by increasing researchers’ and decision-makers’ capacity to use global research to develop locally relevant guidelines for newborn and child health in Malawi, Nigeria and South Africa. To facilitate collaboration with stakeholders, we adopted an integrated knowledge translation (IKT) strategy within GELA—systematically mapping and engaging decision-makers and stakeholders and ensuring ongoing communication and dissemination. Given limited research on IKT in African settings, we documented our team’s IKT capacity, skills and process for developing and implementing IKT in these countries.
Methods: Six IKT champions and a coordinator formed the GELA IKT Working Group. We gathered information on our baseline IKT competencies and processes within GELA, and opportunities, challenges and lessons learnt, from April 2022 to March 2023 (Year 1). We held (1) five 2-hour Working Group meetings during Year 1 (slides and video recordings); (2) developed process documents (flowcharts and templates); and (3) recorded our experiences with developing and implementing the country-specific IKT strategies.
Results: Figure 1 outlines the process followed to develop and implement country-specific IKT strategies. All KT champions have similar training in evidence-informed decision-making, but their training in IKT varied. KT champions felt it was challenging to carry out a comprehensive mapping of stakeholders, determine stakeholders’ level of interest and influence on GELA using the power-interest matrix, and identify realistic indicators for monitoring the country-specific strategies. Champions drew on existing relationships with national Ministries of Health and other stakeholders within their respective countries to identify new and relevant stakeholders. Implementation of initial stakeholder engagement activities required heterogeneous and iterative approaches to achieve similar KT goals.
Conclusions: Our experiences of adopting an IKT approach within GELA can help colleagues in similar African settings intentionally and systematically plan for IKT, as well as communication and dissemination of project outcomes. The flexible and evolving nature of IKT can help champions and researchers tailor activities and processes according to stakeholder needs and preferences in their context. There is a need for further research on indicators for monitoring and evaluating IKT.

GELA KT_Cochrane Colloquium Figure1_BMS27Feb23.pdf

Background: Overlap of primary studies among systematic reviews (SRs) is a key methodological challenge for overviews. There are limited reports of the methods being used to deal with overlap by overview authors. The corrected covered area (CCA) is a widely accepted method to assess overlap, but no detailed CCA assessment of a representative sample of overviews has been made using an overall and a pairwise approach.
Objectives: To describe the approaches used to deal with the overlap within overviews, and to estimate and compare the assessment of overlap using an overall and a pairwise approach.
Methods: We searched PubMed for overviews published in 2018. Two independent authors conducted the screening process. We described the strategy used for assessing overlap and calculated overall and pairwise CCA for each overview using the GROOVE tool.
Results: We analysed a random sample of 30 out of 89 eligible articles (Figure 1). Eleven did not address the overlap. Of the remaining 19, the most common strategies used were visual representation (11/19) and discussion of overlap as a limitation (10/19). Median overall CCA among the included overviews was 6.7%. The pairwise analysis showed that 52.8% of SR pairs had slight overlap, whereas 28.3% had very high overlap. Our analysis showed that 8/30 overviews with overall slight or moderate overlap had at least 20% of pairwise nodes (pairs of SRs) with high overlap, and 2/30 overviews with high overlap had at least 20% of pairwise nodes with slight or moderate overlap (Figure 2). Figure 3 provides an example of an overview with moderate overall overlap and a high proportion of highly overlapped pairs of SRs.
Conclusions: Reported strategies for addressing overlap vary considerably among overview authors. The pairwise approach for assessing the CCA revealed highly overlapped pairs of SRs in overviews with overall slight overlap and vice versa. We encourage authors to complement the overall CCA assessment with a pairwise approach.
Patient, public and/or healthcare consumer involvement: No patients or consumers were involved in this research.

ABSTRACT FIG 1 COCHRANE.png
ABSTRACT FIG 2 COCHRANE.png
ABSTRACT FIG 3 COCHRANE.png

Background: Individual participant data meta-analysis (IPD MA) of randomised trials (RCTs) is considered the gold standard for evaluating whether an intervention is beneficial and further to identify which individuals benefit most from the intervention. However, IPD requires considerable time and resources from both reviewers and trial investigators. It is therefore important that IPD is used optimally. Objective and
Methods: Our objective is to elucidate some key considerations and possibilities for optimal IPD use throughout the entire course of an IPD MA project using an example to demonstrate the potential benefits this brings. Firstly, it is critical that appropriate and complete data is collected, thoroughly checked, and documented. Units and coding systems, precise outcome definitions, and covariate adjustment factors should be harmonized across trials. Prior to analysis, missing data, power for treatment-covariate interaction analysis, design complexities, and non-proportionality of hazards (for time-to-event outcomes) should be considered at the trial level. Considerations at the meta-analysis level may include scale of effect measure (e.g., relative vs absolute), avoidance of aggregation bias when analysing treatment-covariate interactions, and consideration of multiple interactions and/or higher-order interactions. Finally, results should be presented and described clearly so as to be understood by different stakeholder groups. All methods should be detailed in advance within protocols and analysis plans.
Results: In our example, an IPD MA of RCTs investigating the long-term benefits of adding docetaxel chemotherapy to standard of care for men with metastatic hormone-sensitive prostate cancer, we demonstrate considerable gains from maximising the use of collected IPD. This includes identification of clinically important complex interactions which may not have been identified using simpler methodology.
Conclusions: Given the considerable effort, resources, and time it takes to collect IPD, it is important to plan, check data, and undertake analysis in a manner which fully maximises its potential. Considering detailed investigations at the trial level can greatly inform the eventual MA, bringing substantial benefits. Patient, public, and/or healthcare consumer involvement: Two patient representatives have been involved in the design, interpretation, and dissemination of our example meta-analysis. Both are supportive of the importance of maximising the value of IPD in our MAs going forward.

Background: There is substantive variation in how living systematic reviews (LSRs) are reported. An extension to the current Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2020 statement is needed to better address the unique aspects of LSR reporting.
Objectives: To develop an extension of the PRISMA 2020 statement for LSRs.
Methods: We followed the EQUATOR Network’s guidance for developing health research reporting guidelines. The methodology included a literature review to identify possible new items or elements, a survey of stakeholder groups’ representatives on the proposed additions, and online discussions with those representatives. We then drafted the checklist extension and associated flow diagrams for final feedback and approval.
Results: PRISMA-LSR includes four new items on (1) living mode parameters, (2) changes to the methods, (3) changes to the results, and (4) authors and their roles. It also includes a number of new elements under existing PRISMA 2020 items on identification of the report as ‘living’ in the title, versioning, justification for adopting the living mode, plans for retirement from the living mode, updating collected data and risk of bias information for a previously included study, analytical methods specific to the living mode, a description of what triggered the current version, illustration of results of the search and selection processes of in the different versions, limitations related to the living mode, changes since the preceding version to the implications of the results, planned changes to review methods in upcoming versions, changes since the preceding version in sources support, competing interests, and accessibility of data, code, and materials. The extension also addresses the LSR status which serves to inform about the status of the review in interim period between published versions. The checklist is designed for full reports, a modular approach, and is designed to allow flexibility to users.
Conclusions: PRISMA extension for LSRs accounts for the unique aspects of LSR reporting and can be used by authors and publishers to guide and standardize LSR reporting. Patient, public, and/or healthcare consumer involvement: Improving the reporting of LSRs is essential for providing stakeholders with synthesized evidence in a timely manner.

Background: Systematic reviews are the gold-standard of evidence syntheses and underpin decision making which affects outcomes for patients and health consumers. Our living systematic review has found 485 articles documenting 67 discrete problems relating to the conduct and reporting of systematic reviews which can potentially jeopardise their reliability or validity.[1] A dedicated website has been created (https://systematicreviewlution.com/) to thematically categorise and elaborate these issues around four overarching domains relating to systematic reviews being 1) comprehensive, 2) rigorous, 3) transparent, and 4) objective. The next phase of this research is to engage researchers working in systematic review to discuss the challenges around the trustworthiness of systematic reviews and explore solutions to the problems identified. (PROSPERO: CRD42020181371; Open Science Framework (OSF): https://osf.io/2hmv9/)
Objectives: To perform a prioritisation exercise of the problems with systematic reviews identified by consulting with researchers internationally and across disciplines to improve future systematic reviews.
Methods: Corresponding study authors of the 485 included papers will be invited via email to take part in a prioritisation exercise. Systematic review problems will be ranked based on relative importance, the severity or risk posed, and the ease of solution for the 67 problems identified thus far. Study methods for this work will be prespecified and registered with the OSF.
Results: Results of the prioritisation exercise with researchers will be twofold: 1) The project website will be expanded to incorporate the results of the engagement with researchers and to increase the value of www.systematicreviewlution.com as a research resource in systematic reviews and 2) A corresponding journal article will be used to further promote the research findings.
Conclusions: Systematic reviews have a pivotal role in medical decision-making due to apparent comprehensive, rigorous, replicable, and objective processes. In addition to outlying the many problems that systematic reviews are prone across these domains we will work with the research community to ensure that outputs which inform patient care represent trusted evidence. References: Uttley, L., Quintana, D. S., Montgomery, P., Carroll, C., Page, M. J., Falzon, L., ... & Moher, D. (2023). The Problems with Systematic Reviews: A Living Systematic Review. Journal of Clinical Epidemiology.

Background: Many people find it difficult to make decisions about what to believe or do regarding their own and others’ health. To avoid being misled by misinformation and to make good choices, people must be able to understand and apply some basic concepts. We developed the Informed Health Choices (IHC) secondary school resources to help students learn and use nine such concepts.
Objectives: To develop digital IHC secondary school resources that increase students’ ability to think critically about health information and choices, are compatible with the curricula in Kenya, Rwanda, and Uganda, and are suitable for use in low-resource settings with limited digital infrastructure.
Methods: We used a human-centered design approach, drawing extensively on the experiences and perspectives of multiple stakeholders, including teachers, students, and curriculum developers, over a 2-year period. In Phase 1, we conducted four preliminary studies, including context analyses and an overview of systematic review of teaching strategies for critical thinking. In Phase 2, we carried out three iterations of content development, collecting data through user testing, individual and group interviews, and school pilots; brainstorming solutions to problems; creating new versions of the resources; and triangulating analyses and solutions with curriculum developer offices.
Results: Four main challenges persisted throughout development: the Student-computer mode of lesson plans (which we deactivated), time for scheduling and teaching lessons, correct comprehension of the concepts, and use of engaging examples that were not misleading. The final version of the resources includes lesson plans for ten 40-minute lessons (for use in classrooms equipped with either a blackboard/flipboard or a projector), teachers’ guide, and extra resources, including materials for teacher training.
Conclusions: We developed lessons to teach secondary school students to think critically about health information and choices: https://besmarthealth.org/. They are open-access, work in low-resource settings, and are adaptable/translatable. Relevance to public: Teaching young people to think critically about health actions has the potential to protect them from unnecessary suffering, harm, and resource waste.
Patient, public and/or healthcare consumer involvement: Using a human-centered design approach, teachers, students, and curriculum developers were continuously involved in developing the intervention.

Final version figure 4-01.png
Final version figure 4-02.png
Screen Shot 2023-03-06 at 14.46.28.png

Background: Individual participant data meta-analysis (IPDMA) projects herald huge promise [1], but are time-consuming, often taking upwards of two years to obtain, clean, harmonise, and meta-analyse IPD. Therefore, before embarking on an IPDMA project, researchers and funders should ascertain if it is worth their investment and, if so, how best to proceed.
Objectives: In this talk, aimed at a broad audience, I discuss the role of power calculations in IPDMA projects and how they reveal (1) whether the project is capable of identifying meaningful clinical effects and, if so, (2) which studies to prioritise for IPD collection.
Methods: Firstly, IPDMA investigators should identify the set of studies likely to provide their IPD and extract key characteristics, such as number of participants and outcome events per group, and covariate distributions. Using this information and assumptions about true study effect sizes, Fisher’s information can be applied to derive the approximate standard error of each study’s effect estimate. Then, these are used to calculate the anticipated standard error of the IPDMA summary result, and subsequently the IPDMA power.
Results: Real applications will be showcased (rather than statistical formula), focusing on IPDMA projects of randomised trials examining treatment-covariate interactions (treatment effect modifiers) at the individual level. Continuous, binary, and survival outcomes are covered. In one example, an IPDMA of 31 trials has 42% power for a treatment-sex interaction but 90% power for a treatment-age interaction. The calculations also reveal the contribution of each study towards power, to help inform which studies should be prioritised for IPD collection. I show that a study’s power contribution depends on covariate distributions not just total participants or events.
Conclusions: Careful assessments of power and sample size should be an integral part of planning and commissioning IPDMA projects, in advance of IPD collection. Alongside other information (e.g., risk of bias assessments), power calculations help direct investment toward high-quality IPDMA projects most likely to provide trusted evidence that informs clinical practice to improve patient outcomes. Patient, public, and/or healthcare consumer involvement: None. [1] Riley RD, Tierney J, Stewart L(Eds). Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research. Wiley, Chichester; 2021

Background: Although the production of living systematic reviews (LSRs) follows the core principles of systematic review methodology, their key distinguishing feature is the continual incorporation of relevant new evidence as it emerges.
Objectives: As part of an effort to develop an extension of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 statement for LSRs, we discuss conceptual issues relevant to the reporting of LSRs and highlight a few challenges.
Methods: The issues discussed are based on a scoping review of the literature and discussions amongst authors. We also build on previous work related to updating systematic reviews.
Results: We first describe aspects of the LSR production process relevant to reporting. The production cycles differ by whether the literature surveillance identifies new evidence and whether newly identified evidence is judged to be consequential. This impacts the timing, content, and format of LSR versions. Second, we discuss four types of information that are specific to the reporting of LSRs: justification for adopting the living mode, LSR specific methods, changes between LSR versions, and LSR updating status. We also discuss the challenge of conveying changes between versions to the reader. Third, we describe two commonly used reporting formats of LSRs: full and partial reports. Although partial reports are easier to produce and publish, they lead to the scattering of information across different versions. Full reports ensure the completeness of reporting. We discuss the implications for the extension of the PRISMA 2020 statement for LSRs.
Conclusions: The extension of the PRISMA 2020 statement for LSRs would need to account for LSR specific information to report. A dynamic publication platform that decreases the burden of production and publication of full reports would facilitate complete and timely reporting of LSRs. Patient, public, and/or healthcare consumer involvement: There was no involvement. Improving the reporting of LSRs is essential for providing stakeholders with synthesized evidence in a timely manner.

Production processes of standard SR, LSR.png
Scenarios for update cycles of LSR.png

Background: Systematic reviews of public health interventions can be methodologically challenging because they are more likely to incorporate complexity in populations, interventions, contexts, outcome measurements and study designs. When a systematic review or protocol manuscript is submitted for publication to Cochrane Public Health (CPH), Methods Editors conduct an initial triage review and complete methods review.
Objectives: To identify common methodological errors requiring amendment in manuscripts and increase awareness of the available guidance to reduce their occurrence in future reviews.
Methods: We conducted a retrospective audit of written feedback provided by CPH Methods Editors to authors of a protocol or review manuscript submitted for publication between July 2019 and June 2022. A data extraction form was used to document methodological issues identified in written feedback documents. Four Methods Editors piloted the data extraction form on two manuscripts, after which one Methods Editor extracted data for the remaining documents.
Results: Forty-six feedback documents relating to twenty-seven reviews were identified. Examples of the errors identified related to the following: application of risk of bias domains, inconsistency between the methods described and used in the analysis, application of synthesis methods other than meta-analysis, inconsistent reporting of results in different sections of the review and incomplete reporting of results. Some of these errors had important implications for the conclusions of the review. In addition, it was common for manuscripts to lack details necessary to allow readers to understand and appraise the methods used. Some errors required authors to review and edit their manuscripts for consistency or to refer to existing guidance, e.g., on risk of bias assessment or synthesis other than meta-analysis. Data extraction for this project is in progress, and complete results will be reported in the presentation.
Conclusions: These results present an opportunity for authors to plan for and appropriately implement methods guidance to avoid common errors in future reviews.
Patient, public and/or healthcare consumer involvement: The public were not involved in this study. Enhancing the methodological strength of published systematic reviews ensures reliable support for the public and other decision makers.

Background: Since 2016, Cochrane UK has been delivering interactive sessions to teach teenagers about Evidence-Based Health Care (EBHC), its importance, and its relevance in our everyday lives.
Objectives: The sessions aim to introduce young people (particularly those interested in studying healthcare subjects at university) to Cochrane’s work by exploring what randomized controlled trials are, how systematic reviews are produced, and why they are a particularly trustworthy source of evidence. We also aim to help school children critically question the trustworthiness of claims they read or hear about health and consider how the media can misrepresent health information. We also aim to increase their understanding of the importance of using reliable, evidence-based sources of information.
Methods: We have developed the sessions with input from the whole Cochrane UK team, including experienced healthcare professionals. Typically, two members of the Cochrane UK team run the sessions, at least one of whom has experience working as a healthcare professional.
Results: As of early 2023, we have reached over 2,500 school children, the majority of whom have been 16-18 years old with an interest in studying healthcare-related subjects at university. We began by running face-to-face sessions in schools and then shifted to delivering online sessions during the pandemic. To date, we continue to predominantly deliver online sessions. We have also run sessions in partnership with Widening Access teams at various UK-based medical schools.
Conclusions: Over time, we have learned that hosting online sessions has been the most effective way to reach as broad an audience as possible. We typically advertise the talks through Cochrane UK newsletters and social media, as well as via our existing connections with schools and universities. This enables us to make the best use of resources and to reach children throughout the UK. We plan to continue this outreach work in this way. Patient, public, and/or healthcare consumer involvement: We use feedback from school children and teachers to refine these sessions, making the content more relevant and engaging over time. In this way, members of the public are directly involved in this project.

Background: Meta-analysis (MA) using individual participant data (IPD) is regarded as the gold standard, allowing more complex analyses (e.g., subgroup analysis), verification of published findings, inclusion of unpublished data, and some outcome harmonisation. Although researchers and funding agencies show strong in-principle support for data sharing required to perform such projects, rates of actual data sharing are frequently low.
Objectives: To document the experiences of obtaining IPD for a large individual participant data on Cord Management at Preterm birth (iCOMP), a large collaborative IPD and network MA in neonatology.
Methods: Lead and senior investigators for eligible trials were approached and asked to share IPD. We recorded the number (%) of trials in which the trialist responded favourably/declined or did not respond, the number of trials that actually shared their IPD, methods used, and barriers in sharing IPD.
Results: Of 118 eligible studies identified for inclusion in iCOMP, 82 (69.5%) agreed in-principle to share IPD. Three (2.5%) trials were uncontactable, and 36 trials (30.5%) did not respond even after multiple contact attempts. Among those who responded, 14 trialists (17%) declined or were unable to share data, frequently citing data access issues because of the trial’s age. Twenty-one trials (17.8%) did not share data despite initial agreement. We received data from 61 (51.7%) trials. A median of 23 (IQR 25) emails were sent from data request to receipt. The longest data acquisition period for a study was 831 days. Sixty datasets were received directly from the trialist and one dataset in the publication appendix. Data were shared in different formats with Microsoft Excel the most common. Additional barriers included lengthy processes for data sharing agreements and ethics requirements.
Conclusions: Data sharing is a resource-intensive and repetitive process which is often plagued by challenges. Streamlined processes and alternative arrangements, such as depositing into a data commons or data trust needs to be developed to better facilitate future collaborative IPD-MA. Sharing experiences in running and collecting data for IPD MA enables mitigating similar challenges in future projects. Patient, public, and/or healthcare consumer involvement: A consumer representative was part of our advisory group and authorship team.

Background: Individual participant data (IPD) meta-analyses (MAs) allow more comprehensive analyses (e.g., individual-level subgroup analyses) and enable in-depth risk of bias and trustworthiness checks. Yet often not all eligible studies have IPD available, leading to concerns about data availability bias. Methods have been proposed to combine IPD with aggregate data (AD) from publications, albeit some have raised concerns about lower quality of AD. Limited guidance exists on when to combine IPD with AD.
Objectives: To develop a checklist on whether to combine IPD and AD and apply this to a large IPD-MA in neonatology.
Methods: We conducted a literature review and consulted methods experts to collate proposed guidance. Checks were reviewed and consolidated by an advisory group of content and methods experts and documented in a pre-specified statistical analysis plan. The checks were applied to the large IPD-MA of 104 eligible studies, in which 61 provided IPD and 43 only AD.
Results: The developed checklist encourages researchers to use a range of prespecified criteria for comparing IPD and AD studies, including checking differences in key baseline and study characteristics, effect sizes, risk of bias, integrity concerns, and analysis methods (Table). In the large IPD-MA, we found major differences between IPD and AD in baseline characteristics, risk of bias, and integrity assessments (Table). We also found much larger effect sizes for some outcomes in the AD studies, indicating potential systematic differences between IPD and AD trials. Thus, the advisory group decided to only include IPD studies for the primary analysis, but present combined sensitivity analyses. Conclusion: In our case study, AD studies appeared to have higher risk of bias, be less trustworthy, and have much larger effect sizes. This highlights the importance of carefully assessing differences between IPD and AD, to not dilute the high-quality data derived in an IPD-MA with potentially lower quality AD. The developed checklist can be easily adjusted to inform future IPD-MAs. Relevance to patients: IPD-MAs frequently inform how patients are treated. For instance, our case study will form the evidence base for upcoming international guidelines. A consumer representative was part of our advisory group and authorship team.

IPDvsADCochrane_table.pdf

Background: Methods guidance prompts systematic review authors to define their questions and specify criteria for including studies in the review. Less emphasis has been given to specifying criteria for each synthesis (the 'PICO [population, intervention, comparator and outcome] for each synthesis’), although choices about the structure of the synthesis have important implications for review findings, particularly in reviews addressing multiple research questions.
Methods: We examined the extent to which PICO criteria were specified for syntheses (comparisons) in 100 randomly sampled systematic reviews published in 2018 from the Health Evidence and Health Systems Evidence databases. We used evidence from this study to draft the InSynQ (Intervention Synthesis Questions) checklist and guide for development and reporting of synthesis questions in systematic reviews of interventions. We conducted two online webinars and a public evaluation survey to invite feedback and refine the final version.
Results: Authors commonly group studies for synthesis based on population, interventions, outcomes and methodological characteristics, but the use of these groups is often incompletely reported. For example, of 41 systematic reviews that identified or used intervention groups as the basis for comparisons, 28 (68%) mentioned the groups in the background or methods; 12 (29%) defined the groups in enough detail to replicate allocation of studies to groups; 6 (15%) provided a rationale; and 24 (59%) stated that the groups would be used to define comparisons. Having identified aspects of reporting that are often incomplete, we developed the draft InSynQ checklist and guide (http://InSynQ.info). Revisions made in response to feedback led to the final 11-item checklist with refined wording and an additional item on consumer and stakeholder involvement.
Conclusions: Complete reporting of synthesis questions and the PICO for each synthesis helps ensure that the questions addressed in a review are clear and supports transparent decisions about the eligibility of studies for each synthesis. The InSynQ checklist and guide is designed to help authors develop and report synthesis questions completely. Further feedback is invited to inform future versions. Patient, public and/or consumer involvement: Evaluation of the draft checklist was open to the public. These findings promote clear communication of research questions of importance to consumers and transparent reporting of reviews.

Background: The main objective of the Informed Health Choices (IHC) project is to teach people to assess treatment claims and make informed health choices. For this purpose, the IHC learning resources were developed for primary school children.
Objectives: The aim of this study is to explore students’ and teachers’ experiences when using the IHC resources in primary schools in Barcelona (Spain).
Methods: We conducted a mixed methods study for piloting the IHC resources in a convenience sample of primary schools in Barcelona. The intervention included a workshop with teachers and nine lessons with the students. We collected data using multiple approaches, including ad hoc questionnaires, non-participatory observations, and semi-structured interviews. We performed quantitative and qualitative analyses. Finally, we formulated recommendations for using the IHC resources in this setting.
Results: Two schools, with a total of 143 students in 4th and 5th grade (9 to 11-year-olds) and six teachers, participated in the study. One school followed the suggested IHC teaching plan and completed all the lessons; the other school modified the plan substantially and did not complete all the lessons. Overall, students and teachers from both schools understood, were interested in, and were able to apply the content of the lessons. During the lessons, students used the IHC resources (mainly the textbook), and teachers used them in a variable way. Teachers adapted the IHC resources to increase student participation and used Information and Communications Technologies to support their teaching. We observed more facilitators than barriers to teach the lessons. The teachers suggested some ideas to improve the lessons based on the activities they developed and implemented. We proposed seven recommendations for using the IHC resources in this setting.
Conclusions: It is feasible to use the IHC resources in Barcelona primary schools; however, these resources should be adapted to promote classroom participation. Patient, public, and/or healthcare consumer involvement: The IHC resources can empower children around the world to make well-informed health choices as adults.

IHC@BCN_PilotBCN_Photo_20230227.pdf

Background: During the COVID-19 pandemic, living systematic reviews with network meta-analyses (NMAs) summarized the growing evidence base to assist with decision-making. However, such evidence syntheses can produce an overwhelmingly large amount of information necessitating additional strategies for communication of findings. Objective: To describe how an interactive website was developed and used to present results from a living systematic review and NMA addressing COVID-19 prophylaxis and treatment.
Methods: A working group of health research methodologists, clinicians, and knowledge users brainstormed ideas for the website including what information to include and what functions to make available. We collaborated with a web developer who utilized results from the working group discussion to draft the website, incorporating subsequent feedback from the working group until we approved the final version.
Results: The website (www.covid19lnma.com) provides a summary of the results of each living systematic review including the number of patients, studies, interventions, and outcomes included alongside a table categorizing interventions from the most to the least effective/harmful for all outcomes. Additionally, users can select an intervention and comparator to generate a summary of findings (SoF) table that includes plain language summaries per outcome. This interactive feature allowing users to generate SoF tables for pairwise comparisons of interest to them is unique to this format of publication. Launched in August 2021, the website has had a steady flow of traffic with users around the world. Updates are made as new results become available with the date of analysis provided to end users.
Conclusions: Interactive websites can summarize the results of NMAs while providing users with the ability to tailor these results for their needs while overcoming challenges with publication delays to ensure timely access to information. By incorporating plain language summaries, such a website can also facilitate knowledge translation to the general public. Studies are required to understand whether these websites increase access to and improve understanding of NMA results and how to best improve them.
Patient, public and/or healthcare consumer involvement: Patient partners were not involved in the development of the website.

Background: Individual participant data (IPD) meta-analysis is regarded as the ideal approach for providing evidence on intervention effect estimation because it can derive standardized outcome definitions and use a consistent analysis method. Our previous study has summarized the methodological and reporting quality of published IPDMA (BMJ 2021;372:n736). However, the current practices and perceived best strategies in IPD retrieval are still unclear.
Objectives: The aim of this study is to explore the perceived and practical strategies in IPD retrieval.
Methods: A cross-sectional survey was conducted. Email addresses of IPDMA authors were identified through PubMed. An e-questionnaire with 32 questions, related to the authors’ demographic, their views and practices on different strategies in conducting IPDMA, was created based on literature. Qualitative email interviews were then conducted to gather in-depth information about their barriers and perceived strategies to maximize the IPD retrieval. Descriptive statistics, inferential statistics and linear regression models were used to analyze the data collected from the e-questionnaire while content analysis was undertaken for the qualitative interview. Ethics approval was granted by the university institutional review board.
Results: Of 151 respondents, most were male (62%), aged ≥35 years (80.1%), and academic staff (69.5%) and had >5 years of experience (80.1%). The mean successful rate of retrieving the IPD data was 67.2%. A total of 64.2% included their primary studies in the IPDMA and 44.4% provided authorship for primary study authors as incentive. Email was the most common method to contact study authors (90.1%) and share data (67.5%); it was also ranked the most effective way of requesting IPD. Linear regression models revealed that those (i) aged ≥65 years, (ii) who were academic staff as the first contact person, and (iii) who had a primary study included in the IPDMA had significantly higher success rates. Two main categories were identified from the email interview qualitative responses: (i) ‘Contributors of successful IPD retrieval’ and (ii) ‘Reasons for failed IPD retrievals’. (See Tables)
Conclusions: It was revealed that senior academics working in the field with primary data themselves had a higher chance to retrieve the IPD. Providing authorship of the IPDMA publication to the primary study author may also help to improve the success rate.
Patient, public and/or healthcare consumer involvement: None.

Cochrane abstract 2 - IPDMA.pdf

Background: The COVID-19 pandemic instigated unprecedented challenges for clinical and policy decisions. In response to these challenges, the pandemic also resulted in much-awaited innovations. Objective: To perform a scoping review describing challenges and innovations in evidence synthesis and evaluation during the COVID-19 pandemic.
Methods: We searched medical research databases, up to July 2022, for articles describing challenges or innovations in evidence synthesis and evaluation during the COVID-19 pandemic. We used a reflexive thematic analysis to generate themes that summarize and describe the data.
Results: We identified 91 references, reporting on 21 and 20 unique challenges and innovations, respectively. Salient challenges included the volume and rapidity of published COVID-19 research (often of poor quality), which precluded efficient identification of practice-changing research. To help evidence users navigate the volume of research, several initiatives produced living databases and catalogues of COVID-19 studies. During the pandemic, the scientific community very enthusiastically embraced preprint servers to allow for the rapid dissemination of potentially lifesaving research. The medical community, however, expressed concerns that preprints may lead to the dissemination of research that contains errors or fraudulent or fabricated data. Most of the evidence guiding the treatment of patients with COVID-19 came from platform trials—adaptive trials designed with the option to add and remove arms based on accumulative evidence. Although there are many trials addressing pharmacologic interventions, there were few trials addressing nonpharmacologic interventions to curb transmission, despite their widespread global implementation. The pandemic also resulted in a breakthrough for living systematic reviews and guidelines, which, unlike traditional reviews and guidelines that are only up to date at a single point in time, are updated as new evidence emerges.
Conclusions: The COVID-19 pandemic posed unprecedented challenges to evidence synthesis and evaluation but also spurred critical innovations. As the pandemic shifts from an acute threat to a long-term public health issue, now is a critical time to evaluate and learn from our response to inform future preparedness efforts.
Patient, public and/or healthcare consumer involvement: While we did not include consumers as part of this review, several publications included in our review describe challenges and innovations related to consumer involvement.

Background: Awareness and understanding of systematic reviews and the role that they play in healthcare is quite low in the general population. Existing work by Cochrane aims to better support people to make sense of and co-produce health evidence. The People’s Review will actively involve the public in the planning, production and dissemination of a systematic review, which will help them understand how evidence guides healthcare provided by a health service and everyday health decision-making. Our work builds on the success of The People’s Trial, in which members of the public designed, planned and took part in an online randomized trial (https://doi.org/10.1186/s13063-021-05984-1) with the aim of improving the understanding of clinical trials.
Objectives: The People’s Review aims to increase the public’s understanding of what systematic reviews are, how they are developed and why they matter. A secondary aim is for reviewers to learn more about and develop ways to involve people in systematic reviews.
Methods: The People’s Review includes a systematic review with public involvement throughout, from designing the research question through to knowledge translation. It will be conducted online in a multi-phase approach through a custom-built, open-access website. We will design, produce and share educational material (e.g., animated videos) to support the public’s understanding at each phase of the review. Public contributors will make key decisions about the review and participate in review tasks, utilising a ‘learning by doing’ approach. The ACTIVE framework for stakeholder involvement in systematic reviews will inform this review.
Results: The People’s Review is in the early stages of development. At the colloquium, we will present a protocol detailing how we plan to involve the public throughout the review.
Conclusions: The People’s Review, if successful, will increase the public’s capacity for critically appraising and using systematic reviews (including Cochrane reviews) to inform everyday health decisions.
Patient, public and/or healthcare consumer involvement: The People’s Review Steering Group includes four public partners who are supporting the development of this project. The wider public will then be invited to contribute to decisions and tasks related to the review when the website is launched through a widespread social and traditional media campaign.

Background: In 2019, Cochrane released a revised Risk of Bias (RoB2) tool to address limitations of the previous tool. Since then, difficulties with implementation, adherence, and time taken to complete assessments have been reported. For large Individual Participant Data (IPD) Meta Analyses (MAs), applying RoB tools leads to additional challenges and opportunities.
Objectives: To develop a modified RoB tool combining elements of the original Risk of Bias (RoB1) with RoB2, incorporating checks assessing RoB using IPD, and implementing this in a meta-analysis (MA), including IPD from 59 trials and aggregate data (AD) from 33 trials.
Methods: We collated data from all included trials, and reviewed data availability for all outcomes. Content and methods experts were consulted to develop a consensus on domains and response options. Assessments were completed by two trained team members, and disagreements resolved in consensus meetings including a third team member. Remaining items were resolved by an advisory group, including members with extensive clinical, biostatistical, systematic reviews, and consumer experience.
Results: Instead of assessing RoB for each outcome, as done in RoB2, or for each study, as done in RoB1, the developed checklist (Figure) pre-specified RoB to be assessed per outcome category. These included primary outcome mortality, delivery room outcomes, beyond delivery room outcomes, and long-term outcomes. We judged RoB to be nearly identical for each category as most commonly either all or no follow-up outcomes were blinded. Additional checks utilising IPD were completed, assessing random allocation and incomplete outcome data. Overall RoB for each study, per outcome category, was judged as high, some concerns or low, matching RoB2. The modified RoB was implemented for all 92 trials. Availability of IPD and direct contact with investigators enabled higher rates of certainty for each domain, and strong consensus across raters compared to AD trials.
Conclusions: RoB1 and RoB2 have shortcomings for application in large IPD-MAs. A modified RoB tool for application in a large IPD meta-analysis was successfully developed and implemented within our study. Patient, public, and/or healthcare consumer involvement: It is essential to assess RoB of trials included in IPD-MAs, as these frequently inform patient care guidelines.

Cochrane_Abstract_RoB_1Mar2023_figure.pdf

Background: Choosing between the many effective disease-modifying therapies (DMARDs) for rheumatoid arthritis requires timely, high-quality systematic review of the best available evidence. ‘Living’ systematic reviews fast-track evidence synthesis without compromising rigorous systematic review methods. Network meta-analysis (NMA) is an attractive option for comparing rheumatoid arthritis treatments, but the development of ‘living’ NMAs presents unique methodological challenges.
Objectives: Our aim is to establish a sustainable, living NMA off DMARDs for rheumatoid arthritis and to facilitate the evolution of living NMA methods.
Methods: We are conducting three linked Cochrane living systematic reviews and living NMAs of DMARDs for rheumatoid arthritis using methods recommended by the Cochrane Living systematic Review Network, supplemented by new methods developed for this review. The three reviews will compare all current DMARD therapies for three separate populations of rheumatoid arthritis patients, grouped by prior therapy received (DMARD-naïve, DMARD-inadequate response, biologic DMARD- inadequate response).
Results: Several novel and emerging methods for the production of a living NMA have been incorporated and are described in five categories (“The 5 A’s”). 1. Accessibility: All raw data extracted will be formatted and hosted with the model code on the Open Science Framework in order to promote research efficiency and transparency. 2. Automation: Novel computer code has been developed to automatically read and format extracted data and generate GRADE tables. 3. Alliances: International collaborations aim to provide ‘global evidence for local recommendations’. 4. Applicability: These living NMAs will provide a direct living evidence base for living rheumatoid arthritis guidelines. 5. Atomisation: Identification, categorisation and extraction of data from RCTs is performed as ‘microtasks’ by a trained ‘crowd’ of volunteers.
Conclusions: Near real-time incorporation of evidence into systematic reviews is now possible, with incorporation into living guidelines as the next step. Novel methodologies will streamline the NMA process and inform living NMA methods.
Patient, public and/or healthcare consumer involvement: Trained ‘crowd’ of volunteers contributed to step 5.

Background: Knowledge users (KUs) need the highest-quality studies to make decisions about which interventions and policies should be used. The most reliable way to answer questions is with a systematic review (SR). AMSTAR and ROBIS tools are used to assess quality/bias in SRs. However, no automated tool currently exists to assess the quality/biases in SRs (Figure 1).
Objectives: (1) Develop a set of features to extract from SRs related to quality/bias; (2) develop a labelled dataset of 10,000 SRs; and (3) test, train and validate models and compare their accuracy.
Methods: A structure was proposed by the international steering group, and features were collected using a methods review and a survey. We used the results of a previous study comparing the tools which mapped an item’s concept across the three tools. A core team reviewed them and determined their feasibility in being automatically identifiable and predictable by our model (rather than through a manual process). A flowchart of activities is proposed (Figure 2). Five organisations with databases of preappraised SRs will supply the SRs. Duplicates will be removed and topics will be mapped. If topic fields, settings or conditions are missing, we will fill these gaps with a search for SRs. We will use ML Random Forests and DL Neural Network classification models such as Facebook’s StarSpace and Fasttext. We will use a ‘supervised’ learning model which learns by making predictions given examples of data.
Results: The proposed structure is found in Figure 1. The 21 items mapped from AMSTAR 1 and 2 and ROBIS will be used as the quality features. Other features chosen include PROGRESS-Plus items related to sex, gender and equity, 20 methods features (e.g., number of databases searched, PICO, and meta-analysis model used) and 10 results features (e.g., effect estimates and adverse events). SRs were collected and cleaned and the other features were extracted in duplicate. Testing will begin in March 2024.
Conclusions: An artificial intelligence tool that critically appraises SRs would dramatically reduce the financial and human resources currently needed to appraise SRs and update SR databases (e.g., McMasterPlus, HealthEvidence and SysVac).
Patient, public and/or healthcare consumer involvement: None.

Fig 1 and 2 2023 Colloquium.PNG

Background: The hierarchy of evidence is considered as widely known to evidence-based medicine experts. However, it remains unknown how nonexperts understand it and how they use it in decision-making.
Objectives: To assess what level of evidence is needed for different stakeholders in the healthcare system to make treatment effectiveness decisions.
Methods: A quantitative cross-sectional study was conducted from November 2021 to February 2022 using an online survey. The participants were researchers, healthcare workers and consumers from Croatia. The survey had six scenarios about the same medical treatment presented within different study designs and in random order. Participants were asked to assess whether the descriptions presented a sufficient level of evidence to conclude that the treatment was effective.
Results: A total of 584 participants were eligible for inclusion (97 researchers, 201 healthcare workers and 286 consumers). Participants were mainly women (N=434 (74%), median age 43.5, interquartile range 33-52). For researchers, as the number of participants and degree of variable control in the study design increased, the perceived level of sufficient evidence also increased significantly. Among consumers, no significant differences were observed in scores between cross-sectional study, cohort study, randomised controlled trial and systematic review. Healthcare workers’ assessments were significantly lower for case studies and case series compared to other study designs (Figure 1).
Conclusions: Consumers and healthcare workers did not increase their certainty about the effectiveness of the therapy when higher-level study designs were presented compared to lower-level study designs. There is a need to implement educational courses on basic research methodology in lower levels of education and as part of the Continuing Medical Education for all stakeholders in the healthcare system.
Patient, public and/or healthcare consumer involvement: Healthcare consumers were involved in the study as participants. Educating patients and other stakeholders in the healthcare system would lead to more informed decision-making.

Figure 1.pdf

Background: Individual participant data (IPD) meta-analyses rely on the availability of raw IPD. Yet, calls for open data and data sharing seem to often go unheard and many IPD meta-analyses struggle to attain sufficient IPD. The practice and culture of data sharing might be improved if data sharing policies were prominent and actionable at all stages of the research process.
Objectives: To describe the epidemiology of data sharing policies across the health research lifecycle and identify pathways to improve rates of data sharing.
Methods: This was a cross-sectional study, describing the epidemiology of data sharing policies of major stakeholders across the health research lifecycle. We included the following major stakeholders in medical research: the largest public, philanthropic, and private funders, all national ethics committees, all primary clinical trial registries, the 5 highest-impact journals for each of the 59 fields of medicine, and all research data repositories. We assessed each stakeholder for the presence of a data sharing policy, magnitude of support for data sharing (data sharing mentioned, recommended, or required), policy characteristics, and COVID-19 specific policies.
Results: We included 935 stakeholders: 110 funders (55 public or philanthropic, and 55 private), 124 ethics committees, 18 trial registries, 273 journals, and 410 data repositories. Data sharing was required by 41% (45/110) of funders, no ethics committees or trial registries, 19% (52/273) of journals, and 6% (24/410) of data repositories (Figure 1). Private pharmaceutical and biotechnology funders were 6.3 times more likely than public funders to require data sharing. Among those requiring data sharing, limited guidance was provided on what, when, who with, how, and why to share data. No stakeholders required data sharing specifically for COVID-19 studies.
Conclusions: Increased data sharing rates are crucial to enable IPD meta-analysis. Calls for data sharing have accelerated, but major stakeholders across the health research lifecycle provided limited requirements or guidance to ensure the scientific and ethical imperatives for data sharing are met. We see particular opportunities to increase data sharing rates for public funders and ethics committees. Patient, public, and/or healthcare consumer involvement: IPD meta-analyses frequently inform guidelines that directly inform how patients are treated.

Figure 1. Data sharing policies by stakeholders.png

Background: Living systematic reviews (LSRs) of prognostic studies rely on screening many observational studies that are not clearly labeled.
Objectives: To assess the performance of a machine learning (ML) algorithm for screening in an LSR for venous thromboembolism (VTE)–related outcomes and baseline risks in patients with COVID-19.
Methods: As part of a guideline development project for the American Society of Hematology (ASH) on the use of anticoagulation for thromboprophylaxis in patients with COVID-19, the team conducted an LSR to establish and maintain relevance of the baseline risk for VTE-related outcomes. The search was conducted in September 2020 (baseline search) and updated monthly until July 2021. At baseline, the search identified 69,560 citations. The team trained an ML classifier algorithm using the manual screening of the baseline search to partially automate the screening process for the next search iterations. The algorithm ranked captured citations based on likelihood for inclusion, with those appearing on top as most likely to be included. The algorithm was integrated in a new software, “Laser AI,” which will allow the team to screen prioritized citations in future updates. In this study, we screened manually, in duplicate and independently, a sample of 5% (n=3,478) of captured citations at two iterations of the living search and that were not allocated the highest likelihood for inclusion. In parallel, we retrospectively applied a model trained on the most recently screened documents to the initial set of search results to explore how the data distribution changed over time.
Results: We manually screened in duplicate and independently 3,478 citations, of which 377 were included at title/abstract level and full text screening is ongoing. We will compare these results with the algorithm’s classification to measure the algorithm’s performance (precision, recall, accuracy, and specificity). We will also assess whether there were identified studies eligible for inclusion that were not selected for screening and if they affected the pooled baseline risk for VTE. Results will be ready by March 31, 2023.
Conclusions: The efficiency and relevance of LSRs for prognostic studies can be enhanced when combining manual with ML-directed screening.
Patient, public and/or healthcare consumer involvement: N/A.

Background: Evidence synthesis is a constantly evolving field. There is now a plethora of evidence synthesis approaches used across many different disciplines. Historically, there have been attempts to organize the types and methods of evidence synthesis in the form of classification systems, typologies, or taxonomies. This scoping review sought to identify all the available classification systems, typologies, or taxonomies; identify how they were developed; their characteristics; and the types of evidence syntheses included within them.
Objectives: The objective of this scoping review is to identify evidence synthesis types and previously proposed classification systems, typologies, or taxonomies that have guided evidence synthesis.
Methods: This review has been developed in consultation with evidence synthesis taxonomy initiative advisory group, which includes over 100 evidence synthesisers. Utilising JBI guidance for scoping reviews, a rigorous database search of MEDLINE (Ovid); Embase (Elsevier); CINAHL with Full Text (EBSCO); ERIC (EBSCO); Scopus (Elsevier); Compendex (Elsevier); JSTOR; and grey literature search has identified 292 evidence sources for inclusion, and they are currently undergoing extraction. This scoping review considered documents such as discussion papers, commentaries, books, editorials, manuals, handbooks, and formal guidance from major organizations.
Results: This review identified many different types of classification systems identified for categorizing evidence synthesis approaches. These systems varied in how they were formatted, their characteristics, the evidence synthesis approaches included, and how they have organized evidence synthesis types. Lastly, this scoping review identified differing terminology relating to evidence synthesis types being utilized across fields and how this could be perpetuating confusion and poor conduct of evidence synthesis.
Conclusions: The development of typologies, taxonomies, and classification systems of evidence synthesis have not as yet been conducted through a systematic process, nor with extensive stakeholder engagement from members of the academic community. They have also been criticized for being incomplete and lacking consistency in the terminology. Although many of the pre-existing categorisation systems serve as a useful reference point for researchers, a comprehensive, continuously updated, and community-approved evidence synthesis taxonomy is needed. Patient, public, and/or healthcare consumer involvement: Yes, through the ESTI advisory board (100+ members).

Objectives: The internet is an important resource for health information (HI). However, many users have difficulties in deciding whether the information they found online is trustworthy and objective. We aimed to develop and validate a checklist for helping laypersons with evaluating the trustworthiness of online HI.
Methods: We searched for existing checklists and conceptual research for HI assessment and extracted all items/indicators. After we categorized, deduplicated, and translated items into German, six HI experts identified the most pertinent items in a modified Delphi process (“item-shortlist”). The item-shortlist was tested by laypersons for comprehensibility and applicability (cognitive tests, n=19) and usability (assessment of 15 selected HI, n=20). The research team applied the item-shortlist to 100 HI, and two researchers independently assessed the objective certainty of evidence for each HI. These assessments were used to determine the predictive validity of the items.
Results: We extracted a total of 1,740 items from 73 documents. After all reduction and modification steps, we tested a shortlist of 23 items. The claimed strength of evidence of the HI relative to the objective certainty of evidence for the specific health topic was used as outcome measure to test the predictive validity regarding the correctness of the HI of each item as well as a set of items. To create the final checklist, we considered the laypersons’ qualitative results, the effect estimates of each item, the inter-reliability measures among laypersons, among experts, and between laypersons and experts. The final checklist comprises the following 6 items: absence of advertisement, balanced presentation of information, citation of sources, limited use of technical terms, presence of a publication date, and a medical disclaimer. It will be published on the project webpage https://www.infos-ohne-nebenwirkung.at/.
Conclusions: The creation of a checklist for the assessment of online HI that can be understood and used by laypersons is a great challenge. The final checklist has to be tested with further HI test sets to strengthen its validity. Patient, public, and/or healthcare consumer involvement: We involved potential users of the checklist in two stages of the development process of the checklist to ensure the comprehensibility and applicability of the checklist.

Background: There are an increasing number of randomised controlled trials (RCTs) in paediatric pain, but the quality of these studies is unknown. It is imperative that RCTs in paediatric pain are of high quality.
Objectives: To determine the risk of bias and reporting quality of RCTs in paediatric pain.
Methods: This was a cross-sectional study of RCTs in paediatric pain which were published in full text in 2021. Studies were identified by searching MEDLINE, Embase, Cochrane Library, CINAHL, and Scopus. Risk of bias was assessed with the RoB 2 tool and reporting quality was assessed with the CONSORT 2010 statement. Risk of bias was assessed as either high, some concerns or low, and reporting quality was assessed as adequate if ≥75% of items in the reporting guideline were reported. All studies were screened and assessed in duplicate by two independent investigators.
Results: We identified 212 RCTs in paediatric pain published in full text in 2021. Almost three quarters of RCTs (n=154, 73%) were at high risk of bias, and almost all other RCTs (n=57, 27%) had some concerns (see Figure 1). Only one RCT (0.4%) was at low risk of bias. Few RCTs (n=27,13%) were adequately reported and none reported ≥50% of items in the abstract. There were no author characteristics associated with risk of bias. RCTs that were blinded were more likely to be at a lower risk of bias and adequately reported X2 ([1, N=212] =17.7, 0.001; [1, N=212] =6, p=0.014). RCTs that reported a conflict of interest were prospectively registered or reported the use of reporting guidelines were more likely to be adequately reported ([1, N=212] =7.02, p=0.008; [2, N=212] =17.1, 0.001; [2, N=212] =7.56, p=0.023). The reporting of some PedIMMPACT outcomes, specifically symptoms and adverse events and the global judgement of satisfaction with treatment, were associated with reduced risk of bias and improved reporting quality ([1, N=212] =4.26, p=0.039; [1, N=212] =4.59, p=0.032; [1, N=212] =5.88, p=0.015; [1, N=212] =5.78, p=0.016).
Conclusions: Recently published RCTs in paediatric pain are at the high risk of bias and have poor reporting quality. Patient, public, and/or healthcare consumer involvement: None.

Figure 1. Risk of bias of randomised controlled trials.png

Background: Cochrane guidelines recommend broad electronic bibliographic database (EBD) searches and additional clinical trial register (CTR) searches to inform systematic reviews (SRs) of randomized clinical trials (RCTs) involving interventions. This two-step approach limits the number of unidentified RCTs to prevent the synthesis of biased results. Clinical trial registration is performed prospectively for RCTs; thus, the publication status does not interfere with the availability of the CT registration within the register. Performing broad CTR searches by themselves may be a more efficient approach to identify RCTs for SRs of RCTs involving interventions.
Objectives: To evaluate how broad CTR searches compare to broad EBD searches in identifying RCTs for SRs of RCTs involving interventions.
Methods: We performed identical searches in the CTRs of ClinicalTrials.gov and the International Clinical Trials Registry Platform, as well as the EBDs CENTRAL, EMBASE, and MEDLINE, and compared whether we could identify the same RCTs. Our search was based on a subset of RCTs from an SR that studied the underrepresentation of people with chronic kidney disease from cardiovascular trials.
Results: We identified 128 eligible RCTs, of which 68 (49% [95% CI 42 – 58]) were identified by both search approaches. The EBD search missed 54 (43% [95% CI 34 – 51]) of the eligible RCTs. Of these, 37 were still ongoing and 13 were missed although available in the searched EBDs. Three RCTs missed by the EBD search had not been published while having a study completion date before 2020. One RCT had been published in a journal that was not indexed in the EBDs searched in our study. The CTR search missed 11 (8% [95% CI 5 – 15]) of the eligible RCTs. Of these, seven were not identified by the CTR search although available in the included CTRs. We were unable to locate a CT registration for four of the RCTs missed by the CTR search.
Conclusions: Broad CTR searches are suitable to inform SRs of RCTs involving interventions. Patient, public, and/or healthcare consumer involvement: No involvement. Our study has implications for the methodology of SRs of RCTs involving interventions.

Background: One of the tenets of Cochrane’s knowledge translation strategy is to encourage policy makers and healthcare managers to integrate Cochrane evidence into clinical practice. Cochrane reviews are among the most valuable, high-quality sources of evidence for improving quality and care delivery, but strategies used by healthcare organizations to implement the findings of these reviews are less well-understood.
Objectives: To describe the knowledge translation and implementation strategies used by Kaiser Permanente Southern California, a healthcare delivery system serving more than 4.6 million patients, and the Evidence Scanning for Clinical, Operational and Practice Efficiencies (E-SCOPE) program, to integrate Cochrane systematic reviews into clinical practice.
Methods: E-SCOPE staff identify published Cochrane reviews through searches of the Cochrane library and other scientific databases. A priority is placed on identifying high-quality Cochrane reviews of diagnostic and therapeutic interventions that yield improved health outcomes, quality, and/or efficiency of care delivery. E-SCOPE staff then work closely with clinical and operational stakeholders to interpret and translate Cochrane evidence into a feasible implementation plan, leveraging existing organizational processes and resources. E-SCOPE project managers oversee implementation efforts, facilitate practice owner identification, and as needed, create measurement plans utilizing data drawn from electronic health systems and other data sources.
Results: The E-SCOPE program has implemented 18 practices based on results from Cochrane systematic reviews to improve the overall quality of care provided to patients. The time from review publication to implementation averages 16 months (ranging from 4 to 36 months).
Conclusions: E-SCOPE enhances Cochrane knowledge translation processes by making optimal use of Cochrane reviews, translating the evidence and findings to clinical stakeholders, and leveraging existing organizational channels for practice implementation. Prioritizing the identification of practices with an established, high-quality, and high-confidence evidence base is generalizable and can efficiently and effectively promote adoption of Cochrane reviews within any healthcare system. Approaches to knowledge translation prioritizing Cochrane systematic reviews have the capacity to significantly improve the quality and delivery of patient care. Examples of Cochrane review-based E-SCOPE initiatives will be discussed. Patient, public, and/or healthcare consumer involvement: Patient values and preferences are key determining factors in every implementation plan undertaken by E-SCOPE.

Background: The results of Cochrane Reviews are packaged and presented in many different ways, including through tweets, podcasts, plain language summaries, and journal reports. Regardless of product type, media, or audience, all dissemination products should ideally provide a reasonable representation of the evidence, as well as a good user experience.
Objectives: Cochrane’s Dissemination Checklist aims to improve the quality and consistency of dissemination products and to improve communication with non-experts (i.e., people not familiar with systematic review methodology).
Methods: Development of the checklist was led by Cochrane Norway with support from an advisory board. We conducted a literature search and framework analysis and then went through several cycles of feedback and revision. Results/
Conclusions: Cochrane’s dissemination checklist is a tool for anyone preparing a dissemination product to share the results of a Cochrane intervention review. The checklist is aimed at people creating dissemination products that present review findings. If you are building awareness about a review without presenting the findings, you might find some helpful tips here as well. You can use this guidance when you plan the dissemination of a Cochrane review, create a dissemination product, or create or improve a dissemination product template. The checklist covers a range of issues, including considering and involving your target audience in your dissemination product, ensuring a reasonable representation of the evidence, using plain language, structure and readability, connecting with people, and ensuring transparency and building trust. There are three layers to the checklist and guidance: 1. A one-page overview of the checklist. 2. The checklist with a brief description of what each checklist item means and a set of minimum expectations. 3. Full guidance, with examples and references. This layer includes the information you need to fully apply each checklist item. The checklist was first made available in 2021 and is available here: https://training.cochrane.org/online-learning/knowledge-translation/how-share-cochrane-evidence/dissemination-essentials-checklist. At the link, you can also find a series of videos that walk you through the checklist. Patient, public, and/or healthcare consumer involvement: The checklist includes guidance regarding how you should consider and involve your target audience in your dissemination product.

Background: Core outcome sets (COS) represent agreed-upon minimum outcomes that should be reported in all studies in a given topic area. Among the first 100 Cochrane systematic reviews (SRs) published in 2019, 7 (7%) cited a COS in relation to choosing outcomes. A relevant COS existed but was not mentioned (or cited) for 27 of the remaining 93 SRs (29%). Among Cochrane Review Group editors surveyed in 2019, 86% felt that COS should definitely/possibly be used in Cochrane SRs. The Cochrane Handbook now recommends including outcomes from relevant COS.
Objectives: (1) Examine whether authors are currently considering COS to inform outcome choice for Cochrane SRs. (2) Understand author barriers and facilitators of using COS in Cochrane SRs.
Methods: We examined the extent to which all Cochrane SRs published in the last 3 months of 2022 and all Cochrane protocols published in 2022: (a) cited a COS, (b) searched for COS, and (c) reported outcome inconsistency among included studies and/or noted the need for COS. We will conduct an online anonymous survey of authors of these SRs to identify barriers and facilitators of using COS to inform outcome choice.
Results: Among the 88 eligible SRs, 13 SRs (15%) cited a COS and 1 additional SR (1%) mentioned searching for COS. Among the 74 SRs that did not cite or search for COS, 17 SRs (23%) reported outcome inconsistency among included studies and/or the need for standardized outcomes/COS. Among the 231 eligible protocols, 24 (10%) cited a COS and an additional 14 protocols (6%) searched for COS. Twenty-three of the 37 protocols that searched for COS (62%) found and used one. At the Colloquium, we will describe how the SRs and protocols that cited COS used those outcomes. We will also present insights from the author survey.
Conclusions: Although more than twice the Cochrane SRs (16%) and protocols (16%) in 2022 are considering COS than Cochrane SRs in 2019 (7%), the persistent problem of outcome inconsistency merits greater use of COS in Cochrane SRs. Patient Involvement: Although no patients were involved in this abstract, COS are developed with various stakeholders, including patients.

Background: Interventional studies may have high risk of bias, which can mislead medical practice and waste research funding. Reducing or predicting the risk of bias during study design or conduct would help to address this issue.
Objectives: To describe interventions during study design and conduct to reduce the risk of bias in interventional studies.
Methods: This was a scoping review of three electronic bibliographic databases (MEDLINE, Embase, Cochrane Library) and eight grey literature sources (ClinicalTrials.gov, ANZCTR, WHO ICTRP, MedNar, National Guideline Clearinghouse, WHO Guidelines summaries, Analysis & Policy Observatory, Open Grey). Records were included if they described the implementation or effectiveness of an intervention during study design or conduct which aimed to reduce the risk of bias in interventional studies.
Results: We screened 17,334 records from the electronic bibliographic databases and 15,591 records from the grey literature databases. We identified four studies which were eligible for inclusion. These assessed (1) report writing aids, (2) material incentives for study participants, (3) third-party interventions (independent clinical events committee), and (4) education and training for researchers. At the study design stage, a more rigorous participant screening process and systematic participant tracking programme can be designed to ensure a lower rate of lost follow-ups. Financial incentives can be designed for young participants, which will help to increase their response rate and participation. During the conduct phase of the study, an independent clinical committee can avoid the inherent bias of the investigator in judging event outcomes, reduce the risk of bias, and correct the results. Report writing aids could address the current situation of poor adherence to clinical reporting guidelines and significantly reduce the risk of selective reporting.
Conclusions: We identified a few interventions that have reduced risk of bias, which do not encompass all the risks of bias faced by interventional studies. These interventions require adequate research funding and staff support and cannot be applied to all interventional studies. We also identified various pathways to reducing the risk of bias by intervening during study design which have yet to be tested in practice. Patient, public, and/or healthcare consumer involvement: None.

Background: Clinical trial registers provide information on planned, ongoing and completed trials, both published and unpublished. Gold standard systematic reviews mandate searching trial registers because it allows retrieval of unpublished information, thereby reducing publication bias and research waste and improving reliability of the generated evidence. However, searching trial registers is often challenging, and guidance is lacking.
Objectives: We aimed to address this gap by providing step-by-step guidance on how to search clinical trial registers to harness their full potential.
Methods: We developed guidance based on 1) a literature review, 2) information available on trial registry websites, 3) a consensus workshop with information specialists, trial registry staff and systematic reviewers collating their experiences and expert opinions, and 4) an online survey of Cochrane information specialists and health technology assessment reviewers.
Results: We derived several key steps encompassing guidance for where to search, formulating the search strategy, conducting the search, screening registration records, obtaining data and updating and reporting searches (Figure 1). For each step, we highlight key differences to searching bibliographic databases, demonstrate practical application using a case study, and offer recommendations such as which registers to search [World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP) and clinicaltrials.gov as a minimum]; avoid using filters (e.g., by participant age or study type); and adjust search strategies for each registry.
Conclusions: Clinical trial registers are rich in data, and this guidance outlines how best to find and harvest trial information from them. The recommendations can be used to retrieve unpublished results data directly from the registry or via communication with registrants. They may also facilitate identification of evidence gaps to inform research prioritisation, identification of ongoing trials for collaborative methodologies such as prospective meta-analysis, assessment of publication bias, and planning of updates of traditional or living systematic reviews.
Patient, public and/or healthcare consumer involvement: This guidance was tested by two first-time registry users as well as several experienced systematic reviewers. Implementation of these methods will help to streamline evidence synthesis, reduce bias and enhance the reliability and validity of systematic reviews, ultimately leading to better health outcomes.

Figure 1.pdf

Background: Allocation concealment (the concealment of the randomization sequence from personnel enrolling participants) and blinding (the concealment of the arm to which participants have been randomized from one or more individuals involved in a trial) have long been considered important to prevent bias in randomized trials. Objective: Assess the impact of allocation concealment and blinding on the results of COVID-19 trials.
Methods: We searched the World Health Organization (WHO) COVID-19 database (up to February 2022). We included randomized trials that compared drug therapeutics with placebo or standard care in patients with COVID-19. We performed random-effects meta-regressions comparing the results of trials with and without allocation concealment and blinding of healthcare providers and patients.
Results: We identified 488 trials. We found that, compared with trials with allocation concealment, trials without allocation concealment may estimate treatments to be more beneficial for mortality, mechanical ventilation, hospital admission, duration of hospitalization, and duration of mechanical ventilation, but results were imprecise. We did not find compelling evidence that, compared with trials with blinding, trials without blinding produce consistently different results for mortality, mechanical ventilation, and duration of hospitalization. We found that trials without blinding may estimate treatments to be more beneficial for hospitalizations and duration of mechanical ventilation.
Conclusions: We did not find compelling evidence that COVID-19 trials with and without blinding produce consistently different results, but trials without allocation concealment estimate treatments to be more beneficial compared to trials with allocation concealment. Our findings have implications for the design of future COVID-19 trials and trials to inform future health emergencies and for systematic reviewers, guideline developers, and clinicians who are interpreting and applying evidence from these trials. Patient, public, and/or healthcare consumer involvement: We involved patient partners in selecting the outcomes of interest.

Blinding COVID-19 Figure.pdf

Background The University of Mississippi Medical Center School of Nursing Doctor of Nursing Practice (DNP) Program initiated an innovative partnership with Children’s of Mississippi (COM) hospital to teach DNP students and COM nurses (COMNs) how to translate evidence into practice. The project engaged stakeholders to prioritize the individual evidence needs of five COM units and build relationships among evidence-based practice (EBP) champions. Academic/practice partnerships can enhance educational programs and help nurses become well-positioned to lead change and improve the quality and safety of healthcare delivery. Objectives The purpose of this partnership was to pair DNP students with COMNs and train them to: -Develop clinical leadership skills to lead change; -Define EBPs for the clinical area; -Conduct unbiased clinical audits (CAs) and engage in quality improvement (QI) processes; -Design and execute strategies to implement EBPs. Methods Five COMNs were paired with five DNP students to address priority questions identified by hospital leadership. All completed an evidence implementation training program (EITP) that included two, week-long intensive workshops. In the intervening 6 months between the workshops, the pairs led evidence implementation projects (EIPs) using CA methodology on five COM units. JBI Evidence Summaries were used to identify current evidence and CA criteria for each EIP. Compliance with EBPs were measured at baseline and after implementing strategies while prioritizing the needs of the evidence users at the clinical sites. Results Overall Hospital leadership identified the COMNs as QI Champions who have the skills and tools to continue to lead EIPs within the organization. EIP reports were submitted for publication, and 4/5 (80%) were accepted for publication. Impact on patients and healthcare consumers Positive patient and organizational outcomes were achieved with the EIPs. CAs revealed improvements in compliance (32%, 38.5%, 43%, 47%, and 78%) with EBPs on all five units. Conclusions The EITP improved DNP scholarship and provided a framework for translating evidence into practice in a short period of time. This innovative academic/practice partnership benefits DNP students, clinical nurses, and healthcare organizations. Other organizations can use this as a model to implement partnerships to improve the quality and safety of healthcare delivery using evidence.

Background: Clinical Practice Guidelines (CPG) played a key role during the COVID-19 pandemic. CPGs oriented decision-making in a period of high information requirements and massive research generation. One of the fundamental steps in the development of CPGs is the prioritization of the outcomes that will inform the recommendations. In this process, various stakeholders, including patients, rate and select the most relevant outcomes for decision-making. Ideally, these are the outcomes that researchers should address in the studies that inform such CPGs. However, the reporting and selection of outcomes in primary studies are not necessarily in accordance with the CPG.
Objectives: To assess if randomized trials evaluating pharmacological treatments for nonsevere COVID-19 prioritize patient-important outcomes in their results.
Methods: We will include randomized trials evaluating pharmacological treatments for nonsevere COVID-19 identified in the COVID-19 living systematic review. We will consider the outcomes used in the COVID-19 living guidelines of the World Health Organization as the list of outcomes critical and important for decision-making (admission to hospital, death, quality of life, adverse events leading to drug discontinuation and time to symptom resolution).
Results: This project is in progress. We will describe the frequency that each outcome is reported in the included trials. We will also describe how many of the outcomes are reported at the trial level. In addition, we will describe whether authors present their results in a manner that can be included in a meta-analysis. We will perform a description by subgroups: funding sources and the availability of a published protocol before/after the publication of the first version of the World Health Organization guideline (September 2020).
Conclusions: This study can evaluate whether the initiatives that promote the measurement and reporting of patient-important outcomes have had an impact on the development of COVID-19 randomized trials. In addition, we can know if there is any factor that affects prioritization. This study is relevant to patients because the selection and reporting of outcomes should be patient centered.

Background: Forward citation chasing, defined as the use of a citation index to retrieve references that cite a source, is currently recommended as a complement to find all possibly relevant research for systematic reviews (SRs). Recently, new tools have been developed to ease this task, but their performance has not been compared yet.
Objectives: To compare the performance of different forward citation indexes and tools for forward citation chase in a sample of Cochrane SRs.
Methods: We searched for Cochrane SRs with at least two published versions 5 years apart. Starting from an ‘index reference set’ (i.e., references of the included studies in the original version of the SR, plus the reference of the original SR itself), we conducted a forward citation search using the following resources: Web of Science (WoS), Scopus, Google Scholar, Citationchaser, Paperfetcher, and Citation cloud. We assessed the performance of each tool regarding the identification of the ‘target reference set’ (i.e., primary study references included only in the updated version of the SR but not in the original version) in terms of sensitivity and precision.
Results: Preliminary, median sensitivity and precision results were, respectively, as follows: WoS: 85.7%/1.4%, Scopus: 78.6%/1.3%, Google Scholar: 78.6%/0.7%, Citationchaser: 41.1%/0.4%, Paperfetcher: 44.6%/1.0%, and Citation cloud: 57.4%/1.3%.
Conclusions: From our preliminary analysis, WoS showed the best performance in terms of sensitivity and specificity for conducting forward citation searches. The full findings of our research will be presented at the colloquium.
Patient, public and/or healthcare consumer involvement: None.

Background: The South African National Department of Health (NDoH), members of the National Essential Medicines List Ministerial Advisory Committee (NEML-MAC) on COVID-19 Therapeutics, and the South African GRADE Network, co-led by Cochrane South Africa (SA) and the Centre for Evidence-Based Health Care, collaborated on rapid evidence reviews to inform the National Essential Medicines List and Guidelines for COVID-19 management since the onset of the COVID-19 pandemic in 2020. A strategy was needed to communicate the findings of these rapid reviews to healthcare professionals managing COVID-19 in South Africa (SA).
Objectives: To develop infographics of rapid reviews to aid access for healthcare professionals to reliable, high-quality evidence to inform decisions around COVID-19 management in SA.
Methods: We collaborated with a graphic designer to create relevant infographics from rapid reviews. Infographic topics were prioritised in consultation with the NEML-MAC on COVID-19 Therapeutics. We worked with the designer to ensure correct content and readability. The final graphics were sent to the NEML-MAC on COVID-19 Therapeutics members for a final content check and sign off. Infographics were then shared with the NDoH and uploaded on their website (https://www.health.gov.za/covid-19-rapid-reviews/).
Results: Four infographics of rapid reviews were developed for the use of baricitinib (Figure 1), inhaled corticosteroids (Figure 2), oral corticosteroids (Figure 3) and ivermectin (Figure 4) to treat COVID-19. These were shared on the NDoH website, on social media and among academic networks. A COVID-19 inpatient and outpatient management algorithm infographic is also currently under development.
Conclusions: It is important to translate research into digestible information packages for healthcare professionals to access and be able to use. Infographics are a useful means to communicate these findings in an understandable and readable product which can be shared easily and widely.
Patient, public and/or healthcare consumer involvement: Shared with healthcare professionals through the NDoH website, social media and professional networks.

Figure 1.pdf
Figure 2.pdf
Figure 3.pdf
Figure 4.pdf

Background: An outcome may be selected as a primary or a secondary outcome in a randomized clinical trial (RCT). Researchers may expect a larger effect size from this outcome when it is selected as a primary outcome than when it is selected as a secondary outcome, leading to expectation bias.
Objectives: To compare the effect estimates between RCTs selecting an outcome as a primary outcome (RCT-POs) and those selecting the same outcome as a secondary outcome (RCT-SOs) assessing the same clinical question.
Methods: This was a retrospective cohort study on RCTs. We screened six high-impact journals (the Annals of Internal Medicine, The BMJ, JAMA, JAMA Internal Medicine, The Lancet, and PLoS Medicine) published between 2017 and 2022 for meta-analyses that assessed the efficacy of clinical interventions and produced statistically significant estimates. In each meta-analysis, RCT-POs were included in the exposure group, whereas RCTs-SOs were in the control group. The ratio of risk ratio (RRR), hazard ratio (RHR), or odds ratio (ROR, with OR transformed from SMD) between RCTs-PO and RCTs-SO were pooled to form a single estimate by random-effects meta-analyses. We planned to include 100 meta-analyses.
Results: This was an interim analysis. A total of 59 meta-analyses were identified, including 27 assessing medical products and 32 assessing other interventions. Six hundred six RCTs were included, comprising 259 RCT-POs and 347 RCT-SOs. On average, RCT-POs produced an effect estimate 1.18 (95% CI: 1.08-1.30, I2=44.0%) times greater than RCT-SOs. RCT-POs assessing medical products produced an effect estimate 1.24 (95% CI: 1.12-1.38; I2=44.3%) times greater than RCT-SOs, whereas RCT-POs assessing other interventions produced an effect estimate 1.09 (95%: 0.93-1.29; I2=44.7%) times greater than RCT-SOs.
Conclusions: RCT-POs generally produced an effect estimate 18% greater than RCT-SOs, implying that the expectations bias could significantly distort the results of RCTs, especially those assessing the efficacy of medical products. Patient, public, and/or healthcare consumer involvement:

Background: Core Outcome Sets (COSs) are defined as an agreed standardized set of outcomes that should be measured and reported, as a minimum, in all clinical trials in specific areas of health or healthcare. They help in reducing waste in research by improving the consistency of outcomes measured in trials and other research on the same health condition, ensuring that all important outcomes are measured. Systematic reviews of interventions summarise all the available evidence on a given topic and are used to inform healthcare decision-making. Frequently, the choice of outcomes for a systematic review raises concerns due to clinical trialists and systematic reviewers being interested in different types of outcomes. Cochrane reviews are well known for their rigorous methods and high-quality standards, with the relevance of COSs in defining review questions and planning the review acknowledged in the Cochrane Handbook for Systematic Reviews of Interventions. In 2019, a COS was published for trials testing any intervention intended to modify mechanical ventilation (MV) duration in critical care. This specific COS uptake in Cochrane systematic reviews has not been assessed.
Objectives: To assess the uptake of the MV COS in Cochrane systematic reviews of interventions.
Methods: This was a cross-sectional study. We included completed Cochrane systematic reviews and protocols of Cochrane systematic reviews of any intervention related to mechanically ventilated patients through a search in the Cochrane Database of Systematic Reviews. We extracted key information on participants, interventions, comparators, and MV COS outcomes. We also extracted data related to outcomes not included in the MV COS. For reviews or protocols published after 2019, we analysed the citation of the MV COS.
Results: We assessed 310 Cochrane reviews and protocols for eligibility. We will show the complete results at the London Colloquium.
Conclusions: We will show the conclusions at the London Colloquium.
Patient, public and/or healthcare consumer involvement: Not applicable.

Background: Climate change is the great challenge facing humanity in the 21st century. It has a wide range of health impacts that policymakers need to act upon. High-quality evidence syntheses (ES) are vital for policymaking. Search filters—validated combinations of search terms on a specific topic—play an important role in ensuring robust search methods for ES. Developing search strategies for climate health is complicated by the breadth of impacts and by the fact that relevant studies may not be labeled as related to climate change. Currently, no validated search filters for climate health studies are available.
Objectives: Our aim is to develop and validate search filters for MEDLINE to identify studies on health-related impacts of climate change per exposure pathway.
Methods: Climate change impacts human health via several pathways: extreme weather events, heat stress, changes in air quality, changes in water quality and quantity, effects upon food supply and safety, vector distribution and ecology, and social factors. We conducted a search for climate health–related ES in MEDLINE, Embase, CINAHL, Cochrane Library, and ProQuest Dissertations and Theses in February 2021. The search yielded 8,614 records after deduplication. These were dually screened and 110 climate-related ES were identified for inclusion, classified into exposure pathways, and sorted by number of included studies. We extracted each included study from the first 79 ES, classified each study according to exposure pathway, and checked each study for indexing in PubMed.
Results: We present an ongoing project by the Cochrane Climate-Health Working Group to develop MEDLINE search filters for the seven major exposure pathways through which human health is affected by climate change. We extracted 2,327 studies and are currently developing and validating search filters per pathway.
Conclusions: We will present the development and validation of climate health search filters for different exposure pathways and will discuss challenges encountered.
Patient, public and/or healthcare consumer involvement: We describe information retrieval methods; therefore, patients or healthcare consumers were not involved. However, robust search methods for climate health ES are of interest to the global health community.

Background: According to the World Health Organization (WHO), poor quality of care in low- and middle-income countries contributes to 5.7-8.4 million deaths annually. Failure to apply and access reliable healthcare information is a leading cause of preventable deaths and suffering worldwide. Healthcare Information for All (HIFA) is a global social movement born in 2006, with a commitment to promote communication, understanding and advocacy among all stakeholders in the production, exchange and use of reliable healthcare information.
Objectives: To understand the HIFA mission, vision and strategy to promote universal access to reliable healthcare information and prevent misinformation and to identify collaborative approaches to strengthen global health communication and coproduction of evidence-based health information.
Methods: HIFA utilises a “Theory of Change” model whereby HIFA’s inputs, activities and outputs translate into essential outcomes and impact (Figure 1). The impact of HIFA contributes to meeting basic needs, empowering communities and achievement of global health goals like Universal Health Coverage and Sustainable Development Goals (Figure 2). HIFA evaluates ‘value co-creation’ across several domains, using quantitative and qualitative indicators, including value for HIFA members and participating organisations.
Results: With more than 20,000 multidisciplinary members from more than 180 countries, 274 country representatives, interacting over 6 global forums in 4 different languages, and 434 international support organisations, HIFA continues to evolve as a robust global health information system (Figure 3). Accomplishing instrumental milestones (Figure 4), the HIFA strategy describes seven strategic shifts necessary to accelerate progress towards universal access to reliable health information (Figure 5)
Conclusions: With a visionary future, HIFA’s priority is to convene key stakeholders to accelerate progress towards universal access to reliable healthcare information. Amidst challenges with funding and staff infrastructure, HIFA continues to engage strategically with the WHO to develop initiatives such as the WHO-HIFA Collaboration Plan (Figure 6) to identify best practices, opportunities and challenges from stakeholders and support multilingualism in global health. Healthcare consumer involvement: HIFA is an organisation for the people, by the people. The members represent the full spectrum of healthcare providers and consumers including patients, caregivers, general public, policymakers, researchers, social scientists, etc., whose contributions are fundamental to the advocacy of HIFA’s vision.

Figure 1.png
Figure 2.png
Figure 3.png
Figure 4.png
Figure 5.png
Figure 6.png

Background: Previous meta-epidemiological studies based on Western medicine have found several trial characteristics influencing the effect estimates of randomized controlled trials (RCTs). However, whether there were similar associations in Chinese herbal medicine RCTs is unclear. Furthermore, the characteristics specific to Chinese medicine have not been investigated.
Objectives: This meta-epidemiological study aimed to explore potential trial characteristics associated with treatment effect estimates in Chinese herbal medicine RCTs, including both Chinese medicine–specific characteristics and characteristics common to Chinese and Western medicine.
Methods: Three English databases and two Chinese databases were searched for systematic reviews on Chinese herbal medicine treatment published from January 2011 to July 2021. The difference in effect estimates was presented as the ratio of odds ratio (ROR) for binary outcomes and the difference in standardized mean differences (dSMD) for continuous outcomes. The two-step method was used for meta-epidemiological analyses.
Results: We included 91 systematic reviews, comprising 1,338 RCTs. For binary outcomes, RCTs with syndrome differentiation (ROR: 1.23, 95% CI: 1.07, 1.39), adopting Chinese medicine formula (ROR: 1.19, 95% CI: 1.03, 1.34), of low risk of bias in incomplete outcome data (ROR: 1.29, 95% CI: 1.06, 1.52) and selective outcome reporting (ROR: 1.12, 95% CI: 1.01, 1.24), and having sample size ≥100 (ROR: 1.23, 95% CI: 1.04, 1.42) tended to have greater effect estimates. For continuous outcomes, RCTs based on Chinese medicine diagnostic criteria (dSMD: 0.23, 95% CI: 0.06, 0.41), judged as high/unclear risk of bias on allocation concealment (dSMD: -0.70, 95% CI: -0.99, -0.42), rated as low risk of bias for incomplete outcome data (dSMD: 0.30, 95% CI: 0.18, 0.43), being single-center (dSMD: -0.33, 95% CI: -0.61, -0.05), not using intention-to-treat data analysis (dSMD: -0.75, 95% CI: -1.43, -0.07), and without funding support (dSMD: -0.22, 95% CI: -0.41, -0.02) showed larger effect estimates.
Conclusions: Our findings will provide empirical evidence for developing a critical appraisal tool specific to assessing risk of bias of Chinese herbal medicine RCTs.
Patient, public and/or healthcare consumer involvement: No patient or member of the public was involved in conducting this study.

Background: Shared decision-making (SDM), a process in which clinicians and patients work together to select treatments based on clinical evidence and the patient’s informed preferences, is the optimal outcome to evaluating evidence-based practice. The Accreditation Council for Graduate Medical Education (ACGME) also included SDM as an essential skill of Competencies Interpersonal and Communication Skills for physicians to provide patient-centered care.
Objectives: This study aimed to codesign and coproduce an evidence-based SDM simulation curriculum with clinical teachers, standard patients (SP), and undergraduate medical students to fulfill the needs of students and patients. We also evaluated students’ learning effects quantitatively and qualitatively.
Methods: We first performed SP focus group interviews to explore patient needs for SDM and then held five consensus meetings using the Delphi method with four representative students, three SP, and four teachers to determine core competencies, teaching objectives, course contents, and learning outcomes; finally, we used a paired t-test to evaluate the progress of students’ SDM skills improvement in objective structured clinical examinations (OSCE) quantitatively, as well as content analyzed the learning effects from students’ interviews and teachers/SP’s focus groups qualitatively.
Results: A total of 122 undergraduate medial students (55% male, mean age 24.2 years) with mean clinical experience of 10.1 months completed the course and pre-/post-OSCE multirater feedback assessments. After training, students’ SDM skills were significantly improved with change scores of 2.8, 9.5, and 7.8 among students, SP, and teachers, respectively. SDM self-efficacy of students and patients, as well as teachers’ ratings of the scores of entrustable professional activities, were also significantly improved. The content analysis of students’ interviews and SP and teachers’ focus groups indicated that the codesign and coproduction of an SDM curriculum with participants’ feedback could facilitate students’ empathy and weaken the medical autonomy mindset, as well as deepen their attitude, knowledge and behaviors of patient-centered care.
Conclusions: To alleviate medical paternalism and facilitate patient-centered care, clinical teachers, medical students and patients must have a voice in developing highly impactful curricula in the healthcare practice. Our codesigned and coproduced SDM curriculum could help improve students’ subjective and objective SDM skills, as well as the SDM self-efficacy quantitatively and qualitatively.

Background: Deprescribing is the process of discontinuing an inappropriate medication. Research into deprescribing has grown rapidly in recent years, including a number of Cochrane systematic reviews (CR). It is important to study whether discontinuation can be done successfully without an increased risk of a return of symptoms, an event, or need to restart because this is important for clinicians and patients. In this review, we want to review primary outcomes in deprescribing CR.
Objectives: To review reported primary outcomes in CR of deprescribing specific medications and to assess which CR included successful discontinuation rate and how this is defined. We also aimed to assess whether CR reported the proportion of patients who successfully stopped, frequency of relapse or event, and of restarting medication as primary outcome.
Methods: We screened all published CR in the Cochrane Database of Systematic reviews until February 2023. Reviews were included if they aimed to assess discontinuing a specific medication (classes) or interventions to support discontinuing medication. We extracted all primary outcomes.
Results: We included 14 CR related to deprescribing (9 CR assessing discontinuation of 1 medication such as cholinesterase-inhibitors, or antipsychotics in dementia, and 5 CR of interventions for discontinuing medication such as approaches to stop antidepressant and interventions for the reduction of prescribed opioid use in chronic pain). All included more than one primary outcome, which resulted in a variety of primary outcomes. Four (29%) reviews used a primary outcome related to ‘successful discontinuation rate’, e.g., the number of patients who successfully stopped the specific medication; however, studies varied in the definition of successful discontinuation. The nine remaining reviews used frequency of symptomatic relapse or occurrence of a medical event as the primary outcome.
Conclusions: A broad range of primary outcomes was reported in CR of deprescribing medication. A few CR have examined successful discontinuation rate, with variation in definition. The results support a broader discussion with involvement of patients and clinicians on primary outcomes and how successfully discontinuation needs to be defined in deprescribing Cochrane reviews. Patient, public, and/or healthcare consumer involvement: Need to be involved in the broader discussion.

Background: Increasingly, systematic review searchers need to limit searches to specific populations. This is a difficult task given the wide array of synonyms, semantic drift over time, and poor population reporting in abstracts. This is especially evident in the terms associated with Black Persons Living in the United States (BPLiUS), i.e., US-born Black Americans and US immigrants of Sub-Saharan African ancestry. Objective: To increase the efficiency, reliability, and inclusivity of population searches, we will develop a novel method to create and validate search filters to retrieve biomedical and health sciences–related information on BPLiUS persons in Ovid MEDLINE and PubMed.
Methods: We will use a handsearch method to create our gold standard citation set. To gather the most representative citation sample, we will draw from Black-focused journals along with journals in the following categories: race/ethnicity-focused, cultural anthropology/psychosocial, representative high-impact journals, clinical/health systems, and, lastly, regional/state medical journals. To capture the breadth and depth of Black ethnonym terms over time, we will sample citations in the above categories in 20-year increments—i.e., 1955 (Jim Crow era), 1975 (Civil Rights era), 1995 (African American era), 2015 (People of Color era), and 2021 (BIPOC era)—to capture more recent language. We will include explicit population terms (e.g., Blacks), implicit terms (e.g., minorities), and contextual terms occurring with explicit/implicit terms (e.g., immigrant). To identify filter terms, we will text-mine our test set and Black-focused journal citations using the AntConc software; consult the National Library of Medicine’s MeSH database; identify example search filters; review published systematic review search strategies; and analyze potential implicit words/phrases for inclusion. The gold standard set will be divided into test and validation sets. The filters will be internally and externally validated. We will create a highly sensitive (recall)-maximizing filter (prespecified performance minimum ≥ 75%), a precision-maximizing filter, and a best balance of sensitivity and precision filter. Relevance and importance to patients: This article contributes to methods resulting in more robust evidence production.

Background: PROBAST (Prediction model Risk Of Bias Assessment Tool) has been launched in January 2019. Since then, there has been much progress and literature on the methodology for prediction modelling in general and on the use of artificial intelligence (AI) and machine learning (ML) techniques in this field in particular. Hence, it is timely to develop PROBAST+AI that applies to studies on developing and evaluating (validating) multivariable diagnostic and prognostic prediction models using any data analytical (statistical) AI or ML technique.
Objectives: To develop and test PROBAST+AI, a quality and risk of bias assessment tool that applies to studies on developing and evaluating (validating) multivariable diagnostic and prognostic prediction models using any AI/ML data analytical technique.
Methods: Using a Delphi process (at least 3 survey rounds) among a diverse and large (>200) group of key stakeholders and experts on prediction model and AI/ML, we identified the relevant items for PROBAST+AI. Participants gave their opinion on a large series of quality, risk of bias, and applicability domains and signalling questions, using a 5-Point Likert scale. Participants were also asked to add new suggestions using free-text boxes.
Results: Currently, two Delphi rounds have been conducted, and a third is planned in the spring of 2023, followed, if deemed necessary, by a final consensus expert meeting. The results and PROBAST+AI will be presented at the meeting. Conclusion: PROBAST+AI will provide key stakeholders in diagnostic and prognostic prediction models (including primary study authors, systematic reviewers, guideline developers, healthcare providers, prediction model developers, and patients), guidance on the relevant methodological aspects of diagnostic, and prognostic prediction model studies using any AI/ML data analytical technique. Patient or healthcare consumer involvement: Part of the PROBAST+AI Delphi and working group. Our project has methodological implications for systematic reviews. The images we use when sharing evidence are important. Images can convey ideas quickly, simply, more universally, and sometimes more powerfully than text. They can engage and inform our target audiences of people who may use the evidence for their health decisions. Well-chosen images can draw people in, arouse emotions, and influence someone’s decision whether to read on. They can also enhance or reinforce written information and contribute to the credibility of the content. By contrast, bad image choices can repel, misrepresent, and undermine the trustworthiness of the written information they accompany, as well as of the organisation sharing the image. Here, we will encourage people to reflect on the impact of images by exploring some positive examples of image choice, as well as some problematic ones. We will share some key considerations to guide your image choices, including pitfalls to look out for. We will explore, for example, the importance of depicting a diverse range of people to ensure wide representation and inclusivity, accurately depicting the evidence, choosing realistic and relatable images that depict topics sensitively, and avoiding images that stigmatize or reinforce stereotypes. We will also consider how alternatives to stock images, especially art made by people about their health experiences, can be particularly powerful and relatable. As we look at the impact of image choice, we will draw on Cochrane’s Guide to Choosing Images for Sharing Evidence, produced by Cochrane UK with input from a diverse global advisory group which is now a Cochrane learning resource. Patient, public, and/or healthcare consumer involvement: One of the abstract authors is a consumer. The Guide we drew on was developed with the help of a global advisory group which included healthcare consumers. We include, and advocate for, images made by people to reflect their health experiences and talk about involving consumers in image choice.

Background: Active placebo controls are designed to mimic the nontherapeutic adverse effects of drugs in randomised trials. Active placebos are rarely used but could reduce the risk of bias due to unblinding.
Objectives: We aimed to estimate the difference in drug effects when an experimental drug is compared with an active placebo versus a standard placebo control intervention, and to explore causes for heterogeneity. In the context of a randomised trial, this difference in drug effects can be estimated by directly comparing the effect difference between active placebo and standard placebo intervention. Design: A systematic review.
Methods: We searched PubMed, CENTRAL, Embase, and other sources up to October 2020 and included randomised trials directly comparing active placebo versus standard placebo. Our primary inverse-variance, random-effects meta-analysis used standardised mean differences (SMDs) of participant-reported outcomes at earliest post-treatment assessment. An SMD <0 favoured the active placebo. We conducted seven sensitivity analyses, four subgroup analyses and a meta-regression. In secondary analyses, we investigated observer-reported outcomes, harms, attrition, and cointervention outcomes.
Results: We included 21 trials (1,462 participants). The pooled SMD in our primary analysis was -0.08 (95% CI -0.20 to 0.04; Figure 1). The difference was more pronounced and statistically significant in the sensitivity analyses restricting to trials with overall low risk of bias (pooled SMD -0.24; 95% CI -0.34 to -0.13) and using a fixed-effect model (pooled SMD -0.15; 95% CI -0.23 to -0.07). The pooled SMD of observer-reported outcomes was similar to the primary analysis. The pooled odds ratio for harms was 3.08 (95% CI 1.56 to 6.07), and for attrition, 1.22 (95% CI 0.74 to 2.03). Cointervention data were limited. Meta-regression found no statistically significant association with adequacy of the active placebo or risk of unintended therapeutic effect.
Conclusions: We did not find a statistically significant difference between active and standard placebo interventions in our primary analysis, but the result was imprecise and compatible with a difference ranging from important to irrelevant. Furthermore, the result was not robust, for example, when restricting to trials with overall low risk of bias.
Patient, public and/or healthcare consumer involvement: None.

figure 1.png

Background: Translating evidence into clinical knowledge and improved health is challenging. A collective impact approach, bringing together stakeholders through common goals, deploying mutually reinforcing activities and shared measurement systems, provides a framework for more effective translation. The Targeted Therapies Alliance (TTA) designed, developed and implemented a national program in Australia to improve the quality use of biological medicines using a collective impact approach to achieve better health outcomes for people with inflammatory arthritis, inflammatory bowel disease and psoriasis. The TTA included nine member organisations and four affiliate organisations representing consumer and clinical audiences, program development expertise and implementation capability.
Objectives: Program objectives were to optimise (1) first-line disease-modifying antirheumatic drug (DMARD) therapy; (2) first-choice biologic; (3) biosimilar prescribing and dispensing; (4) biologic dosage; and (5) glucocorticoid and analgesic use.
Methods: Steps included (1) consortium development with key stakeholders; (2) research and review of practice/behavioural drivers; (3) using behavioural change theory and implementation science to codesign program objectives, key messages and strategy; (4) development and testing interventions with audiences; and (5) promotion of interventions via consortium channels.
Results: Seventy interventions supporting key messages were implemented between September 2020 and September 2022 for consumers, rheumatologists, gastroenterologists, dermatologists, immunologists, pharmacists, nurses and hospital/jurisdictional decision-makers. They were applicable to specialist practices, private and public hospitals, hospital and community pharmacies, consumers in these settings and wherever self-management occurred. All resources were available from a dedicated hub (www.nps.org.au/bdmards) with links to TTA member websites, and vice versa. Endorsed national living guidelines were an engagement point for the rheumatology program addressing priority questions for rheumatologists and providing the basis for developing interventions for health professionals and consumers. Positive impacts on knowledge, confidence and practice/behaviour were seen across all therapeutic areas, despite already high levels of knowledge and practice in some areas. Outcome evaluation should be performed in 2025, ~24 months after program implementation.
Conclusions: The consortium-based approach provided a collaborative model for developing a multifaceted program addressing multiple perspectives to optimise DMARD use.
Patient, public and/or healthcare consumer involvement: Consumers were partners in all stages: governance, research, program design, intervention design, testing, implementation and evaluation.

Background: This work links to the international Core Outcome Measures in Effectiveness Trials (COMET) Initiative. Core Outcome Sets (COS) can reduce research waste by promoting consistency in outcome reporting. They represent the minimum that should be measured and reported for a particular health condition. COS should be relevant to all research stakeholders, including patients and the public. Patient participation in COS development is considered crucial to ensure core outcomes are relevant to patients and other healthcare decision-makers. There has been a dramatic increase in patient participation over recent years. Once developed, COS need to be used to reduce rather than increase research waste. Patient organisations are influential in policy decisions in healthcare, and many provide research funding for clinical studies and collaborate closely with the research community in their disease area.
Objectives:
• Describe how COS developers work with patient organisations in developing COS and promoting COS uptake.
• Identify how patient organisations’ influence could be harnessed to enhance COS uptake, normalise their use, and maximise collaborations between COS developers and patient organisations.
Methods: We consulted with COS developers via a survey to explore how they worked with patient organisations in developing COS and promoting their use. Additionally, we held meetings with four key stakeholders from patient organisations to understand what they knew of COS, any previous involvement in COS, and their views on how their organisation might support COS uptake.
Results: COS developers often described working with patient organisations in activities, such as identifying public research partners or recruiting patient participants. However, they rarely described patient organisations changing their practice as a result of being involved in a COS project. The patient organisations interviewed expressed keen interest in COS and could see the relevance and importance of COS uptake, although some challenges were noted.
Conclusions: Whilst COS developers often turn to patient organisations for help, these partnerships could provide opportunities for promoting COS uptake. Indeed, patient organisations appear eager to hear about COS, recognising the importance of COS uptake. Patient, public, and/or healthcare consumer involvement: We worked in partnership with two patient organisations, one partner is also a patient research partner.

Background: The development of systematic reviews (SRs) is resource intensive. The Sustainable Knowledge Platform (SK platform) is a new system that streamlines all the steps of SRs, applying technology to make the process efficient. It includes a comprehensive taxonomy, as well as off-the-shelf Boolean strategies and predeveloped artificial-intelligence algorithms for each term. This platform decreases the workload, but it is not clear whether this is at the expense of compromising quality.
Objectives: To compare the performance of the SK platform against the traditional approach, to search and select studies in the context of an SR of trials of aspirin for the prevention of colorectal cancer in average-risk adults.
Methods: For the traditional approach, we designed a standard search strategy for MEDLINE, Embase and CENTRAL up to January 2023. For the alternative approach, we matched the components of the review question to the relevant terms of the taxonomy, ran the predeveloped Boolean strategies and applied the automated classifiers. Title/abstracts will be assessed by a single reviewer for each approach, and the full text of the potentially eligible studies will be assessed in duplicate (independent reviewers for each approach). As a measure of workload, we will estimate the total number of records. We will calculate the sensitivity and specificity of the alternative approach using the traditional approach as the reference standard. We will report the number of studies detected by the alternative approach only and will attempt to explore the potential reasons (e.g., different repositories, different strategies).
Results: The traditional approach identified 2,766 potentially eligible records. The alternative approach identified 916 records based on the predeveloped search strategies. After applying the automated classifiers, the number to screen decreased to 189. The complete results will be reported during the Colloquium.
Conclusions: New approaches may save time and resources. However, it is important to know whether this efficiency gain comes at a cost in quality. If our results are positive, the SK platform would offer an alternative worth exploring in future research.
Patient, public and/or healthcare consumer involvement: Using technology in the SR process may help to produce rigorous, relevant and rapid evidence synthesis.

Background: Systematic reviews (SRs) of patient-reported outcome measures (PROMs) are an important way to select the appropriate PROMs. High-quality SRs can provide a comprehensive overview of the PROMs and provide evidence-based recommendations for pediatricians. However, because the risk of bias evaluation of included studies, data extraction and analysis are different from other types of SRs, and there are currently no tools applicable to evaluate risk of bias of SRs of PROMs.
Objectives: We aim to develop an evidence-based methodological quality evaluation tool for SRs of PROMs using an explicit, systematic and transparent method: a critical apPRaisal tOol for systeMatic reviews of patient-repOrted ouTcome mEasures (PROMOTE).
Methods: We will develop the PROMOTE in five stages using established methods. First, we establish an international and multidisciplinary advisory committee of experts who will oversee the project and provide methodological support. Second, we will develop an initial checklist item list using a scoping review to describe the entries of existing SRs of risk of bias checklists and the methodological characteristics of published SRs of PROMs. Third, 10 SRs of PROMs will be randomly selected using the random number table, and methodological quality will be evaluated by four researchers using the initial checklist item list. The initial items will be modified, added or deleted through focus group discussion. Fourth, we will use a two-round Delphi designed to achieve a high level of expert consensus on the list of items for PROMOTE. We will recruit 30 multidisciplinary international experts, including journal editors, clinical researchers, patient-reported outcomes researchers, guideline developers, policy makers and patients who will participate in the Delphi process. Fifth, after reaching consensus on the items included, we will produce and disseminate PROMOTE risk of bias checklist. Discussion: A systematically developed critical appraisal tool for SRs of PROMs would facilitate the selection of appropriate PROMs in clinical practice or research. A well-defined and transparent critical appraisal tool would further improve the methodological quality of SRs while reducing study waste and ultimately supporting optimal patient health outcomes. Patient or consumer involvement: Patients and consumers will be directly involved in the Delphi process as expert participants.

Background: Rehabilitation is a health strategy to enable people with disabilities to reach and maintain their optimal physical, sensory, intellectual, psychological, and social functional levels. Cochrane Rehabilitation aims to ensure that all rehabilitation professionals apply Evidence-Based Clinical Practice, combining the best available evidence from high quality Cochrane systematic reviews, with their own clinical expertise and patients’ preferences. Cochrane Rehabilitation’s priority is to share high quality health information in multiple languages. The translated content is intended to reach rehabilitation professionals and students, researchers, patients and their families and health policy makers. Blogshots are images with key information from a single Cochrane review.
Objectives: The aim of this work is to present the collaborative model of Cochrane Rehabilitation’s blogshot translations and their dissemination.
Methods: The process of creating new translation partnerships starts with spontaneous requests from potential translation partners or with an initial contact by the Field Co-Director or Translation manager. For each language, we seek to involve the National Society of Physical and Rehabilitation Medicine and the local Cochrane Geographical Group (see table 1). These partnerships ensure the quality of translations and enable effective dissemination throughout local communities. The partnership continues with constant communication between translation managers and translation partners. Production times of each translation partner are respected as they are all volunteers and work as full-time clinicians, professors, and researchers.
Results: Until March 2023, Cochrane Rehabilitation has translated a total of 531 blogshots: 92 in Dutch, 87 in French, 80 in Italian, 74 in Spanish, 56 in Hungarian, 52 in Croatian, 23 in German, 20 in Serbian, 20 in Polish, 11 in Simplified Chinese, 9 in Japanese, 5 in Malay and 2 in Portuguese. Blogshot translations are available at the Cochrane Rehabilitation website, social media, newsletters and Cochrane archives on Tumblr.
Conclusions: Blogshot translations reach a huge number of scientists, rehabilitation professionals, consumers, and policy makers. Partnerships with national societies and Cochrane geographic groups ensure the quality of translations. The success of partnerships is due to close collaboration and clear communication. We aim to continue expanding this collaborative model to include more countries and to increase the dissemination of blogshots.

Table 1.jpg

Background: Plain language summaries (PLSs) are condensed summaries of research articles written in plain, easy-to-understand language aimed at a non-scientific audience. Most health research is not written with the public in mind, as it contains jargon and acronyms and is usually written at a high reading level. PLSs are an increasingly important tool for disseminating reliable health information in a way that a lay audience can understand.
Objectives: Our first objective was to provide a better understanding of the PLS instructions currently available to authors. Our second objective was to compare published PLSs with author instructions from health journals.
Methods: Initially, we conducted a scoping review of 534 health and biomedical journals to locate journals that contained author instructions for writing a PLS. We searched journals from 11 categories linked to the top 10 noncommunicable diseases. We included journals that published in English and recommended the inclusion of a PLS. We reported characteristics and elements of author instructions such as the label used, whether the PLS was optional or mandatory, word, count, length, and use of jargon. We followed this up with a study to determine the level of compliance between author instructions and PLS for these same journals. Before assessed compliance, we had to determine the extent to which PLSs were included with original articles from the journals.
Results: From our scoping review, we located 27 journals (5.1%) that had author instructions for PLSs. Author instructions were highly heterogeneous between journals. Only one journal recommended one consumer involvement in PLS development and one the use of a readability tool. In our follow up study, approximately half (53.8%) achieved medium or high levels of compliance. A medium rating was 51%-79% and high ≥ 80% compliance between PLSs and author instructions. PLSs were mandatory for seven journals, four of which had low or very low compliance.
Conclusions: This research was an important step in the development of evidence-based, consistent, and uniform instructions for writing PLSs that are easily understood by consumers. Patient, public, and/or healthcare consumer involvement: A consumer was engaged as part of the research team, offering knowledge user insight.

Background: Systematic reviews of randomised trials are the gold standard of evidence for interventions. Therefore, it is imperative that trials are open to all relevant patients and that reviews include all appropriate trials. In gynaecology, it is common that patients with the same diagnosis experience differing symptoms. Where an intervention may benefit the underlying condition, how do trialists define eligibility and select outcomes in the likely scenario that potential participants have no symptoms in common? Similarly, how often do reviewers exclude trials because outcomes are relevant to other symptoms?
Objectives: To identify inefficiencies and sources of research waste in the design of recent gynaecology trials and Cochrane reviews.
Methods: We searched for Cochrane reviews in polycystic ovarian syndrome (PCOS) and endometriosis as exemplar conditions. Reviews were included if the intervention was intended to treat all condition-specific symptoms. We restricted to trials published since 2012 to consider ‘current’ approaches. For each trial we recorded the number of potentially eligible participants excluded because of their symptoms. Similarly, for each review we recorded the numbers of trials and participants excluded unnecessarily.
Results: There were 89 distinct PCOS trials in 13 reviews and 13 endometriosis trials in 11 reviews. Most trials restricted eligibility to participants with specific symptoms (55% PCOS, 46% endometriosis). Reviews excluded 27% of available trials (n=27 in PCOS, n=1 in endometriosis) based on prespecified outcomes not relevant to the symptoms of randomised participants. These excluded trials could have added 1,914 participants (17%) and 44 participants (3%) to the evidence base for PCOS and endometriosis, respectively.
Conclusions: Cochrane reviews excluded a quarter of available trials. Excluding trials because prespecified review outcomes are not relevant to randomised participants greatly reduces the evidence base. The issue of outcome reporting bias is now well recognised. We present a subtly different issue of outcome reporting that may lead to imprecision, rather than bias, in the context where an underlying condition presents diverse symptoms. Further work is ongoing to propose more efficient strategies in this context for both trialists and reviewers. Patient involvement: Involvement since idea inception; patients continue to help guide project progress through regular meetings.