Using GRADE-CERQual to assess confidence in findings from qualitative evidence syntheses: how well are review authors applying the approach?
2Gender and Women’s Health Unit, Centre for Health Equity, School of Population and Global Health, The University of Melbourne, Australia
3UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP), Department of Sexual and Reproductive Health and Research, World Health Organization, Switzerland
4School of Health and Related Research (ScHARR), Faculty of Medicine, Dentistry and Health, University of Sheffield, United Kingdom
5School of Medical and Health Sciences, Bangor University, United Kingdom
6Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, The University of Melbourne, Australia
7The Centre for Epidemic Interventions Research (CEIR), Norwegian Institute of Public Health, Norway
8Department of Health Sciences Ålesund, Norwegian University of Science and Technology; South Africa Medical Research Council; and Norwegian Institute of Public Health, Norway
Assessing how much confidence users can place in qualitative evidence synthesis (QES) findings is important for producing trusted qualitative evidence on patients’ experiences and perspectives of health issues and interventions. Applying the GRADE-CERQual approach to assess confidence in the evidence is a required step in a Cochrane QES. Use of GRADE-CERQual has increased rapidly globally.
Objectives: We evaluated how GRADE-CERQual has been used in the literature by, firstly, describing GRADE-CERQual’s uptake in QES; and, secondly, developing and applying reporting and fidelity criteria to identify good examples and areas for improvement.
Methods: We undertook citation searches in six databases for 18 key publications on the GRADE-CERQual approach and also used keyword searches. Two researchers screened the output, and publications identified as evidence syntheses that used GRADE-CERQual went on to full-text coding and charting. Reporting and fidelity criteria were developed and then applied using NVivo12 software. We used qualitative content analysis and descriptive statistical approaches to analyse the data.
Results: Our searches identified 1,312 records. Two hundred thirty-three of these were reviews claiming to have applied GRADE-CERQual. Approximately 41.6% (97 studies) had seriously misapplied the GRADE-CERQual approach and were excluded from further fidelity and reporting assessment. Serious misuses were applying GRADE-CERQual in quantitative evidence synthesis, interpreting it to be a critical appraisal tool, and applying it to review findings as a whole, instead of individual review findings. One hundred thirty-six studies applied it as intended to individual review findings. We categorised the most common reporting issues into three broad areas: labelling, terminology, and completeness. The most common fidelity issues centred on the conceptualisation and application of the four GRADE-CERQual components. Fewer fidelity concerns were identified in syntheses citing the most recent guidance compared to those citing earlier publications. Conceptual challenges were identified around distinguishing between quality and confidence, between themes and review findings, and between making assessments at the study level versus the review finding level.
Conclusions: Findings from this evaluation help to inform new agendas for the field of QES, identify topics for further GRADE-CERQual guidance development, and support review authors to avoid common pitfalls and improve reporting and fidelity.