Background: Assessing how much confidence users can place in qualitative evidence synthesis (QES) findings is important for producing trusted qualitative evidence on patients’ experiences and perspectives of health issues and interventions. Applying the GRADE-CERQual approach to assess confidence in the evidence is a required step in a Cochrane QES. Use of GRADE-CERQual has increased rapidly globally.
Objectives: We evaluated how GRADE-CERQual has been used in the literature by, firstly, describing GRADE-CERQual’s uptake in QES; and, secondly, developing and applying reporting and fidelity criteria to identify good examples and areas for improvement.
Methods: We undertook citation searches in six databases for 18 key publications on the GRADE-CERQual approach and also used keyword searches. Two researchers screened the output, and publications identified as evidence syntheses that used GRADE-CERQual went on to full-text coding and charting. Reporting and fidelity criteria were developed and then applied using NVivo12 software. We used qualitative content analysis and descriptive statistical approaches to analyse the data.
Results: Our searches identified 1,312 records. Two hundred thirty-three of these were reviews claiming to have applied GRADE-CERQual. Approximately 41.6% (97 studies) had seriously misapplied the GRADE-CERQual approach and were excluded from further fidelity and reporting assessment. Serious misuses were applying GRADE-CERQual in quantitative evidence synthesis, interpreting it to be a critical appraisal tool, and applying it to review findings as a whole, instead of individual review findings. One hundred thirty-six studies applied it as intended to individual review findings. We categorised the most common reporting issues into three broad areas: labelling, terminology, and completeness. The most common fidelity issues centred on the conceptualisation and application of the four GRADE-CERQual components. Fewer fidelity concerns were identified in syntheses citing the most recent guidance compared to those citing earlier publications. Conceptual challenges were identified around distinguishing between quality and confidence, between themes and review findings, and between making assessments at the study level versus the review finding level.
Conclusions: Findings from this evaluation help to inform new agendas for the field of QES, identify topics for further GRADE-CERQual guidance development, and support review authors to avoid common pitfalls and improve reporting and fidelity.