Extracorporeal membrane oxygenation (ECMO) has been extensively used during both the H1N1 influenza pandemic and COVID-19 pandemic.

When the number of patients receiving ECMO support and ECMO centres sharply increased, many ECMO guidance documents were published.

High-quality guidance documents can provide explicit recommendations to healthcare professionals and improve the outcomes of patients, whereas low-quality guidance documents might mislead clinical practice or restrict the promotion of guidance documents.

The rigour of the methodology, and an adequate dissemination and implementation strategy, are important components of high-quality guidance documents.

However, the quality of published ECMO guidance documents has not been systematically reviewed, which could confuse healthcare professionals when choosing reliable guidance documents and recommendations in clinical practice.

The objective of this systematic review was to use the Appraisal of Guidelines Research and Evaluation II (AGREE II) instrument to evaluate published ECMO guidance documents to help clinicians efficiently utilise the guidance documents and provide valuable information to develop and update ECMO guidance documents in the future.

Five scientific databases and grey literature databases were searched from inception to December 18, 2022. We implemented the Population, clinical indication(s), and condition(s); Intervention(s); Comparator(s), comparison(s), and (key) content; Attributes of the CPG; Recommendation characteristics and “other’’ considerations (PICAR) framework to define guidance document (including guidelines, consensus, and statements) eligibility criteria.

 After online training, three appraisers independently assessed the quality of guidance documents using the AGREE II instrument.

 Double weight was assigned to domain 3 (rigour of development) and domain 5 (applicability).

 A guidance document was ‘recommended’ if the overall score was >60%, whereas the scores of ‘recommended with modifications’ were between 30% and 60%. A guidance document was ‘not recommended’ if the overall score was <30%. In addition, we defined a guidance document as high quality if the scaled domain score was >70% for domain 3 and >50% for all other domains.

Statistical significance among groups was determined by analysis of variance (anova) tests whenever data met normal distribution and equal variance assumptions. Otherwise, Mann–Whitney U-tests were applied. Full details on methods are provided in Supplementary files 2 and 3.

We identified 81 guidance documents, of which 37 were updated versions. The AGREE II score of included guidance documents ranged from 17.5% to 77.3%, with a median of 42.6%. The domain with the highest score was domain 4 (‘clarity of presentation’; median 66.7 [inter-quartile range, IQR: 57.4–79.6]), followed by domain 1 (‘scope and purpose’; median 61.1 [IQR: 53.7–75.9]). The scores of the other four domains, including domain 6 (‘editorial independence’; median 50.0 [IQR: 33.3–77.8]), domain 2 (‘stakeholder involvement’; median 44.4 [IQR: 33.3–63.0]), domain 3 (median 33.3 [IQR: 20.8–53.5]), and domain 5 (median 30.6 [IQR: 25.0–41.7]), were <60%. (Fig 1). Fourteen guidance documents were recommended for use without modification, but only four met the criteria for high quality, suggesting a discrepancy between overall rating and high-quality status. Only one specialised ECMO guidance document was recommended for use without modification, but it was not high quality. Subgroup analysis showed that AGREE II scores of initial versions of guidance documents were significantly lower than those of the updated versions in domains 1 (59.3 vs 68.9), 2 (42.2 vs 52.1), 3 (31.0 vs 45.5), and 4 (63.5 vs 71.5). The guidance documents with funding had higher scores in domains 1 (70.3 vs 60.7), 2 (58.2 vs 41.6), 3 (48.5 vs 32.7), and 6 (69.0 vs 42.7) compared with those that were unfunded. Moreover, specialised ECMO guidance documents had lower scores in domains 1 (58.8 vs 65.6), 3 (27.0 vs 41.8), 4 (61.0 vs 69.5), and 5 (28.9 vs 35.0) compared with non-specialised ECMO guidance documents. In the subgroup analysis, domains 1 and 4 were still the top two domains in scoring, the results of which were similar to those of the primary analysis. The ICC was 0.81 (0.75–0.85), suggesting good consistency among the three appraisers.