Extracorporeal membrane oxygenation (ECMO) emergencies require skilled clinical specialist (CS) who manage ECMO circuits. While tools for assessing CS skills have been published, there is significant variation in protocols and circuit design. This study aims to further develop these checklists to produce a generalizable ECMO skill assessment with adequate validity evidence to support its use as a summative evaluation tool.

An initial survey determined variation in ECMO circuit components and configurations, and the original checklists and simulations were altered through a modified Delphi process. The finalized checklist and simulation were then assessed for validity and reliability. Three trained raters assessed ten simulations from five subjects at two different institutions using two circuit designs. Data analysis was conducted using a fully crossed subject x rater x circuit generalizability (G) and decision (D) study.

The G-study coefficient was 0 with 0% variance across subject and circuit. The greatest variance was among raters (28.7%). Significant variance was also associated with the subject and pump type relationship (27%).

Despite the rigorous process used to modify the assessment, generalizability was poor. Lack of familiarity with center-specific circuit design played a key role. Future endeavors in ECMO skill assessment should focus either on developing and validating site-specific tools or standardizing circuit designs.