Automated Versus Instructor Checklist Scoring in Emergency Medicine Simulation: A Prospective Agreement Study
Podium presentation, ACEP 2026, Chicago
Prospective, single-blinded agreement study in the emergency medicine residency at Hadassah Ein Kerem Medical Center, affiliated with the Hebrew University of Jerusalem. Fifteen recorded encounters across two 20-minute high-fidelity scenarios — trauma with methanol poisoning, and an unconscious patient with carbon monoxide toxicity. Checklists were written in the national board examination format and revised by two emergency medicine specialists who serve as examiners for the Israeli Association of Emergency Medicine. Two blinded, board-certified emergency physicians reviewed every recording to set the reference standard. Approved by the Helsinki Committee, Faculty of Medicine, Hebrew University of Jerusalem (IRB #26112025).
70.2%
DeepBrief
63.4%
Instructor, in real time
Exact ordinal agreement with the expert reference standard, across 372 scored items. Agreement within one ordinal category was 96.5% and 94.4%.
Weighted kappa
Favoured automated scoring in both scenarios: 0.38 (95% CI 0.22–0.50) vs 0.28 (0.14–0.41) in Case 1, and 0.79 (0.68–0.87) vs 0.65 (0.50–0.78) in Case 2. All p < 0.001.
