Performance of machine learning versus the national early warning score for predicting patient deterioration risk: a single-site study of emergency admissions

Watson, Matthew; Boulitsakis Logothetis, Stelios; Green, Darren; Holland, Mark; Chambers, Pinkie; Al Moubayed, Noura

Abstract

Objectives Increasing operational pressures on emergency departments (ED) make it imperative to quickly and accurately identify patients requiring urgent clinical intervention. The widespread adoption of electronic health records (EHR) makes rich feature patient data sets more readily available. These large data stores lend themselves to use in modern machine learning (ML) models. This paper investigates the use of transformer-based models to identify critical deterioration in unplanned ED admissions, using free-text fields, such as triage notes, and tabular data, including early warning scores (EWS).

Design A retrospective ML study.

Setting A large ED in a UK university teaching hospital.

Methods We extracted rich feature sets of routine clinical data from the EHR and systematically measured the performance of tree- and transformer-based models for predicting patient mortality or admission to critical care within 24 hours of presentation to ED. We compared our proposed models to the National EWS (NEWS).

Results Models were trained on 174 393 admission records. We found that models including free-text triage notes outperform structured tabular data models, achieving an average precision of 0.92, compared with 0.75 for tree-based models and 0.12 for NEWS.

Conclusions Our findings suggests that machine learning models using free-text data have the potential to improve clinical decision-making in the ED; our techniques significantly reduce alert rate while detecting most high-risk patients missed by NEWS.

Citation

Watson, M., Boulitsakis Logothetis, S., Green, D., Holland, M., Chambers, P., & Al Moubayed, N. (2024). Performance of machine learning versus the national early warning score for predicting patient deterioration risk: a single-site study of emergency admissions. BMJ Health & Care Informatics, 31(1), Article e101088. https://doi.org/10.1136/bmjhci-2024-101088

• Journal Article Type: Article
• Acceptance Date: Nov 29, 2024
• Online Publication Date: Dec 4, 2024
• Publication Date: 2024-12
• Deposit Date: Dec 10, 2024
• Publicly Available Date: Dec 10, 2024
• Journal: BMJ Health & Care Informatics
• Electronic ISSN: 2632-1009
• Publisher: BMJ Publishing Group
• Peer Reviewed: Peer Reviewed
• Volume: 31
• Issue: 1
• Article Number: e101088
• DOI: https://doi.org/10.1136/bmjhci-2024-101088
• Public URL: https://durham-repository.worktribe.com/output/3213197