Evaluating ChatGPT text mining of clinical records for companion animal obesity monitoring

Fins, Ivo S.; Davies, Heather; Farrell, Sean; Torres, Jose R.; Pinchbeck, Gina; Radford, Alan D.; Noble, Peter‐John

Abstract

Background: Veterinary clinical narratives remain a largely untapped resource for addressing complex diseases. Here we compare the ability of a large language model (ChatGPT) and a previously developed regular expression (RegexT) to identify overweight body condition scores (BCS) in veterinary narratives pertaining to companion animals. Methods: BCS values were extracted from 4415 anonymised clinical narratives using either RegexT or by appending the narrative to a prompt sent to ChatGPT, prompting the model to return the BCS information. Data were manually reviewed for comparison. Results: The precision of RegexT was higher (100%, 95% confidence interval [CI] 94.81%–100%) than that of ChatGPT (89.3%, 95% CI 82.75%–93.64%). However, the recall of ChatGPT (100%, 95% CI 96.18%–100%) was considerably higher than that of RegexT (72.6%, 95% CI 63.92%–79.94%). Limitations: Prior anonymisation and subtle prompt engineering are needed to improve ChatGPT output. Conclusions: Large language models create diverse opportunities and, while complex, present an intuitive interface to information. However, they require careful implementation to avoid unpredictable errors.

Citation

Fins, I. S., Davies, H., Farrell, S., Torres, J. R., Pinchbeck, G., Radford, A. D., & Noble, P. (2024). Evaluating ChatGPT text mining of clinical records for companion animal obesity monitoring. Veterinary Record, 194(3), Article e3669. https://doi.org/10.1002/vetr.3669

• Journal Article Type: Article
• Acceptance Date: Nov 7, 2023
• Online Publication Date: Dec 6, 2023
• Publication Date: Feb 2, 2024
• Deposit Date: Dec 20, 2023
• Publicly Available Date: Dec 20, 2023
• Journal: Veterinary Record
• Print ISSN: 0042-4900
• Electronic ISSN: 2042-7670
• Publisher: Wiley
• Peer Reviewed: Peer Reviewed
• Volume: 194
• Issue: 3
• Article Number: e3669
• DOI: https://doi.org/10.1002/vetr.3669
• Public URL: https://durham-repository.worktribe.com/output/1985207