Evaluating large language models for structuring cardiology reports: a real-world clinical study on patient subtyping and trial recruitment

F. Pescol, T. Buonocore, V. Tibollo, G. Failla, E. Traversi, Maria Teresa La Rovere, L. Sacchi, Walter Ricciardi, Riccardo Bellazzi

TL;DR: This study presents a notable step towards the integration of AI-driven approaches into real-world clinical practice for patient recruitment in CTs, highlighting both its practicality and effectiveness in meeting the stringent demands of healthcare settings.

BACKGROUND Artificial Intelligence (AI) methods have emerged as useful tools for supporting patient recruitment in clinical trials (CTs). Despite several studies having recently proposed promising applications of Large Language Model (LLMs) for patient recruitment in CTs, their implementation in routine clinical practice remains limited. METHODS In this study, we present a comprehensive pipeline, developed and tested in a real-world clinical setting, to obtain highly detailed patient subtyping and eligibility assessment for specific CTs. Our solution leverages cardiological discharge letters, a rich yet underutilized source of patient data, to extract detailed structured clinical information through LLMs. Patient subtyping and eligibility assessment are performed through a rule-based approach, based on the extracted information, to maximize deterministic and interpretable outputs. We employed OpenAI's GPT-4.1 within the cloud-based service Microsoft Azure Machine Learning Studio, deployed in the hospital infrastructure. Validation was conducted on a sample of 100 discharge letters through exact-match comparison between the model's output and a ground-truth template, pre-populated by expert clinicians. RESULTS Our results confirm the feasibility and effectiveness of the proposed approach in real-world clinical scenarios. GPT-4.1 achieved high values of information extraction accuracy for most clinical variables (0.94 ± 0.08), resulting in a limited number of false negatives (FN) and false positives (FP) in both patient subtyping (0.12 and 0.13, respectively) and eligibility assessment. At the criterion-level, the proportion of FNs and FPs was below 3% for most criteria (13 and 11 of the 14 criteria examined, respectively). CONCLUSION Overall, our study presents a notable step towards the integration of AI-driven approaches into real-world clinical practice for patient recruitment in CTs, highlighting both its practicality and effectiveness in meeting the stringent demands of healthcare settings.