Leveraging large language models for information extraction from free-text liver MRI reports and assessment of clinical utility: a multicenter study.

Authors

Abstract

Large language models (LLMs) have shown considerable potential for extracting information from free-text radiology reports, enabling efficient data use, large-scale data mining, and a wide range of secondary analyses and clinical applications. This study aimed to evaluate the performance of LLMs in extracting diagnostically relevant information from multicenter free-text liver magnetic resonance imaging (MRI) reports, explore the clinical utility of LLM-generated structured reports, and investigate optimal prompting strategies for multicenter data. In this retrospective multicenter study, 800 free-text liver MRI reports from four medical centers (Beijing Friendship Hospital, Tianjin Medical University General Hospital, The Second Affiliated Hospital of Xi'an Jiaotong University, Sir Run Run Shaw Hospital) were collected to evaluate the information extraction performance of two LLMs-DeepSeek-V3 and ChatGPT-4o-using radiologist-annotated structured data as the reference standard. Three prompting strategies were applied: zero-shot prompting, global few-shot prompting (using shared examples across centers), and center-specific few-shot prompting (using examples specific to each center), with example counts set to 2-12. Model performance was evaluated using field-level F1 scores, and report-level extraction success was defined as the correct extraction of ≥80% report fields. Additionally, an exploratory clinical evaluation was conducted using 20 reports from one center, in which 10 radiologists and 10 clinicians evaluated the readability and clinical usability of free-text, manually structured, and LLM-generated reports on a 5-point Likert scale. Few-shot prompting significantly outperformed zero-shot prompting for both LLMs, with the largest gains in macro F1 observed when k was increased from 0 to 2 (∆DeepSeek-V3: global 0.106, center-specific 0.127; ∆ ChatGPT-4o: global 0.086, center-specific 0.107). Performance plateaued at k=4 [DeepSeek-V3: global 0.848 (0.838-0.858), center-specific 0.865 (0.856-0.875); ChatGPT-4o: global 0.835 (0.824-0.845), center-specific 0.861 (0.851-0.870)], with adjacent-k gains <0.01. Center-specific prompting consistently outperformed global prompting (∆F1: 0.017-0.024 for DeepSeek-V3; 0.014-0.026 for ChatGPT-4o). In the exploratory clinical evaluation, structured reports received higher scores for clarity and communication than free-text reports (both P<0.001), while LLM-generated reports received scores comparable to those of manually structured reports (both P>0.05). LLMs demonstrated strong performance in extracting diagnostically relevant information from Chinese multicenter liver MRI reports. In the exploratory clinical evaluation, the LLM-generated structured reports showed the potential to improve report clarity and facilitate clinical communication. Global prompting showed good performance across centers, while center-specific prompting further improved accuracy by adapting to local reporting styles.

Tags

Topics