Category and Modifier Accuracy in a Five-Model Evaluation of Large Language Model Extraction of CAD-RADS 2.0 from Coronary CT Angiography Reports.
Authors
Affiliations (3)
Affiliations (3)
- Department of Radiology, UC Davis Health, Sacramento, CA, USA. [email protected].
- School of Medicine, UC Davis Health, Sacramento, CA, USA.
- Department of Radiology, UC Davis Health, Sacramento, CA, USA.
Abstract
This study evaluates large language model (LLM) extraction of Coronary Artery Disease Reporting and Data System (CAD-RADS) categories from coronary CT angiography (CCTA) reports using a guideline-derived specification and examines instruction-related improvement, repeatability, and the limitations of extracting plaque and ischemia modifiers alongside the category. Five LLM platforms (ChatGPT, Claude, Gemini, Grok, and Copilot) processed 400 deidentified CCTA report entries under unprompted, specified, and repeated-specified conditions. The primary outcome was exact category agreement with a report-derived consensus reference under the specification (n = 400). Secondary outcomes included three-component agreement (n = 396), modifier accuracy, instruction-related changes, and repeatability. Inference used report-text cluster bootstrap and paired or permutation tests with Holm adjustment. For the primary outcome, category accuracy under the specification was 92.8-95.5% (371-382/400); secondary outcomes included run-to-run category agreement of 97.5-98.8%, whereas three-component accuracy was 59.3-81.6% (235-323/396). Among assigned references, plaque accuracy was 79.2-94.0% (236-280/298) and ischemia accuracy 45.1-90.3% (79-158/175). The specification improved category accuracy in four of five models and three-component, plaque, and ischemia accuracy in all five. Models did not differ significantly in category accuracy (Cochran Q = 8.73; adjusted p = 0.077) but differed in three-component accuracy (Q = 116.48; adjusted p < 0.001). Category extraction was accurate under the specification, with improvement in most models and high repeatability. Joint evaluation of category, plaque, and ischemia revealed limitations in modifier extraction. External and prospective validation is needed to develop a reliable clinical tool and establish its value in routine practice.