On-Premises Open-Weight Large Language Models for Classifying Epilepsy Brain MRI Reports: A Retrospective Study.
Authors
Affiliations (4)
Affiliations (4)
- Department of Neurology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-Ro 43-Gil , Songpa-Gu, Seoul, 05505, Republic of Korea.
- Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-Ro 43-Gil , Songpa-Gu, Seoul, 05505, Republic of Korea.
- Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-Ro 43-Gil , Songpa-Gu, Seoul, 05505, Republic of Korea. [email protected].
- Department of Neurology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-Ro 43-Gil , Songpa-Gu, Seoul, 05505, Republic of Korea. [email protected].
Abstract
Free-text reports limit the use of imaging data for research cohort screening. This study evaluated whether locally deployed open-weight large language models (LLMs) can classify brain magnetic resonance imaging (MRI) reports in epilepsy without external data transfer, and which forms of domain knowledge improve classification. In this single-center retrospective study, 1210 epilepsy-protocol brain MRI reports were labeled by a board-certified neuroradiologist for abnormality status and primary imaging finding (eight classes). A held-out test set of 206 reports was classified by a biomedical (OpenBioLLM-8B) and a general-purpose (Qwen3.5-9B) LLM on a single 24-GB graphics processing unit, with and without a rule set and example retrieval, and compared with a fine-tuned radiology-specific BERT model, an independent neuroradiology fellow, and a same-base control model. The primary outcome was the macro-averaged F1 score for the primary finding. Without task-specific training, the general-purpose model achieved a finding macro-F1 of 0.63 (95% confidence interval (CI), 0.53-0.70), comparable to the fine-tuned BERT model (0.59; difference, - 0.04; 95% CI, - 0.13 to 0.07) and to inter-reader agreement (0.61; κ = 0.59). A rule set stating institutional category definitions improved the biomedical model (difference, 0.23; 95% CI, 0.11-0.33) without significantly reducing performance in any model, whereas biomedical fine-tuning conferred no advantage over the control model and example retrieval did not help. Residual errors concentrated where the readers disagreed. Locally deployed LLMs performed comparably to inter-reader agreement, which may support human-verified cohort screening; in this institution-specific task, explicit category definitions were more useful than biomedical specialization.