Comparative Evaluation of Open-Weight Large Language Models for Pathological Brain MRI Reports Classification: From Prompts to a Fine-Tuned Model.
Authors
Affiliations (7)
Affiliations (7)
- Laboratory of Neuroinformatics, IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli, Brescia, Italy (C.D., C.C., D.A., S.D.F., C.M.B., A.Bo., A.R.).
- IRCCS SYNLAB SDN, Naples, Italy (M.V., V.B., V.A., L.B., M.A.).
- Department of Advanced Biomedical Sciences, University of Naples "Federico II", Napoli, Italy (G.P., A.Br.).
- MAC-Memory Clinic and Molecular Markers Laboratory, IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli, Brescia, Italy (S.F.).
- Laboratory of Alzheimer's Neuroimaging & Epidemiology, IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli, Brescia, Italy (M.P.).
- Medical Physics Section - Department of Biomedicine and Prevention, University of Rome "Tor Vergata", Rome, Italy (N.T.); A.A. Martinos Center for Biomedical Imaging - Harvard Medical School/MGH, Boston, Massachusetts (N.T.).
- Laboratory of Neuroinformatics, IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli, Brescia, Italy (C.D., C.C., D.A., S.D.F., C.M.B., A.Bo., A.R.). Electronic address: [email protected].
Abstract
Automated semantic classification of radiology reports can enhance clinical workflows and research, but often relies on language- and task-specific models, limiting scalability. This study aims to evaluate and improve the 0-shot baseline performance of open-weight large language models (LLMs) in the semantic classification of free-text magnetic resonance imaging (MRI) reports. A corpus of 1385 reports from a single neuroradiology center was used to assess four LLMs (GPT-OSS-20B, Magistral-Small-2509, Gemma-4-31B, Qwen3.5-35B-A3B) of comparable scale (∼20-35B parameters). The brain MRI reports were labeled as Pathological or Non-Pathological, with the Pathological ones further classified into five diagnostic categories (Neoformation, Vascular, Inflammation/Infection, Neurodegenerative, Other). The LLMs were evaluated using zero- and few-shot prompting with majority-vote aggregation. Quantized Low-Rank Adaptation (QLoRA) fine-tuning was then applied to the best-performing model across multiple configurations. Performance was assessed using Macro F1, Micro F1, Precision, Recall, and Exact Match Accuracy. The binary classification task achieved a Macro F1 score of 0.84 on the best-performing Qwen3.5-35B-A3B model, while multilabel classification reached 0.81. Majority voting (MV) outperformed individual-set predictions. QLoRA fine-tuning yielded a Macro F1 score of 0.83 on the multilabel task; the improvement over the 0-shot baseline was statistically supported for Macro F1, Micro F1, Macro Precision, and Exact Match, whose bootstrap 95% confidence intervals excluded zero, but not for Macro Recall. Fine-tuning and few-shot prompting strategies are complementary, with neither consistently outperforming the other. Error analysis showed that the models struggled with borderline annotation cases and often missed secondary labels. Open-weight LLMs demonstrate strong capability for neuroradiology report classification without extensive task-specific training. Fine-tuning further improves performance, supporting integration into scalable semantic radiology data pipelines. However, the present evidence comes from a single Italian center without external validation, and the fine-tuning gain was estimated on a small held-out test set, for which the Macro F1 improvement, though statistically significant, has a confidence interval with a lower bound close to zero. In this light, the approach is best positioned as an internal indexing and semantic-retrieval tool, supporting faster report retrieval, structured cohort identification, internal audit, and scalable phenotyping for neuroradiology research and data warehousing, as well as efficient active-learning strategies, with broader generalizability requiring additional external multicentre validation before clinical deployment.