Enhancing Radiologists' Diagnostic Performance for Primary Bone Tumors Using Structured Report-Driven Assistance From Multiple Large Language Models.
Authors
Affiliations (4)
Affiliations (4)
- Department of Radiology, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea.
- Visual Analysis and Learning for Improved Diagnostics (VALID) Lab, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea. [email protected].
- Department of Pathology, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea.
- Department of Orthopedic Surgery, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea.
Abstract
This study aimed to determine whether artificial intelligence (AI) assistance through structured report-driven diagnostic suggestions ensembled from multiple large language models (LLMs) can improve radiologists' diagnostic accuracy for primary bone tumors. We retrospectively included the data of 232 patients (15 for prompt development and 217 for testing) with pathologically confirmed primary bone tumors (January 2019-January 2026) who underwent MRI and either radiography or CT. Three radiologists with different levels of expertise generated structured radiology reports and established diagnoses with and without LLM assistance. The structured radiology reports were input into an output-ensemble pipeline of four LLMs (GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.6, and Llama 4 Maverick), generating ranked differential diagnoses aggregated across models and model-specific justifications, which radiologists reviewed to establish LLM-assisted diagnoses. Generalized linear mixed-effects models were used to evaluate the LLM-assistance effect on the Top-1 and Top-3 diagnostic accuracies. The readers' responses to correct and incorrect multi-LLM diagnostic suggestions (acceptance or rejection) were analyzed descriptively. Across the three readers, LLM assistance significantly improved the overall Top-1 and Top-3 accuracies from 57.1% to 72.0% and from 81.1% to 90.9%, respectively (both adjusted <i>P</i> < 0.001). Top-1 accuracy increased by 6.5, 14.3, and 24.0 percentage points for the most- to least-experienced readers, respectively, while Top-3 accuracy increased by 4.2, 7.4, and 18.0 percentage points, respectively. Reader responses to multi-LLM suggestions differed by experience: less-experienced readers accepted incorrect suggestions more frequently, whereas the most-experienced reader rejected incorrect suggestions more frequently but also rejected correct suggestions more often. The multi-LLM ensemble showed a standalone Top-1 accuracy ranging from 56.7% to 62.2% and Top-3 accuracy ranging from 85.3% to 86.6%. Structured report-driven multi-LLM assistance improves radiologists' performance for diagnosing primary bone tumors. The readers' responses to AI suggestions varied according to their experience levels. Although numerically greater increments in the diagnostic accuracy were observed with decreasing reader experience, this observation requires further validation using larger reader samples.