Diagnostic Accuracy and Fine-Grained Image-Interpretation Performance of Multimodal Large Language Models on Bilingual Radiology Questions: Cross-Sectional Benchmark Study.

Authors

Abstract

Multimodal large language models (LLMs) are rapidly expanding in radiology and digital health; however, their diagnostic performance across diverse clinical scenarios and fine-grained visual reasoning tasks requires systematic evaluation. The study aims to evaluate the diagnostic accuracy and fine-grained visual reasoning performance of 5 multimodal LLMs with bilingual medical imaging questions. This cross-sectional benchmark study evaluated bilingual medical imaging questions using zero-shot prompting on 5 LLMs: GPT-5.2, Gemini 3 Pro, Doubao, Yuanbao, and Llama 4 Maverick. Subgroup analyses by language (English/Chinese), question type (image/text), imaging modality (computed tomography, radiography, magnetic resonance imaging, ultrasound, and nuclear medicine), and anatomical system (neurologic, respiratory, cardiovascular, digestive, genitourinary, and musculoskeletal) evaluated intermodel accuracy and intramodel consistency. Image-based visual reasoning performance was assessed across modality, anatomy, lesion detection, and localization. A total of 298 questions (198 image-based, 100 text-based) were evaluated. Gemini 3 Pro achieved the highest overall accuracy (n=268, 89.9%, 95% CI 86.0%-92.9%), significantly outperforming Doubao (n=235, 78.9%; <i>P</i><.001), GPT-5.2 (n=233, 78.2%; <i>P</i><.001), Yuanbao (n=216, 72.5%; <i>P</i><.001), and Llama 4 Maverick (n=216, 72.5%; <i>P</i><.001). All models performed better on text-based than image-based questions (Yuanbao, <i>P</i><.001; Doubao, <i>P</i>=.02; GPT-5.2, <i>P</i><.001; Gemini 3 Pro, <i>P</i>=.002; Llama 4 Maverick, <i>P</i><.001). GPT-5.2 (<i>P</i><.001), Llama 4 Maverick (<i>P</i>=.03), and Yuanbao (<i>P</i>=.03) favored English questions, while intramodel performance remained stable across modalities (Yuanbao, <i>P</i>=.46; Doubao, <i>P</i>=.21; GPT-5.2, <i>P</i>=.26; Gemini 3 Pro, <i>P</i>=.96; Llama 4 Maverick, <i>P</i>=.45) and anatomical systems (Yuanbao, <i>P</i>=.82; Doubao, <i>P</i>=.52; GPT-5.2, <i>P</i>=.35; Gemini 3 Pro, <i>P</i>=.13; Llama 4 Maverick, <i>P</i>=.97). Despite near-perfect macroscopic visual recognition (modality and anatomy: 99.0%-100.0%), accuracy dropped in complex tasks. Gemini 3 Pro outperformed Yuanbao (<i>P</i><.001), GPT-5.2 (<i>P</i>=.002), and Llama 4 Maverick (<i>P</i><.001) in lesion detection (175/198, 88.4%) and outperformed Yuanbao (<i>P</i><.001), Doubao (<i>P</i>=.004), and Llama 4 Maverick (<i>P</i>=.004) in lesion localization (189/198, 95.5%). Gemini 3 Pro demonstrated optimal, comprehensive performance and cross-lingual stability. However, current multimodal LLMs face a technical bottleneck in complex visual interpretation-particularly in fine-grained lesion detection and localization-emphasizing the need to prioritize local feature perception in future model development.

Tags

Topics