Benchmarking large language model performance in generating and assessing radiology objective structured clinical examination.
Authors
Affiliations (6)
Affiliations (6)
- Department of Radiology, Wake Forest University School of Medicine, Winston-Salem, NC 27157, United States.
- Department of Periodontics, Virginia Commonwealth University School of Dentistry, Richmond, VA 23298, United States.
- Department of Biostatistics and Data Science, Wake Forest University School of Medicine, Winston-Salem, NC 27157, United States.
- Department of Radiology and Interventional Radiology, Altnagelvin Hospital, Londonderry, Northern Ireland BT47 6SB, United Kingdom.
- Department of Radiology, UT Southwestern Medical Center, Dallas, TX 75390, United States.
- Department of Radiodiagnosis, Ram Krishna Medical College Hospital and Research Centre, Bhopal, MP 462042, India.
Abstract
High-quality radiology assessment questions are essential for education competency evaluation but labor-intensive to create. To compare four large language models (LLMs) in generating and evaluating radiology objective structured clinical examination (OSCE)-style questions and responses. Fifty Radiopaedia cases across 10 subspecialties were used to generate 200 radiology OSCE-style sessions (3 questions per session) using 4 LLMs: GPT-4o (4O), Llama 3-70b (LM), Claude 3.5 Sonnet (CL), and Gemini 1.5 Flash (GM). Three expert radiologists blindly evaluated sessions for clarity, clinical relevance, option accuracy, assessment accuracy, and feedback quality on 5-point Likert scales, with difficulty rated on a 3-point categorical scale. Generalized estimating equations compared model performance, accounting for rater correlation. The primary accuracy metric was Top Two Box Accuracy (all raters ≥4); secondary metrics included Top Box Accuracy, Average Score ≥4 Accuracy, and Perfect 5 Accuracy (all raters = 5). Final rankings were determined via Borda count. Inter-rater reliability ranged from fair to good agreement (Gwet's AC2: 0.39-0.73 across the 5 retained metrics). Generalized estimating equations analysis showed significant performance variations (<i>P</i> < .05) across metrics. Final Borda scores: 4O (18.5), LM (14.5), CL (12.0), GM (5.0). Top Box Accuracy was high (70.7%-100%). For the primary metric Top Two Box Accuracy, 4O showed significantly higher odds of achieving high-quality scores vs. GM for clarity (odds ratio [OR] 4.52, <i>P</i> < .001) and overall assessment accuracy (OR 3.78, <i>P</i> < .001). Under the strictest threshold (Perfect 5 Accuracy), 4O led in composite Question General Accuracy (52.7% vs. 32.7% LM, 21.3% CL, 1.3% GM). Qualitative failure modes included distractor ambiguity, conceptual repetition, and information leakage. This study provides a benchmark of LLM performance for radiology OSCE-style content generation and evaluation during a specific snapshot of artificial intelligence development (August 2024). 4O demonstrated the highest relative performance, though none consistently achieved expert-level quality. Although LLMs can augment radiology education, expert review remains mandatory for high-stakes assessment.