Performance of ChatGPT-5 under a minimal binary prompt for detection of distal radius fractures on plain radiographs.
Authors
Affiliations (3)
Affiliations (3)
- Technion - Israel Institute of Technology, The Ruth and Bruce Rappaport Faculty of Medicine, Haifa, 3109601, Israel. [email protected].
- Rambam Health Care Campus, Department of Orthopedic Surgery, Haifa, 3109601, Israel. [email protected].
- Rambam Health Care Campus, Department of Orthopedic Surgery, Haifa, 3109601, Israel.
Abstract
General-purpose multimodal large language models (LLMs) are being applied to radiology, but their performance under the simplest realistic clinical query (a single binary fracture-or-not prompt) has not been rigorously quantified. We evaluated ChatGPT-5 on this task using a public distal radius dataset. In this retrospective diagnostic accuracy study, 396 anteroposterior distal radius radiographs from the Stanford MURA dataset were submitted to ChatGPT-5, each in a fresh temporary chat session. The prompt was: "Is there a fracture in this X-ray? Answer with yes or no." The reference standard was an image-level label assigned by consensus between a senior orthopedic resident and a board-certified orthopedic surgeon; MURA study-level labels were used only to select candidate images. Twenty-six radiographs (6.6%) returned a hedged response and were excluded from the primary analysis, leaving 370; all 26 were added back in three sensitivity analyses. Outcomes were sensitivity, specificity, PPV, NPV, F1, balanced accuracy, and Cohen's kappa, with 95% Wilson confidence intervals and McNemar's test for error asymmetry. Fracture prevalence in the analyzed sample was 45.9% (170/370). Sensitivity was 51.2% (95% CI 43.7-58.6), specificity 70.5% (63.8-76.4), accuracy 61.6% (56.6-66.4), PPV 59.6% (51.5-67.2), NPV 62.9% (56.5-69.0), F1 0.551, Cohen's kappa 0.219 (0.112-0.318). False negatives exceeded false positives (McNemar χ<sup>2</sup> = 4.06, p = 0.044). Non-binary responses occurred in 7.6% of fracture-positive and 5.7% of fracture-negative radiographs (p = 0.54); counting all as errors lowered sensitivity to 47.3% (40.2-54.5), and no assumption raised it above 54.9%. Projected to 20% prevalence, PPV was 30.3% and NPV 85.2%. Under a minimal binary prompt and without task-specific fine-tuning, ChatGPT-5 missed approximately half of distal radius fractures and fell well below published performance for dedicated CNN-based fracture detectors. Results are specific to this model version, this prompt, this anatomical region, and this dataset; they constrain claims about autonomous, minimal-prompt use rather than the broader potential of engineered LLM-based radiological workflows.