ChatGPT-5 missed about half of distal radius fractures on x-rays, showing limited diagnostic capability in a real-world scenario.
Key Details
- 1Study evaluated ChatGPT-5 using the Stanford MURA dataset: 370 anteroposterior wrist x-rays with 45.9% fracture prevalence.
- 2ChatGPT-5 was given a simple yes/no prompt to detect fractures via its public web interface (zero-shot, no additional context).
- 3The model had 51.2% sensitivity, 70.5% specificity, and 61.6% overall accuracy for fracture detection.
- 4ChatGPT-5 missed 83 fractures and had more false negatives than false positives (McNemar χ²=4.06, p=0.044).
- 5Authors caution that general-purpose LLMs should not be used as standalone diagnostic tools for fracture detection.
Why It Matters
This study highlights the limitations of using general-purpose LLMs like ChatGPT-5 for medical imaging diagnosis, underscoring the need for model adaptation, specific prompt engineering, and rigorous radiologist benchmarking before clinical use in radiology workflows.
Source
AuntMinnie
Related News
•AuntMinnie
AI Triage Tool Speeds Up ICH Head CT Turnaround in ED
AI-based triage for intracranial hemorrhage on head CT reduced emergency department report turnaround by 13 minutes.
•AuntMinnie
In-House LLM Accurately Flags Critical Pediatric Radiology Findings
A locally deployed LLM system at Stanford Health achieved 99.7% sensitivity for detecting critical results in pediatric radiology reports, supporting workflow automation.
•Radiology Business
New AI Flags Uncertainty in Breast Cancer Imaging Predictions
University of Pittsburgh researchers developed an AI model that quantifies uncertainty in breast cancer diagnostic predictions to enhance trustworthiness.