Agreement of Multimodal Large Language Models and Novice Readers With Expert-Based Radiographic Scoring of Presumptive Septic Carpal Arthritis in Calves.
Authors
Affiliations (6)
Affiliations (6)
- Department of Surgery, Faculty of Veterinary Medicine, Atatürk University, Erzurum, Turkey.
- Department of Surgery, Faculty of Veterinary Medicine, Aydın Adnan Menderes University, Aydın, Turkey.
- Department of Surgery, Faculty of Veterinary Medicine, Necmettin Erbakan University, Konya, Turkey.
- Department of Surgery, Faculty of Veterinary Medicine, Cukurova University, Adana, Turkey.
- Department of Veterinary Surgery, Faculty of Veterinary Medicine, Yozgat Bozok University, Yozgat, Turkey.
- Department of Surgery, Faculty of Veterinary Medicine, Mehmet Akif Ersoy University, Burdur, Turkey.
Abstract
Septic arthritis is an important cause of morbidity in calves, and radiographic interpretation may vary between observers, especially when structured scoring systems are used by less experienced readers. This single-center retrospective reader study evaluated whether general-purpose multimodal large language models (LLMs) could support rubric-based radiographic scoring of presumptive septic carpal arthritis in calves. Fifty calves aged 0-3 months with clinical findings consistent with presumptive septic carpal arthritis were included, and one carpal radiograph per case was assessed. Two novice veterinary surgeons and three multimodal LLMs, ChatGPT-5, Gemini-2.5 Pro, and Claude Sonnet-4, independently scored each case using a Constant-based ordinal radiographic scoring framework. Expert consensus served as the operational reference standard. Agreement with the reference was assessed using exact agreement, agreement within one score category, and quadratic weighted kappa. Novice 1 showed the highest concordance, with a mean exact agreement of 55.6%, mean ±1 agreement of 89.8%, and substantial agreement (mean κ<sub>w</sub> = 0.68; 95% CI, 0.56-0.80). ChatGPT-5 was the best-performing model, achieving moderate agreement (mean exact agreement, 53.0%; mean ±1 agreement, 82.6%; mean κ<sub>w</sub> = 0.58; 95% CI, 0.41-0.72). Claude Sonnet-4 and Gemini-2.5 Pro showed slight agreement overall. The best-performing novice reader remained more reliable than the evaluated models, although ChatGPT-5 performed comparably to, or better than, one novice evaluator in selected parameters. Selected LLMs may have limited adjunctive value for structured review or training, but not as replacements for human interpretation.