Comparative performance of three large language models and expert radiologists in ACR TI-RADS classification of structured thyroid ultrasound reports: a retrospective analysis using cytology as reference.

Authors

Abstract

Large language models (LLMs) are increasingly explored for medical applications. This study evaluates the agreement of three LLMs-ChatGPT, Copilot, and Gemini-with two expert radiologists in assigning ACR TI-RADS categories to thyroid nodules, using fine-needle aspiration cytology as the reference standard. A retrospective analysis included 135 thyroid nodules from 135 unique patients (108 benign, 27 malignant). Structured TI-RADS feature scores and final categories (TR1-TR5) were extracted from existing reports generated by each LLM and two board-certified radiologists (≥8 years' experience). Agreement was assessed using Cohen's κ (feature-level), quadratic weighted κ (category-level), and diagnostic performance (sensitivity, specificity, area under the ROC curve [AUC]) with TR4-TR5 classified as test-positive categories for the binary diagnostic analysis. McNemar's test evaluated discordant classification patterns. Feature-level agreement was fair to substantial for composition and margin (κ up to 0.621) but poor for echogenic foci (κ as low as 0.075). Weighted κ for TI RADS categories was moderate to substantial for ChatGPT (κw 0.614-0.621) and Gemini (0.516-0.621), but fair for Copilot (0.383-0.314). All LLMs showed high sensitivity (81.5-96.3 %) but low specificity (20.4-38.0 %). Ordinal AUC values were 0.757 (ChatGPT), 0.644 (Copilot), and 0.619 (Gemini). McNemar's test suggested a more balanced discordant pattern for ChatGPT than for Copilot and Gemini. ChatGPT, Copilot, and Gemini show moderate composition/margin agreement but poor echogenic foci detection and overclassify benign nodules; standalone use is not recommended. Domain‑specific fine‑tuning and prospective validation are required.

Tags

Topics