Do S-Detect and ChatGPT Improve Diagnostic Performance and Agreement with BI-RADS Morphologic Descriptors Among Early-career Radiologists?
Authors
Affiliations (6)
Affiliations (6)
- Department of Radiology, Izmir Katip Celebi University Ataturk Training and Research Hospital, Izmir, Turkiye (S.U., O.E., S.K., B.S., M.N.Y.). Electronic address: [email protected].
- Department of Radiology, Acıbadem Izmir Kent Hospital, Izmir, Turkiye (M.G.B.).
- Department of Radiology, Izmir Katip Celebi University Ataturk Training and Research Hospital, Izmir, Turkiye (S.U., O.E., S.K., B.S., M.N.Y.).
- Department of Radiology, Kanuni Sultan Suleyman Training and Research Hospital, Istanbul, Turkiye (K.A.S.).
- Department of Information Technologies, Database, Network Design and Management, Isparta University of Applied Science, Isparta, Turkiye (R.G.).
- Department of Radiology, University of Health Sciences Istanbul Haseki Training and Research Hospital, Istanbul, Turkiye (T.I.).
Abstract
We investigated whether S-Detect and ChatGPT assistance improve early-career radiologists' diagnostic performance in breast ultrasound and their agreement with a senior breast radiologist (SBR) for Breast Imaging Reporting and Data System v5.0 morphologic descriptors. Ultrasound images of 299 histopathologically confirmed lesions from 268 patients were retrospectively evaluated by the SBR, one junior attending radiologist, three residents, S-Detect, and ChatGPT (single image-only zero-shot inference via the web interface). These four early-career radiologists reassessed lesions with S-Detect and ChatGPT assistance. Histopathology was the reference standard for diagnostic performance, and the SBR for descriptor agreement. Kappa differences were compared using patient-level cluster bootstrapping. The SBR achieved the highest sensitivity and accuracy, whereas S-Detect achieved the highest specificity. Neither S-Detect nor ChatGPT assistance consistently improved diagnostic performance. S-Detect had the highest standalone agreement for shape, margin, posterior acoustic features, and echo pattern, and S-Detect assistance significantly improved agreement for shape, posterior acoustic features, and echo pattern in selected readers. Histopathologic agreement was highest for the SBR (κ = 0.713; 95% confidence interval [CI], 0.637-0.789), followed by S-Detect (κ = 0.686; 95% CI, 0.604-0.768), without significant difference (difference in kappa, -0.027; 95% CI, -0.128-0.067; Benjamini-Hochberg-adjusted q = 0.877); however, the wide CI could not exclude a clinically meaningful difference. ChatGPT had the lowest agreement (κ = 0.373; 95% CI, 0.271-0.474). In this biopsy-confirmed cohort, S-Detect showed high specificity and strong descriptor agreement but did not consistently enhance early-career radiologists' diagnostic performance. ChatGPT performance was limited under image-only, single-inference zero-shot conditions and should not be generalized to its overall capability.