Diagnostic Accuracy of Large Language Models for Pediatric Chest Radiograph: Comparison With Pediatricians and Radiologists.
Authors
Affiliations (3)
Affiliations (3)
- Department of Pediatrics, Segovia General Hospital, Segovia, Spain.
- Department of Medicine, Francisco de Vitoria University, Pozuelo de Alarcón (Madrid), Spain.
- Department of Radiology, Segovia General Hospital, Segovia, Spain.
Abstract
Chest radiography (CR) is the most widely used radiological test in pediatrics, yet its interpretation remains heterogeneous. Although large language models (LLMs) such as ChatGPT and Perplexity are not designed for medical use, their widespread use justifies evaluating their performance in healthcare. We compared ChatGPT and Perplexity with radiologists and pediatricians, assessing both their screening performance between pathological or non-pathological CR and diagnostic accuracy for specific pathologies. We conducted a cross-sectional, observational and analytical study including all CRs obtained from pediatric patients between November 2023 and April 2024 at a secondary hospital. The screening performance and the diagnostic accuracy of the observers (ChatGPT, Perplexity and pediatricians) compared to the reference standard (a consensus of three radiologists) were evaluated using Phi coefficient and Cohen's Kappa. The study included 502 radiographs from 444 patients. Pediatricians showed better sensitivity, specificity and positive predictive value than ChatGPT and Perplexity, whereas Perplexity obtained a higher negative predictive value. ChatGPT demonstrated higher sensitivity for pneumonia than for the overall sample (0.81 vs. 0.67). Inter-observer agreement was highest between pediatricians and radiologists (Phi = 0.80; Kappa = 0.69). ChatGPT showed higher concordance with radiologists than Perplexity did (Phi = 0.56; Kappa = 0.34 vs. Phi = 0.33; Kappa = 0.32, respectively). Inter-observer agreement was lower for pneumonia cases and within the neonatal population. General-purpose LLMs, both ChatGPT and Perplexity provide fair screening capability for pediatric CR, although their diagnostic accuracy drops significantly for specific conditions. The findings suggest that LLMs cannot replace specialists and highlight the need to develop pediatric-specific AI models.