Back to all papers

Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset.

November 20, 2026pubmed logopapers

Authors

Yamagishi Y,Takamatsu A,Sato M,Kikuchi T,Hanaoka S,Yoshikawa T,Abe O

Affiliations (6)

  • Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan. [email protected].
  • Department of Radiology, The University of Tokyo Hospital, Tokyo, Japan.
  • Department of Radiology, Kanazawa University Hospital, Kanazawa, Japan.
  • Faculty of Medicine, The University of Tokyo, Tokyo, Japan.
  • Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo, Japan.
  • Department of Radiology, School of Medicine, Jichi Medical University, Shimotsuke, Japan.

Abstract

To evaluate whether large language model (LLM)-assisted label cleaning can identify label-report discordance in CT-RATE, a large-scale public chest CT dataset. After report-level deduplication, 24,446 unique radiology reports from 20,209 unique patients were identified. Twelve reports were excluded from the primary GPT-5.4 analysis because of persistent content-safety filtering or unparsable output, leaving 24,434 reports and 439,812 label instances across 18 abnormality categories. GPT-5.4-derived binary labels were generated from report text using structured outputs and compared with the original CT-RATE labels. Discordant instances were adjudicated by radiologists. In addition, 100 randomly sampled reports were manually annotated as a development set to compare the original CT-RATE labels, individual LLM-derived labels, and three-LLM majority-vote labels against radiologist-annotated reference labels. These label sources were then evaluated against a prespecified, stratified confirmatory set of 250 reports independently annotated by two blinded radiologists, and run-to-run consistency was assessed by repeated runs. Overall agreement between GPT-5.4-derived and CT-RATE labels was 96.4%, with a Cohen's κ of 0.884. Lymphadenopathy showed the lowest agreement and κ. In the confirmatory set (pre-consensus inter-reader κ = 0.936), the majority vote achieved an unweighted label-macro-averaged F1 score of 95.6% and κ of 0.940, compared with 90.8% and 0.879 for the original labels (paired F1 difference, + 4.8 points; 95% CI, 3.6-6.2), and each individual LLM also outperformed the original labels. The instance-level flip rate across three repeated runs was 1.02% for the majority vote. LLM-assisted label cleaning identified substantial label-report discordance in CT-RATE, particularly for lymphadenopathy, and may provide a scalable approach to quality improvement of public imaging datasets. The refined report-aligned labels and associated code will be made publicly available upon acceptance. Because all labels were derived from report text, they reflect report alignment rather than image-level truth.

Topics

Large Language ModelsTomography, X-Ray ComputedRadiology Information SystemsJournal Article

Ready to Sharpen Your Edge?

Subscribe to join 11k+ peers who rely on RadAISlice. Get the essential weekly briefing that empowers you to navigate the future of radiology.

We respect your privacy. Unsubscribe at any time.