Postdeployment Performance of a Commercial Artificial Intelligence System for Pulmonary Nodule Detection at Chest CT: Examination-Level Agreement, Clinical Indication, and Image-Based Review Across 13,536 Examinations.
Authors
Affiliations (2)
Affiliations (2)
- Department of Radiology, University of Florida, 1600 SW Archer Road, Gainesville, FL, 32611, USA. [email protected].
- Department of Radiology, University of Florida, 1600 SW Archer Road, Gainesville, FL, 32611, USA.
Abstract
Postdeployment performance of pulmonary nodule detection AI is rarely measured. This retrospective, cross-sectional analysis of prospectively collected routine data evaluated a commercial detection system (ClearRead CT; Riverain Technologies), deployed with a candidate-size threshold of greater than 3 mm, across 13,536 consecutive chest CT examinations at one academic center (January 2025-January 2026). Final reports, classified by natural language processing (NLP; validated against manual review of 896 reports: sensitivity 99.5%, specificity 99.6%, κ 0.99), served as the examination-level comparator; randomly selected discordant examinations were reviewed on images. Because radiologists viewed AI output before reporting, report-referenced results are presented as agreement; accuracy estimates are given only after image review. AI was positive in 9142 examinations (67.5%) and reports documented a nodule in 3919 (29.0%). Positive percent agreement was 80.6% (95% CI 79.4, 81.8), negative percent agreement 37.8%, and report-referenced positive predictive value 34.6%; 5982 examinations (44.2%) were AI positive/report negative, and Cohen κ was 0.13 (maximum attainable, 0.33). Agreement was 98.6% for screening, 38.3% for oncologic, and 32.6% for inpatient/emergency examinations, which contributed 76.1% of discordance. On image review, 291 of 298 (97.7%) AI-positive/report-negative examinations contained only non-nodule structures, and 196 of 198 (99.0%) AI-negative/report-positive examinations contained an unmarked nodule larger than 3 mm, 45 of them (23.0%) larger than 6 mm. Review-adjusted examination-level sensitivity was 81.2%, specificity 38.2%, and positive predictive value 35.4%. These findings describe one configuration at one institution; performance should be measured locally, by indication, and against image review.