Multicohort Evaluation of Acquisition Confounding in Single-Site Lung-CT Malignancy Classification with Frozen Foundation-Model Features.
Authors
Affiliations (4)
Affiliations (4)
- Department of Thoracic Surgery and Critical Care Medicine, Lotung Pohai Hospital, Ilan, Taiwan.
- Department of Radiology, Lotung Pohai Hospital, Ilan, Taiwan.
- Department of Biomedical Engineering, Chang Gung University, Taoyuan, Taiwan. [email protected].
- Department of Cardiology, Chang Gung Memorial Hospital, Taoyuan, Taiwan. [email protected].
Abstract
Frozen foundation-model features are promoted as an annotation-sparing route to medical-image classification, yet whether their performance reflects diagnostic signal or site-specific shortcuts is rarely tested. On a single-site hospital cohort (187 cancer and 194 benign scans, 303 patients), frozen Models-Genesis whole-scan features classified cancer versus benign at an out-of-fold AUC of 0.994 under patient-grouped cross-validation and transferred near chance to two external cohorts. One is LIDC-IDRI, where malignancy is a median radiologist rating, not a confirmed diagnosis. The other carries confirmed diagnoses matching the development endpoint. We tested acquisition as a competing explanation. The two classes had been reconstructed under different protocols (0.7 mm versus 1.0 mm slice spacing), and a classifier reading six acquisition header fields, and no image data, reached 0.996, within 0.001 of the frozen-feature result ( <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi> <mo>=</mo> <mn>0.73</mn></mrow> </math> ). The near-perfect in-site AUC is therefore most plausibly explained by acquisition-related confounding, which extends beyond reconstruction protocol to scanner model and age, and is not evidence of malignancy discrimination. A second, independently pretrained CT foundation model reproduced the pattern, so it is a property of the data, not one checkpoint. Malignancy-related signal appeared at finer spatial granularity. Narrowing the field of view to one prespecified nodule per scan raised AUC from 0.584 to 0.808, and a nodule-level classifier transferred between external cohorts at 0.682 (95% CI 0.559-0.792) against confirmed benign controls under one of the two encoders. Whole-scan embeddings exposed a cohort-assembly shortcut strong enough to mimic a solved task; the signal that transferred was carried by lesion-level representations.