Domain shift-aware evaluation of deep learning and radiomics-based machine learning in multi-vendor mammography.
Authors
Affiliations (4)
Affiliations (4)
- Medical Devices R&D Center, Gachon University Gil Medical Center, 21, 774 beon-gil, Namdong-daero, Namdong-gu, Incheon 21565, South Korea; Department of Nursing, College of Nursing, Gachon University, Incheon 21936, South Korea.
- Medical Devices R&D Center, Gachon University Gil Medical Center, 21, 774 beon-gil, Namdong-daero, Namdong-gu, Incheon 21565, South Korea; Department of Health Sciences and Technology, GAIHST, Gachon University, Incheon, 21999, South Korea.
- Medical Devices R&D Center, Gachon University Gil Medical Center, 21, 774 beon-gil, Namdong-daero, Namdong-gu, Incheon 21565, South Korea; Department of Gachon Biomedical & Convergence Institute, Gil Medical Center, Gachon University, Incheon, 21565, South Korea.
- Medical Devices R&D Center, Gachon University Gil Medical Center, 21, 774 beon-gil, Namdong-daero, Namdong-gu, Incheon 21565, South Korea; Department of Health Sciences and Technology, GAIHST, Gachon University, Incheon, 21999, South Korea; Department of Biomedical Engineering, College of IT Convergence, Gachon University, Seongnam, South Korea; Department of Gachon Biomedical & Convergence Institute, Gil Medical Center, Gachon University, Incheon, 21565, South Korea. Electronic address: [email protected].
Abstract
Artificial intelligence models for mammography are commonly developed under relatively homogeneous data conditions but are increasingly expected to operate across heterogeneous imaging environments, where domain shifts can substantially degrade performance. The comparative domain-shift robustness of radiomics-based machine learning and CNN-based deep learning models remains insufficiently characterized. This study aims to establish a controlled baseline robustness evaluation comparing radiomics-based machine learning (ML) and deep learning (DL) models across a progressive multivendor domain-shift hierarchy. We evaluated 80 model configurations comprising ML and DL architectures under three training strategies. Stratified five-fold cross-validation was performed using mammography datasets from two vendors (GE and Hologic) for same-vendor and cross-vendor evaluation, with CBIS-DDSM used as an external technology-shift stress test. Model performance was assessed using the area under the receiver operating characteristic curve with 95% confidence intervals, along with error profiles including sensitivity and specificity. Performance trajectories were characterized at both the best-performing model and model-family average levels. Model interpretability was evaluated using Shapley additive explanations for ML and gradient-weighted class activation mapping for DL. Under same-vendor conditions, ML achieved slightly higher AUCs than DL on both datasets (GE: AUC 0.96 vs. 0.94; Hologic: AUC 0.89 vs. 0.87). Cross-vendor evaluation demonstrated direction-dependent transfer asymmetry: both paradigms achieved comparable AUCs within each transfer direction (Hologic-to-GE: AUC 0.94; GE-to-Hologic: AUC 0.83), with paradigm-specific error profiles differing within each direction. Model-family trajectory analysis showed larger average AUC declines for ML than for DL as domain distance increased (cross-vendor: 6.04% vs. 2.23%; stress test: 36.58% vs. 25.33%). Under the external technology-shift stress test, DL showed attenuated performance degradation compared with ML under combined training (AUC 0.72 vs. 0.61), whereas ML exhibited pronounced specificity collapse. Radiomics-based ML achieved strong same-vendor performance and interpretable feature attributions, whereas CNN-based DL demonstrated relatively attenuated performance degradation under increasing domain distance, particularly in the stress-test setting and after combined multivendor training. These findings highlight the importance of evaluating AI systems not only by overall discrimination performance but also by vendor-specific error profiles and cross-vendor performance stability, to inform domain-aware validation and clinical translation in heterogeneous mammography environments.