Evaluating the Interplay and Impact of Quality and Technical Variations on Performance of AI Models in Multi-Institutional Prostate MRI Datasets
Authors
Affiliations (1)
Affiliations (1)
- Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, Georgia, USA.
Abstract
PurposeTo evaluate whether visually assessed high-quality prostate MRI, as defined by the Prostate Imaging Quality (PI-QUAL) scoring system, is sufficient to ensure consistent AI model performance, or whether hidden acquisition-related technical effects introduce variability that cannot be detected through visual inspection. MethodsThis retrospective study included 568 bpMRI scans from seven institutions, classified using PI-QUAL v2 into high-quality (scores 2-3) and low-quality (score 1) groups. Quantitative image quality features were extracted using MRQy from T2w and ADC sequences. Three experiments were conducted on high-quality scans: (1) unsupervised clustering to identify hidden technical batch effects; (2) V-Net models (M) for prostate capsule segmentation on T2w using single-cluster and balanced multi-cluster training strategies; and (3) VoxelMorph-based deformable registration (M2) to align ADC maps to T2w images. Performance metrics included DSC, precision, recall, volume ratio, HD95, ASSD, and TRE. Kruskal-Wallis tests assessed cross-cluster variability, Two One-Sided Tests (TOST) evaluated performance equivalence, and bootstrap resampling quantified training efficiency. ResultsClustering identified four clusters among 277 high-quality bpMRI scans. Single-cluster models showed variable performance (DSC=0.41-0.85) with significant batch effects (p<0.001), even with 140 training samples. The balanced multi-cluster model (n=60) eliminated batch effects (p=0.075), achieved equivalent performance (p<0.001), and demonstrated 2.23-fold greater training efficiency despite using 2.3-fold fewer samples. Registration performance also varied significantly across clusters (TRE=0.86-5.47 mm, p<0.001). ConclusionAcquisition-related batch effects significantly impact AI generalizability in prostate MRI, even among high-quality scans. Quantitative clustering enables detection of technical variations and supports diversity-aware training strategies that achieve equivalent performance with 2.3-fold fewer samples while eliminating batch effects. SummaryAcquisition-related batch effects significantly impair AI generalizability in prostate MRI, even among scans meeting PI-QUAL quality standards. Clustering-based diversity-aware training strategies successfully eliminated these batch effects and achieved 2.23-fold greater training efficiency compared to single-site approaches, demonstrating that strategic technical diversity outweighs dataset size for developing robust, clinically deployable AI models. Key PointsO_LIAcquisition-related batch effects significantly impair AI performance on prostate MRI, exceeding the impact of visual image quality as assessed by PI-QUAL scoring. C_LIO_LIEven the model trained on the largest single cluster exhibited significant batch effects (p<0.001). In contrast, a balanced multi-cluster model eliminated cross-cluster variability (p=0.075) while maintaining equivalent performance and requiring a smaller sample size. C_LIO_LIDiversity-aware training achieved 2.23-fold greater training efficiency (95% CI: 2.18-2.29) with 2.3-fold fewer samples, demonstrating that technical diversity outweighs data quantity for robust AI generalization. C_LI Clinical relevanceAddressing batch effects is essential for developing reliable, generalizable AI tools for prostate MRI interpretation.