Evaluation Design and Model Selection Stability in Medical Image Classification Benchmarks: A Multi-dataset Study.
Authors
Affiliations (4)
Affiliations (4)
- Department of Science and Technology, Nanjing Drum Tower Hospital, Affiliated Hospital of Medical School, Nanjing University, Nanjing, Jiangsu, China.
- Department of Obstetrics and Gynecology, Nanjing Drum Tower Hospital, Affiliated Hospital of Medical School, Nanjing University, Nanjing, Jiangsu, China.
- Jinshen College, Nanjing Audit University, Nanjing, Jiangsu, China. [email protected].
- Department of Nuclear Medicine, Nanjing Drum Tower Hospital, Affiliated Hospital of Medical School, Nanjing University, Nanjing, Jiangsu, China. [email protected].
Abstract
Medical image classification benchmarks inform model selection. We assessed how data splits, training seeds, and evaluation conditions affect selection stability and whether repeated evaluation budgets support reproducible rankings. The original study comprised 800 runs on OrganAMNIST and SIPaKMeD, using four ImageNet-initialized convolutional neural networks (CNNs), ten splits, five seeds, and two checkpoint selection and stopping procedures. The 300-run extension evaluated the International Skin Imaging Collaboration (ISIC) 2019 challenge dataset and MURA (musculoskeletal radiographs) using four CNNs, ten splits, and three seeds under procedure A; paired Swin Transformer-Tiny (Swin-T) analyses used five splits and three seeds. Balanced accuracy (BA) defined rankings. Paired resampling quantified selection frequencies and uncertainty; budget analyses compared discovery subsets with held-out splits. Single-context agreement with the full repeated evaluation reference was 44-62% in original strata, 90% for ISIC 2019, and 60% for MURA. The two procedures changed selected identities in 46% of OrganAMNIST and 56% of SIPaKMeD contexts. On matched five-split, three-seed contexts with shared preprocessing, adding Swin-T changed leave-one-split-out agreement from 86.7 to 20.0% in ISIC 2019 and from 66.7 to 80.0% in MURA. At five discovery splits and three seeds, agreement with held-out references ranged from 20.4 to 89.0% in original strata, reaching 100% for ISIC 2019 and 57.1% for MURA. Agreement did not increase uniformly across evaluated budgets. Repeated evaluation revealed both stable and changing selections. Reporting performance alongside selection stability informs budgeting and whether to advance one candidate or retain a shortlist.