A Reproducible Benchmark for Leakage-Resistant Machine Learning on Multi-Site Brain Connectivity Graphs.
Authors
Affiliations (4)
Affiliations (4)
- Center of Data Science, Government College University Faisalabad, Faisalabad 38000, Pakistan.
- Department of Software Engineering, Government College University Faisalabad, Faisalabad 38000, Pakistan.
- Faculty of Computer Studies, Arab Open University, A'ali P.O. Box 18211, Bahrain.
- Department of Computer Engineering, Sivas University of Science and Technology, Sivas 58000, Türkiye.
Abstract
<b>Background/Objectives:</b> Machine-learning studies of autism spectrum disorder from resting-state functional magnetic resonance imaging routinely report near-perfect accuracy, yet the models rarely transfer to an unseen scanner. We audited a pipeline of the kind this literature builds, which reported an accuracy of 0.999, and traced that number to five data-leakage pathways. <b>Methods:</b> Under a nested leave-one-site-out protocol on 871 participants from the Autism Brain Imaging Data Exchange, in which the held-out site influenced nothing but model selection, the threshold, the combiner, or the site correction, we closed every pathway, priced each by reopening it, and validated on a locked external cohort. <b>Results:</b> With all five pathways closed, the graph attention network reached an area under the curve of 0.648 and did not outperform a support vector machine on the same connectivity (0.679; <i>p</i> = 0.009). Decision-level fusion added nothing detectable (0.682, <i>p</i> = 0.72). A network identified its training participants better than it detected autism (0.710 versus 0.610); both questionnaire benchmarks proved label-tautological, and performance held on the external cohort. <b>Conclusions:</b> Reported gains in this literature should be read as claims about evaluation protocols until shown otherwise on an untouched acquisition site; all predictions, models, and code are released.