My Model Is Better Than Yours! Statistically-aware Ranking for Fair Benchmarking of AI Models.
Authors
Affiliations (8)
Affiliations (8)
- Department of Pathology and Laboratory Medicine, Indiana University School of Medicine, Indianapolis, IN, USA.
- Indiana University Melvin and Bren Simon Comprehensive Cancer Center, Indianapolis, IN, USA.
- Departments of Radiology and Imaging Sciences, Neurological Surgery, Biostatistics and Health Data Science, Indiana University School of Medicine, Indianapolis, IN, USA.
- Department of Computer Science, Luddy School of Informatics, Computing, and Engineering, Indiana University, Indianapolis, IN, USA.
- MLCommons, San Francisco, CA, USA.
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, GA, USA.
- Center for Artificial Intelligence & Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
- Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
Abstract
The proliferation of artificial intelligence (AI) in healthcare has triggered the need of fair benchmarking. This, in turn, has inspired the rise of computational challenges, where participants worldwide submit models under standardized evaluation protocols. However, ranking AI models -particularly when declaring winners- raises questions about their difference from the next best (but lower-ranked) model. Here, we present a statistically-aware ranking framework, <b><i>PermRanker</i></b> , designed to be agnostic to computational workloads (e.g., segmentation, classification, registration) and underlying data types (e.g., 2D pathology images, 3D MRI scans, or even non-imaging data). PermRanker provides fair and informative benchmarking of AI models, based on two stages: (i) a ranking score, based on case-wise cumulative rankings aggregated across multiple metrics and testing cases, and (ii) a rigorous statistical significance analysis via pairwise permutation testing across the ranked order of the AI models. This framework has served as the official ranking mechanism for over 33 international challenges between 2017 and 2025, including the BraTS, FeTS, and ISLES challenges. While mainly applied in biomedical AI challenges, PermRanker aims to address the unmet need of fair and informative benchmarking of AI models beyond this scope, tackle actual real-world conditions, and hence contribute in streamlining clinical translation of AI models.