Benchmarking generative AI outputs against human-led analysis of academic radiology workforce disparities.
Authors
Affiliations (3)
Affiliations (3)
- Faculty of Medicine, University of British Columbia, Vancouver, Canada.
- Department of Radiology, Faculty of Medicine, University of British Columbia, 2194 Health Sciences Mall #317, Vancouver, BC, V6T 1Z3, Canada. Electronic address: [email protected].
- Department of Radiology, Vancouver General Hospital, Vancouver, Canada.
Abstract
Generative artificial intelligence (AI) may assist academic radiology workforce research, but agreement with historical publications does not establish accuracy against supplied data. To evaluate descriptive numerical claims from ChatGPT 5.5, Copilot Pro, and Claude Opus 4.7 against independently recalculated 2007-2018 AAMC radiology data. Five AAMC workbooks were independently reanalyzed using Python, openpyxl, and SciPy. Annual counts, source-specific denominators, proportions, and changes were reconciled. A deterministic audit assessed 343 unique descriptive point estimates from archived outputs using exact count agreement and rounding-based tolerances. Rank and tenure denominator compatibility and statistical arithmetic were examined separately. The historical publication served as a comparator. The authors reported three fixed-input runs per tool. Faculty totals increased from 7,505 to 9,973 (32.9%), matching AI-reported endpoints but differing from historical totals of 7,553 and 9,652. Overall, 320/343 estimates matched: 51/51 ChatGPT, 52/75 Copilot, and 217/217 Claude. Copilot had 23 rank-share mismatches; its professor and assistant-professor shares implied incompatible denominators in both 2007 and 2018. Its tenure shares matched a restricted denominator. Degree and tenure totals exceeded demographics and rank by one White man in 2018. Reported repeat runs yielded identical numerical results. Statistical checks assessed arithmetic compatibility, without establishing independence or inferential validity. Independent source analysis distinguished verified AI estimates, source differences, compatible denominator restrictions, and numerical mismatches. Fixed-input repeatability did not eliminate errors or establish statistical validity. AI-assisted workforce research requires explicit denominators and independent validation.