Multi-reader, multi-model benchmark of large language models for modified Outerbridge cartilage grading from knee MRI reports.
Authors
Affiliations (4)
Affiliations (4)
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, 15 Francis St, Boston, MA, 02115, USA.
- Harvard Medical School, 25 Shattuck St, Boston, MA, 02114, USA.
- Department of Radiology, Division of Musculoskeletal Imaging, Brigham and Women's Hospital, Harvard Medical School, 15 Francis St, Boston, MA, 02115, USA.
- Department of Radiology, Division of Musculoskeletal Imaging, Brigham and Women's Hospital, Harvard Medical School, 15 Francis St, Boston, MA, 02115, USA. [email protected].
Abstract
To describe a reproducible framework for bulk large language model (LLM)-based extraction of structured cartilage-lesion data from knee MRI reports and to benchmark seven LLM configurations against multiple radiologists using the modified Outerbridge classification. In this IRB-approved retrospective study, 100 non-contrast knee MRI reports (January 2019 to January 2025) were randomly selected from 66,479 eligible examinations and independently graded by five readers (four fellowship-trained musculoskeletal radiologists with 6-21 years of post-fellowship experience and one fourth-year resident) and seven LLM configurations, comprising six Azure OpenAI deployments (GPT-4.1, GPT-5.1-mini, GPT-5.3, GPT-5.4, GPT-5.4-mini, GPT-5.4-nano) and one locally hosted open-weight model (Qwen2.5-32B-Instruct), across a fixed 20-surface anatomic taxonomy. Pairwise Cohen quadratic-weighted κ was computed at compartment and surface levels. Bootstrap 95% confidence intervals (CIs) and p-values on the paired difference between each rater's mean κ and the human-human reference were derived by case-level cluster resampling (B = 1000). Mean human-human κ was 0.94 (95% CI 0.93, 0.95) at the compartment level and 0.80 (95% CI 0.78, 0.83) at the surface level. Three Azure deployments-GPT-5.4 (surface κ = 0.83), GPT-5.3 (κ = 0.80), and GPT-5.1-mini (κ = 0.77)-had 95% CIs on the difference from the human reference that included zero. Four models had CIs entirely below zero, indicating agreement significantly below the human reference: GPT-4.1, GPT-5.4-mini, Qwen2.5-32B, and GPT-5.4-nano (all p ≤ .02). Three flagship OpenAI deployments achieved cartilage grading agreement indistinguishable from human inter-rater variability, while cost-optimized and open-weight variants performed measurably below.