Large Language Models and a Recently Revised Radiology Classification: Modified CIRSE Grading With and Without Current Criteria.
Authors
Affiliations (5)
Affiliations (5)
- Department of Radiology, Başakşehir Çam and Sakura City Hospital, Istanbul, Türkiye (O.T., A.D., M.F.A., Ç.E., and M.C.). Electronic address: [email protected].
- Department of Radiology, Bağcılar Training and Research Hospital, Istanbul, Türkiye (H.Ö.).
- Department of Radiology, Başakşehir Çam and Sakura City Hospital, Istanbul, Türkiye (O.T., A.D., M.F.A., Ç.E., and M.C.).
- Department of Radiology, Medicana Ataköy Hospital, Istanbul, Türkiye (T.G.).
- Department of Radiology, Memorial Göztepe Hospital, Istanbul, Türkiye (Ö.K.).
Abstract
Recently revised professional classifications may create a version mismatch for large language models, as plausible responses can reflect a superseded rather than the current schema. The modified Cardiovascular and Interventional Radiological Society of Europe classification, which subdivided Grades 1 and 3, provided a controlled setting to examine this problem. We evaluated four large language models with and without the current criteria supplied in the prompt. Sixty-four purpose-written vignettes, eight per category, were independently graded by two interventional radiologists, with consensus serving as the reference standard. Four models (gpt-5.6-sol, claude-sonnet-5, gemini-3.8-flash, and deepseek-flash) were queried under closed-book and guideline-augmented conditions three times per vignette, yielding 1536 requests. The primary outcome was run-1 exact-grade accuracy, compared within models using Holm-adjusted McNemar tests. Closed-book exact-grade accuracy ranged from 15 of 64 to 31 of 64, whereas all four models achieved 64 of 64 with the criteria supplied. All Holm-adjusted p values were <0.001. In a post hoc descriptive analysis, exact-grade accuracy was 0 of 128 classifications in the newly subdivided grades and 99 of 128 in grades retaining established labels. None of the four new subclass labels appeared in any of the 256 run 1 closed-book responses. Across all three closed-book runs, superseded grades accounted for 36.5% of responses. Exact-grade accuracy was higher in all four models when the current criteria were supplied. In a post hoc descriptive analysis, complete exact-grade failure was confined to newly subdivided grades. Current classification criteria should be supplied explicitly for model-based complication grading.