Back to all papers

Large Language Models and a Recently Revised Radiology Classification: Modified CIRSE Grading With and Without Current Criteria.

November 20, 2026pubmed logopapers

Authors

Türksayar O,Özgül H,Dablan A,Arslan MF,Erdim Ç,Cingöz M,Güzelbey T,Kılıçkesmez Ö

Affiliations (5)

  • Department of Radiology, Başakşehir Çam and Sakura City Hospital, Istanbul, Türkiye (O.T., A.D., M.F.A., Ç.E., and M.C.). Electronic address: [email protected].
  • Department of Radiology, Bağcılar Training and Research Hospital, Istanbul, Türkiye (H.Ö.).
  • Department of Radiology, Başakşehir Çam and Sakura City Hospital, Istanbul, Türkiye (O.T., A.D., M.F.A., Ç.E., and M.C.).
  • Department of Radiology, Medicana Ataköy Hospital, Istanbul, Türkiye (T.G.).
  • Department of Radiology, Memorial Göztepe Hospital, Istanbul, Türkiye (Ö.K.).

Abstract

Recently revised professional classifications may create a version mismatch for large language models, as plausible responses can reflect a superseded rather than the current schema. The modified Cardiovascular and Interventional Radiological Society of Europe classification, which subdivided Grades 1 and 3, provided a controlled setting to examine this problem. We evaluated four large language models with and without the current criteria supplied in the prompt. Sixty-four purpose-written vignettes, eight per category, were independently graded by two interventional radiologists, with consensus serving as the reference standard. Four models (gpt-5.6-sol, claude-sonnet-5, gemini-3.8-flash, and deepseek-flash) were queried under closed-book and guideline-augmented conditions three times per vignette, yielding 1536 requests. The primary outcome was run-1 exact-grade accuracy, compared within models using Holm-adjusted McNemar tests. Closed-book exact-grade accuracy ranged from 15 of 64 to 31 of 64, whereas all four models achieved 64 of 64 with the criteria supplied. All Holm-adjusted p values were <0.001. In a post hoc descriptive analysis, exact-grade accuracy was 0 of 128 classifications in the newly subdivided grades and 99 of 128 in grades retaining established labels. None of the four new subclass labels appeared in any of the 256 run 1 closed-book responses. Across all three closed-book runs, superseded grades accounted for 36.5% of responses. Exact-grade accuracy was higher in all four models when the current criteria were supplied. In a post hoc descriptive analysis, complete exact-grade failure was confined to newly subdivided grades. Current classification criteria should be supplied explicitly for model-based complication grading.

Topics

Journal Article

Ready to Sharpen Your Edge?

Subscribe to join 11k+ peers who rely on RadAISlice. Get the essential weekly briefing that empowers you to navigate the future of radiology.

We respect your privacy. Unsubscribe at any time.