Guardrail-Based Radiology Protocolling with Large Language Models: Balancing Automation and Oversight.
Authors
Affiliations (7)
Affiliations (7)
- Institute for Diagnostic and Interventional Radiology, School of Medicine and Health, TUM Klinikum, Technical University of Munich (TUM), Munich, Germany.
- Department of Radiology, McGill University Health Centre, Montreal, QC, Canada.
- Department of Medical Imaging, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
- Department of Pharmacology, Dalhousie Medicine New Brunswick, Faculty of Medicine, Dalhousie University, Saint John, NB, Canada.
- Department of Translational Medicine, The Hospital for Sick Children, Toronto, ON, Canada.
- Department of Radiology, McGill University Health Centre, Montreal, QC, Canada. [email protected].
- Cantonal Hospital of Frauenfeld and Münsterlingen, Frauenfeld, Switzerland. [email protected].
Abstract
Safe clinical deployment of large language models (LLMs) for radiology protocolling requires knowing when not to automate. We evaluated a "guardrail-first" workflow that defers incomplete or safety-critical CT and MRI requests to radiologists before assignment. In this single-center retrospective study, 537 body imaging requests were temporally split into prompt development (n = 109) and validation (n = 428) cohorts. An expert radiologist classified requests as protocol-eligible or requiring review. Six LLM pipelines (Gemini 3.1 Pro, Claude Sonnet 4.6, GPT-5.2, GPT-5.2 high, DeepSeek V3.2, and DeepSeek V3.2 reasoner) were evaluated in two stages (review triage, protocol assignment) against human protocolers. Primary outcome was clinically appropriate assignment (exact/equivalent protocol or correct review escalation), with bootstrapped 95% CIs. The expert reference standard classified 57/428 requests (13.3%) as requiring review (15 safety-critical, 23 major missing information, 19 minor clarification). For the 38 requests with safety-critical issues or major missing information, Gemini 3.1 Pro achieved 100% sensitivity (38/38; 95% CI, 90.7-100), versus 94.7%-97.4% for other pipelines. Among 371 protocol-eligible requests, unnecessary escalation ranged from 1.3% (5/371) to 34.0% (126/371), remaining lowest for reasoning-enabled models. The human benchmark achieved 83.6% (95% CI, 79.8-87.3) exact/equivalent assignment, exceeded by Gemini 3.1 Pro (+12.7 percentage points; 95% CI, 8.6-16.7), Claude Sonnet 4.6 (+12.4; 8.4-16.4), GPT-5.2 high (+11.6; 7.3-15.9), and DeepSeek V3.2 reasoner (+8.6; 4.3-12.9). End-to-end clinically appropriate assignment ranged from 65.0% (60.5-69.4) for GPT-5.2 to 96.3% (94.4-97.9) for Gemini 3.1 Pro; the three highest-performing pipelines did not differ significantly. A guardrail-first LLM protocolling workflow combined high-performance automation with reliable identification of requests requiring radiologist review, supporting further prospective evaluation.