Accuracy of Large Language Model Chatbots in Retrieving MRI Safety Conditions for Medical Implants.
Authors
Affiliations (5)
Affiliations (5)
- Department of Radiology, Chaim Sheba Medical Center, Tel Hashomer, Israel; Affiliated With the Gray Faculty of Medical and Health Sciences, Tel Aviv University, Tel Aviv, Israel.
- Division of Cardiology, Leviev Cardiovascular & Thoracic Center, Chaim Sheba Medical Center, Tel Hashomer, Israel; Affiliated With the Faculty of Medicine and Health Sciences, Tel Aviv University, Tel Aviv, Israel.
- Windreich Department of Artificial Intelligence and Human Health, Mount Sinai Medical Center, New York, New York, USA.
- Department of Neurology, Lady Davis Carmel Medical Center, Haifa, Israel.
- Department of Radiology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania, USA.
Abstract
Retrieving MRI safety conditions for implanted devices is increasingly complex and time-consuming. To evaluate the accuracy of internet-enabled large language model (LLM) chatbots in retrieving implant-specific MRI safety conditions, using manufacturer documentation as the reference standard. Prospective comparative evaluation in a constructed device cohort, followed by a retrospective clinical pilot. Fifty purposively selected device configurations and 20 patients (10 with cardiac and 10 with noncardiac devices) referred to the MRI Safety Committee. No field-strength or sequence restrictions. Standardized single-shot prompts requested eight parameters from ChatGPT-5.0, Gemini 2.5 Pro, and Claude Sonnet 4.5: MRI safety label, field strengths, eligible regions/centering, transmit-coil restrictions, RF exposure limits, maximum spatial gradient, maximum gradient slew rate, and scan duration. Two radiologists (10 and 25 years' experience) and an MRI technologist (32 years) independently scored 1200 parameter-level outputs against manufacturer Instructions for use and technical manuals using a 5-point scale (5, fully correct; 1, substantively incorrect or fabricated); scores of 4-5 indicated accuracy. In the pilot, 160 outputs from ChatGPT-5.6, a newer version of the best-performing chatbot, were scored identically. Intraclass correlation coefficient; Gwet's first-order agreement coefficient (AC1); Friedman test; Wilcoxon signed-rank tests; device-clustered bootstrap 95% confidence intervals (CIs); two-sided significance threshold, p < 0.05. Mean ordinal scores were 4.76 (ChatGPT), 4.20 (Gemini), and 4.17 (Claude); ChatGPT scored significantly higher. Respective pooled parameter-level accuracies were 0.94 (95% CI, 0.89-0.98), 0.79 (0.71-0.86), and 0.78 (0.69-0.87). Pilot accuracy with ChatGPT-5.6 was 0.98 (95% CI, 0.94-1.00). Internet-enabled LLM chatbots correctly retrieved up to 94% of MRI safety parameters in the constructed cohort and 98% in the clinical pilot. Errors occurred even with the best-performing chatbot, requiring direct verification against current manufacturer documentation before clinical use. Stage 2. Stage 2.