Feasibility of Large Language Models for Suggesting Neurosurgical Approaches Under Structured Vignette Conditions: A Multimodal, Multi-Model Evaluation.

Authors

Abstract

Neurosurgical approach planning is cognitively demanding with substantial variability between experts. Large language models (LLMs) show promise on knowledge-based neurosurgical tasks such as board-style questions, but whether they can suggest appropriate surgical approaches for image-based tumor cases-and how neurosurgeons perceive such suggestions-has not been systematically evaluated. To evaluate the feasibility of LLMs for suggesting neurosurgical approaches under structured, expert-curated vignette conditions, to assess how neurosurgeons rate these suggestions, and to explore whether neurosurgeons self-reported that LLM suggestions influenced their approach preference across training levels. Sixty intracranial tumor cases spanning five categories (extra-axial deep, extra-axial superficial, intra-axial deep, intra-axial superficial, and pituitary) were curated from a publicly available, de-identified MRI dataset; one case was excluded for a technical error, leaving 59 for analysis. Each case included a synthetic structured clinical summary authored by an experienced neurosurgeon. Three commercial multimodal LLMs-GPT-5, Gemini 3 Pro, and Claude Sonnet 4.5-were queried between September and November 2025. For each case, a text-only prompt was issued first, followed by an MRI-augmented prompt within the same conversation-a sequential design that does not isolate independent image-based reasoning. Twenty-two neurosurgeons (11 trainees, 11 specialists) rated each blinded suggestion on a 5-point Likert scale across four domains and reported whether any suggestion influenced their approach preference. The primary metric was appropriateness-based acceptability (appropriateness ≥4). As ratings were clustered within raters and cases, key contrasts were verified with cluster bootstrap resampling; inter-rater reliability used Krippendorff's α. 1,296 recommendation-level evaluations (5,184 individual ratings) were collected. Appropriateness-based acceptability was 76.7% for text (95% CI 68.5-84.1) and 72.4% for MRI-augmented prompts (95% CI 64.2-80.2); although both point estimates exceeded the exploratory 70% benchmark, the confidence intervals extended below it, so acceptability above 70% at the population level was not established. Text prompts were rated higher than MRI-augmented prompts, persisting under clustering (p=0.020) and driven mainly by LLM-3. LLM-2 scored highest (mean 3.97), significantly above LLM-3 (p=0.004). Ratings varied markedly by tumor category: pituitary highest (87.4%) and intra-axial superficial lowest (56.5%; p=0.019). Trainees more often reported LLM influence than specialists (26.6% vs. 15.0%), but this was not robust to rater-level clustering (p=0.23) and is exploratory. Inter-rater agreement was low overall (α=0.270), higher among specialists (α=0.439) than trainees (α=0.173). Under structured, expert-curated vignette conditions, current LLMs generated plausible named approaches rated appropriate in roughly three-quarters of cases. Performance differed significantly and consistently by model and tumor category, with markedly lower ratings for intra-axial superficial tumors. Appending MRI after a text answer consistently lowered ratings, most strongly for LLM-3. Critically, error analysis of the lowest-rated cases revealed clinically important failures including incorrect laterality, underscoring that these outputs require expert oversight. LLM suggestions show promise as an educational adjunct complementing-not replacing-expert judgment.

Tags

Topics