AI drafts and grades image-based neurosurgery exam questions with expert oversight

Very short answer questions are prized in medical education because, unlike multiple-choice items, they leave students nowhere to hide: without answer options to cue them, trainees must actually produce the answer. The catch is that writing and grading these questions is labor-intensive, and the burden grows sharply when each question hinges on interpreting a neurosurgical MRI scan. A new study in BMC Medical Education tested whether multimodal large language models, which can process both text and images, could lighten that load under close expert supervision.

The study was a three-phase, single-center project. Researchers gave two multimodal large language models identical clinical information and three MRI images from each of 36 deidentified neurosurgical cases, asking each model to generate one image-dependent very short answer question per case. That process produced 72 candidate items, two for every case.

Three senior neurosurgeons then rated every candidate question on four dimensions: scientific accuracy, image dependency, educational value, and clarity. For each case, the higher-scoring of the two competing items was selected, yielding a 36-item question set the authors call the Golden Set. Both models earned high ratings for scientific accuracy, educational value, and clarity, with no significant between-model differences. ChatGPT did outperform Gemini on image dependency, with a mean rating of 4.06 (SD 0.44) versus 3.58 (SD 0.52), a mean difference of 0.48 (95% CI 0.26–0.71; P < 0.001), while total scores across the other criteria did not differ significantly between the two models.

The Golden Set was then administered to 10 neurosurgery residents, who produced 360 responses in total. Performance showed the questions were genuinely demanding: 275 responses (76.4%) were fully correct, 28 (7.8%) partially correct, and 57 (15.8%) incorrect. Resident total scores ranged from 47 to 63 out of a possible 72.

Grading was the second half of the test. Two senior neurosurgeons independently scored all 360 responses on a 0–2 rubric and resolved disagreements by consensus. Before that consensus step, the two experts already agreed on 347 responses, or 96.4%, with a quadratic weighted kappa of 0.967, indicating near-perfect baseline agreement between human raters.

ChatGPT then graded the same 360 responses using case-specific structured inputs and a fixed prompt. The model’s scores matched the final expert scores exactly for 324 responses, or 90.0%, with a quadratic weighted kappa of 0.905. At the participant level, AI-derived totals correlated strongly with expert totals, with a Pearson correlation of r = 0.970. Between-model ratings were compared with paired t tests, agreement with exact agreement and quadratic weighted kappa, and participant-level totals with Pearson correlation.

The authors conclude that, under the structured conditions evaluated, multimodal large language models showed preliminary feasibility for expert-supervised drafting and first-pass grading of image-dependent neurosurgical very short answer questions. The emphasis on expert supervision is important: in this workflow, humans selected the better question for each case and set the reference scores against which the AI’s grading was measured, rather than the models operating autonomously.

The study’s own limitations frame how far the result can be generalized. It was conducted at a single center with 10 residents and 36 cases, and the grading task was bounded by a fixed prompt and structured inputs. The authors state explicitly that larger multicenter studies and external validation are needed before routine implementation or any use in high-stakes assessment. Within those boundaries, the findings suggest a practical division of labor in which AI handles first-pass drafting and scoring of image-based questions while senior neurosurgeons retain final judgment.

Subject of Research: Testing multimodal large language models for generating and grading image-based very short answer questions in neurosurgical residency assessment

Article Title: Multimodal large language model–assisted workflow for image-dependent very short answer question generation and response grading in neurosurgical residency assessment: development and validation study

Article References: Yuan, Y., Zheng, B., Yao, C., Pan, N., Yan, L., Surau, M., Becker, P. A., Hu, T., & Yan, P. (2026). Multimodal large language model–assisted workflow for image-dependent very short answer question generation and response grading in neurosurgical residency assessment: development and validation study. BMC Medical Education. https://doi.org/10.1186/s12909-026-10295-7

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10295-7

Keywords: large language models, medical education, neurosurgery, residency assessment, VSAQ, MRI, AI grading, ChatGPT, Gemini, question generation, expert supervision, validation study