Five AI chatbots put to the test on patient questions about brain-related vision loss

When a patient suddenly notices a missing patch in their field of vision, the cause may lie not in the eye itself but in the brain. Visual field defects of this kind can signal stroke, intracranial tumors, traumatic brain injury, or demyelinating disease, and the questions patients ask about them span symptom recognition, urgent care seeking, diagnosis, treatment, rehabilitation, and psychological support. A new cross-sectional study published in Frontiers in Physiology has now compared how five leading large language models answer exactly those questions in Chinese, revealing a performance landscape in which no single chatbot comes out on top across all the dimensions that matter clinically.

The stakes are unusually high in this setting. Injury anywhere along the central visual pathway, from the optic nerve through the chiasm, optic tract, and optic radiations to the visual cortex, can produce bitemporal or homonymous field defects. These defects restrict driving, walking, and obstacle detection, and in severe cases are associated with an increased risk of falls requiring hospitalization. Because the underlying lesions may involve ophthalmology, neurology, neurosurgery, diagnostic imaging, and visual rehabilitation simultaneously, patient inquiries extend well beyond symptom explanation within a single specialty. How accurately and safely an automated system answers them may influence whether a patient recognizes the urgency of care and what they do next.

To evaluate this task directly, a research team from Southern Medical University, Fujian Medical University, Wenzhou Medical University, and Shenzhen Eye Hospital constructed a bank of 30 fixed Chinese-language questions phrased from a patient perspective and refined by a clinical expert panel. The framework drew on the European Stroke Organisation guideline on visual impairment after stroke, the Pituitary Society consensus on pituitary incidentaloma, the consensus guideline on idiopathic intracranial hypertension, and the American College of Radiology Appropriateness Criteria for vision loss. Questions covered six predefined domains: symptom recognition and self-assessment, brain diseases and visual-field changes, care-seeking and diagnostic testing, treatment and intervention, rehabilitation and daily care, and psychological support and prognosis.

On 15 August 2026, the researchers submitted each question to five API-accessible models: ChatGPT 5.6 Sol from OpenAI, DeepSeek V4 Flash, Gemini 3.7 Flash from Google, Qwen 3.8 Max from Alibaba, and Kimi K3 from Moonshot AI. All responses were collected through a unified open-source API client within a single testing window, with web search and browsing disabled, no custom system prompts, and conversation history cleared before every query. Each model generated one response per question in an independent single-turn session, producing 150 responses in total. This standardized procedure reduced the influence of hidden prompts, retrieval behavior, and output-format differences, improving internal consistency across the model comparisons.

Three raters engaged in clinical ophthalmology research then scored all 150 responses, blinded to model identity and presented in random order, using a prespecified five-dimensional rubric covering accuracy, logic, coherence, safety, and content accessibility, each rated from 1 to 5. Agreement among the raters was high across all dimensions, with Kendall’s W ranging from 0.820 to 0.844 and all tests significant at P < 0.001. Statistical comparisons used the Friedman test paired by question, followed by Bonferroni-adjusted Wilcoxon signed-rank tests for the ten pairwise comparisons within each measure. The team also recorded response time, word count, non-space character count, sentence count, paragraph count, and table count for every response.

The results showed distinctly multidimensional performance. The five models differed significantly in accuracy, coherence, and content accessibility, but not in logic or safety. ChatGPT 5.6 Sol earned the highest median accuracy score at 4.67 on the interquartile range of 4.00 to 5.00, while Gemini 3.7 Flash received the lowest at 3.17. Kimi K3 achieved the highest median coherence score at 4.50, whereas Qwen 3.8 Max had the lowest at 3.67. Both ChatGPT 5.6 Sol and Kimi K3 posted a median content accessibility score of 4.67, and both scored significantly higher than Gemini 3.7 Flash on that dimension. No model, however, consistently performed best across all clinically relevant dimensions.

Objective output characteristics also varied widely. DeepSeek V4 Flash responded fastest, with a mean response time of 17.3 seconds, while Qwen 3.8 Max was slowest at 36.1 seconds. Qwen 3.8 Max generated the highest mean word count at roughly 1,338 words and the most paragraphs, while ChatGPT 5.6 Sol produced the fewest at about 499 words and 6 paragraphs. Yet when the researchers pooled all 150 responses and explored Spearman correlations between these objective measures and expert quality ratings, the associations were generally weak, with absolute correlation coefficients not exceeding 0.279. Word and non-space character counts, which were almost perfectly correlated with each other, were actually weakly negatively correlated with accuracy, coherence, safety, and content accessibility. Longer or more structurally complex responses were not necessarily better ones.

Safety deserves particular scrutiny. Although no statistically significant between-model difference emerged in safety ratings, and all models received relatively high median scores ranging from 4.17 to 5.00, some individual responses still received low safety marks. Final safety scores of 3 or below were recorded in 2 of 30 ChatGPT 5.6 Sol responses, 3 of 30 DeepSeek V4 Flash responses, 2 of 30 Qwen 3.8 Max responses, and 3 of 30 Gemini 3.7 Flash responses, while Kimi K3 recorded none. One DeepSeek V4 Flash response and two Gemini 3.7 Flash responses scored 2 or below. The authors caution that with only 30 responses per model, rare serious errors may remain undetected, and that the absence of a significant difference under controlled conditions does not imply equivalent or adequate clinical safety, especially when visual field defects may require timely stroke assessment.

Several limitations frame the findings. Each question was submitted only once to each model, so within-model variability across repeated generations could not be estimated, and the results are specific to the model versions tested on 15 August 2026. End-to-end response times included differences in APIs, networks, client software, and manual timing error, and should not be interpreted as intrinsic model reasoning speed. Web retrieval was uniformly disabled, so the results reflect standardized single-turn text interactions without retrieval augmentation and cannot be generalized to systems using real-time search or specialized knowledge bases. Notably, previous research showed that adding a curated professional guideline database to the same GPT-4o model increased guideline-concordant correct responses from 60 percent to 87 percent and reduced incorrect responses from 12 percent to 4 percent.

The study’s conclusion is measured rather than promotional. Large language models are better suited to supporting patient education and clinician-patient communication under clinical supervision than to making diagnostic or treatment decisions independently, and model selection should be task specific, weighing medical accuracy, content accessibility, and clinical safety together rather than relying on any single measure. The authors argue that response speed, length, and structure cannot substitute for direct assessment of medical quality, and that future research should test multi-turn and retrieval-augmented systems, quantify rare but high-risk errors, and evaluate models using patient-relevant outcomes such as patient understanding, care-seeking decisions, and follow-up adherence. As automated health information becomes an everyday fixture of patient life, this study offers a template for how such systems should be judged: not by how fast or how long they answer, but by whether what they say is correct, understandable, and safe.

Subject of Research: A cross-sectional comparison of five large language models in answering Chinese-language patient queries on visual field defects associated with brain diseases

Article Title: A cross-sectional comparison of five large language models in answering Chinese-language patient queries on visual field defects associated with brain diseases

Article References: Li, S., Wang, L., Wang, X., Jiang, L., Yang, W., & Xie, L. (2026). A cross-sectional comparison of five large language models in answering Chinese-language patient queries on visual field defects associated with brain diseases. Frontiers in Physiology, 17, Article 1993774. https://doi.org/10.3389/fphys.2026.1993774

Image Credits: AI Generated

DOI: 10.3389/fphys.2026.1993774

Keywords: cross-sectional, comparison, five, large, language, models, answering, Chinese-language, patient, queries, visual, field