The research was carried out at the Affiliated Eye Hospital of Jiangxi Medical College, Nanchang University, in China, and rests on one of the largest datasets ever assembled for this purpose. The retrospective analysis encompassed 103,237 ophthalmic ultrasound images and 51,618 diagnostic reports drawn from 51,618 patients. The scale matters. Teaching an artificial intelligence system to read an ultrasound image and describe it the way an experienced ophthalmologist would requires tens of thousands of paired examples of images and the corresponding clinical text. The ethics committee of the hospital approved the study, and because it was a retrospective analysis of de-identified existing data, the requirement for individual informed consent was waived.

At the technical heart of OphthUS-GPT is a two-stage framework built on the Bootstrapping Language-Image Pre-training model, widely known as BLIP, a vision-language architecture that learns to connect visual content with descriptive text. The researchers fine-tuned BLIP in two distinct stages. In the first stage, the model was trained to produce clinical findings—the objective descriptions of what appears in the ultrasound image, such as the presence of echogenic material in the vitreous cavity or an elevated retinal membrane. In the second stage, the model learned to integrate those findings with clinical impressions, producing the complete diagnostic report structure that ophthalmologists use in daily practice. This staged approach mirrors the way human clinicians actually reason: first observing, then interpreting.

But generating a report is only half the problem. A written description of an ultrasound scan is of limited value to a patient, a trainee, or even a non-specialist physician if it cannot be understood. To address this, the team incorporated a large language model—DeepSeek, deployed locally—into the system to serve as an interactive question-answering module. Users can ask follow-up questions about a generated report, request clarification of medical terminology, or explore the clinical implications of specific findings, and the system responds through multi-turn dialogue. The result is not merely an automated typist but an interpretive assistant designed to make ophthalmic ultrasound findings accessible and actionable.

Evaluating such a system demands rigorous, multi-dimensional testing, and the researchers deployed three complementary strategies. First, they measured text quality using standard natural language generation metrics: BLEU, which measures n-gram overlap with reference reports; ROUGE, which captures longer matching sequences; and CIDEr, which weighs consensus with human-written descriptions. OphthUS-GPT achieved a BLEU-1 score of 0.5739, a ROUGE-L score of 0.6131, and a CIDEr score of 0.9818 for report generation—figures indicating substantial agreement between machine-generated text and the reports written by clinicians.

Second, the team assessed diagnostic performance through disease classification metrics, measuring accuracy, sensitivity, specificity, and F1 score across the range of conditions that appear in posterior segment ultrasound. The results were strongest for common conditions. For the majority of frequently encountered findings—including vitreous opacities, retinal detachment, posterior scleral staphyloma, and cataracts—classification accuracy exceeded 90 percent. Across all evaluated disease categories, overall accuracy exceeded 80 percent, a level of performance that suggests the system could function reliably as a first-pass drafting and screening tool even for less common pathology.

Third, and perhaps most importantly for clinical credibility, expert ophthalmologists rated the generated reports for accuracy and completeness. More than 90 percent of the reports produced by the system received scores of 3 or higher—the evaluation scale’s threshold for acceptable—on both criteria. This expert-validated quality is crucial, because statistical similarity metrics alone cannot guarantee that a report is medically sound. A report can match the vocabulary of its references while missing a critical finding; human review remains the gold standard, and by this measure OphthUS-GPT performed well.

The interactive question-answering module was evaluated separately, on four dimensions: accuracy, completeness, security, and user satisfaction. Here the study produced one of its most interesting comparative findings. The locally deployed DeepSeek model demonstrated statistically superior accuracy compared with Claude, and higher user satisfaction compared with both Claude and GPT-4 Turbo. No significant differences emerged between the models in completeness or security. The comparison matters for real-world deployment: large language models differ in how faithfully they answer medical questions and how satisfied clinicians and patients are with their responses, and the finding suggests that a locally hosted open model can outperform commercial alternatives on the dimensions that matter most, while also addressing data privacy concerns that often constrain hospital AI adoption.

The implications extend well beyond a single hospital in Nanchang. Manual ultrasound report generation is time-consuming and heavily dependent on clinician expertise, and in many parts of the world—particularly in lower-resource settings and in clinics without an on-site ultrasound specialist—ophthalmic B-scan interpretation is a genuine bottleneck in eye care. An automated system that produces expert-quality reports and explains them conversationally could shorten reporting times, standardize documentation quality, support less experienced examiners, and free clinicians to spend more time with patients and less time at the keyboard. Because the system includes an interpretive dialogue layer, it could also serve as a teaching tool for ophthalmology trainees learning to read ultrasound images, offering immediate, interactive explanations of findings and terminology.

The work, led by Fan Gan, Lei Chen, Weiguo Qin, and colleagues including corresponding author Zhipeng You, with Fan Gan and Lei Chen contributing equally, was supported by the Jiangxi Provincial Health Commission and the Public Hospital High-Quality Development Research Public Welfare Project. It represents a broader shift in medical artificial intelligence: away from narrow, single-task classifiers and toward integrated multimodal systems that combine computer vision, natural language generation, and conversational reasoning in a single clinical workflow. Challenges remain before such systems become routine—the evaluation was retrospective, prospective clinical deployment would require further validation across diverse populations and equipment, and any AI-generated report would still need clinician review. But the study demonstrates that the pieces can fit together. A vision-language model trained on more than one hundred thousand real ultrasound images can produce reports that experts judge accurate and complete, and a large language model layered on top can turn those reports into an interactive clinical conversation. For a diagnostic field where the written report has long lagged behind the image itself, that is a meaningful step forward.

Subject of Research: A multimodal artificial intelligence system for generating and interpreting ophthalmic B-scan ultrasound diagnostic reports

Article Title: OphthUS-GPT: a multimodal AI system for ophthalmic B-scan ultrasound report generation and interpretation

Article References: OphthUS-GPT: a multimodal AI system for ophthalmic B-scan ultrasound report generation and interpretation. (n.d.). https://doi.org/10.1186/s40537-026-01563-w

Image Credits: AI Generated

DOI: 10.1186/s40537-026-01563-w

Keywords: artificial intelligence, multimodal learning, ophthalmic B-scan ultrasound, report generation, BLIP, DeepSeek, large language models, ophthalmology, medical imaging, clinical decision support, vision-language models, deep learning

The technical hurdle the team set out to overcome is deceptively simple to state and extraordinarily difficult to solve. A single whole-slide image can contain billions of pixels, far exceeding the input capacity of conventional vision-language models. Most existing multimodal AI assistants in pathology, including widely cited systems adapted from general-purpose models, operate at the patch level, examining isolated regions of tissue. That approach works for tasks such as detecting mitotic figures or classifying small lesions, but it fails when a diagnosis depends on global tissue architecture. Determining tumor staging, for example, requires appreciating whether invasive cells have breached anatomical boundaries visible only when the entire slide is considered. The researchers demonstrated this limitation directly: in a challenging bladder cancer case, patch-level models misidentified the invasive stage, likely because they could not integrate spatial context across the slide, while SlideChat correctly analyzed the global architecture and reported the accurate pT3 stage.

SlideChat’s architecture combines three essential ingredients. First, it integrates a patch-level pathology encoder, which captures fine-grained cellular and subcellular detail, with a slide-level pathology encoder that aggregates information across the full specimen into a compact representation. Second, it connects these visual encoders to a pretrained large language model, allowing the system to translate visual evidence into fluent natural-language answers. Third, and perhaps most importantly, it is trained on SlideInstruction, a new dataset of 274,233 multimodal instruction samples that pair whole-slide images with diagnostic reports and question-answer pairs. The instruction-tuning paradigm, which proved transformative in general-purpose chatbots, teaches the model not merely to classify images but to interpret complex, realistic clinical queries and respond in the structured language that pathologists actually use.

Building the training data required considerable curation. The team drew on publicly available resources including The Cancer Genome Atlas and the Clinical Proteomic Tumor Analysis Consortium, whose whole-slide images and associated pathology reports are accessible through the National Institutes of Health data commons. Additional data came from the BCNB breast cancer cohort and the HISTAI dataset, while a retrospectively collected hepatobiliary cohort, gathered under approval from the Eastern Hepatobiliary Surgery Hospital, provided controlled-access material. Raw reports were parsed and cleaned, concise captions were extracted to teach image-language alignment, and instruction-style question-answer pairs were generated from the reports and labels. A rigorous quality-control pipeline then applied large language model filtering and pathologist verification to ensure that questions genuinely could not be answered without visual inspection of the slide, guarding against shortcuts that would let the model succeed through text alone.

The evaluation was among the most comprehensive ever assembled for slide-level pathology AI. The team created SlideBench, a benchmark spanning five cohorts and 31 cancer types, comprising 8,836 closed-ended questions, 129 open-ended questions and 3,149 whole-slide diagnostic reports. On closed-ended questions, SlideChat outperformed leading baseline models by 19.1 percentage points in accuracy, a margin the authors validated with two-sided Wilcoxon signed-rank tests and Benjamini-Hochberg correction across 1,000 bootstrap replicates. On report generation, judged by the Metric for Evaluation of Translation with Explicit Ordering, a standard automatic measure of textual overlap with reference documents, SlideChat exceeded the best baselines by 7.7 points. For open-ended questions, expert pathologists scored SlideChat highest across five evaluation dimensions, noting that its answers were the most diagnostically accurate and case-specific. In one representative differential-diagnosis case, SlideChat integrated key morphological features to support a refined melanoma diagnosis, while a leading general-purpose model produced a broad but weakly prioritized differential and competing medical models returned generic, weakly reasoned answers.

The comparison with general-purpose and specialist baselines was particularly revealing. Models such as GPT-4o, LLaVA-Med, Quilt-LLaVA and dedicated slide-level systems including HistoGPT and PRISM all trailed SlideChat on most tasks. In report generation case studies, SlideChat consistently produced accurate, structured reports that captured tumor type, anatomical location, invasion status, lymphovascular features, TNM staging and relevant histological details, closely matching reference clinical reports. HistoGPT, by contrast, sometimes produced anatomically inconsistent or diagnostically mismatched descriptions, while PRISM returned brief diagnoses lacking contextual or morphological explanation. In renal and lung carcinoma examples from the CPTAC cohort, SlideChat correctly identified tumor subtype, Fuhrman grade, pT2a staging, the absence of vascular and perineural invasion and negative margins, and even contextualized incidental benign findings such as emphysematous changes in the lung without drifting into irrelevant speculation.

The study also probes how the model thinks, offering an unusually transparent window into its behavior. Question-guided attention heatmaps show that SlideChat reallocates its visual focus depending on the query: when asked about cytology in an adenocarcinoma case, attention concentrates on nuclear detail, while a question about differentiation shifts attention to glandular architecture; in a breast carcinoma case, the model attends to fibrous stroma when asked about stromal reaction but to tumor nests when asked about cellular arrangement. Ablation experiments confirmed that both the slide-level encoder and the two-stage training procedure are indispensable to performance. Sensitivity tests in which Gaussian noise was injected into patches with high or low attention scores showed a monotonic drop in accuracy when meaningful regions were corrupted, indicating that the model relies on genuine visual content rather than artifacts or dataset shortcuts. When adapted patch-level foundation models were given whole-slide inputs through strategies such as patch voting, thumbnail downsampling or joint multi-patch input, SlideChat still outperformed them, often by statistically decisive margins.

Not everything in the evaluation was flattering, and the authors are candid about the model’s weaknesses. In multiturn conversational settings, SlideChat exhibited cross-turn inconsistency in some cases, at one point initially identifying a node-negative tumor and later contradicting that finding during prognostic assessment. The model also occasionally produced self-contradictory hallucinations, describing a tumor as confined to the epithelium while simultaneously reporting vascular invasion. These failure modes mirror the reliability challenges that afflict large language models generally, and their documentation in a rigorous pathology benchmark provides a concrete target for future work. The researchers also note that performance did not correlate strongly with the number of training samples per organ, suggesting that data scale alone does not determine competence and that data quality and task diversity matter substantially.

The potential applications extend beyond diagnosis. The authors highlight medical education as a promising use case, since a system that can answer natural-language questions about whole slides could serve as an interactive tutor for pathology trainees. Clinical decision support is another frontier: by integrating slide-level reasoning with conversational interaction, the assistant could help oncologists and pathologists rapidly extract staging, grading and prognostic information from complex specimens. Notably, SlideChat achieves performance competitive with specialized pathology foundation models such as CONCH, TITON-class slide encoders, CHIEF and Prov-GigaPath on breast cancer classification tasks involving tumor status and hormone receptor and HER2 status, despite being a general-purpose assistant rather than a task-specific classifier. The team has released the training and evaluation data on Hugging Face, the model source code on GitHub and the model weights publicly, a level of openness that could seed a new generation of slide-level pathology AI research.

The work arrives at a moment when computational pathology is transitioning from narrow, single-task algorithms to foundation models and generative assistants, and it addresses what many in the field identify as the central bottleneck: the gap between patch-level perception and slide-level clinical reasoning. By demonstrating that a single model can answer closed-ended questions, engage in open-ended diagnostic dialogue and generate expert-quality reports across a broad range of cancers, and by subjecting that model to pathologist review, extensive ablations and honest failure analysis, the study sets a new reference point for what clinically useful pathology AI looks like. Substantial work remains before such systems can enter routine practice, including prospective validation, regulatory review and the resolution of conversational reliability issues. But the trajectory is clear. The microscope, long the pathologist’s solitary instrument, is gaining a conversational partner, and that partnership may reshape how cancer is diagnosed, taught and understood.

Subject of Research: A multimodal generative AI assistant for whole-slide computational pathology across cancer types

Article Title: SlideChat is a multimodal generative artificial intelligence assistant for whole-slide computational pathology across cancer types

Article References: Chen, Y., Ma, C., Li, Q., Yan, F., Chen, Y., Li, T., Ye, J., Hu, M., Lin, Y., Li, Y., Wang, G., Xu, H., Dong, H., Wang, X., Xu, X., Zhou, Y., Zhu, X., Yang, S., Wang, X., … Ji, Y. (2026). SlideChat is a multimodal generative artificial intelligence assistant for whole-slide computational pathology across cancer types. Nature Cancer. https://doi.org/10.1038/s43018-026-01220-4

Image Credits: AI Generated

DOI: 10.1038/s43018-026-01220-4

Keywords: SlideChat, computational pathology, whole-slide imaging, multimodal AI, large language models, cancer diagnosis, digital pathology, instruction tuning, Nature Cancer, report generation, pathology AI, deep learning