New AI Assistant Reads Whole Pathology Slides and Answers Clinician Questions Across 31 Cancer Types

Pathology is undergoing a quiet revolution, and a new study published in Nature Cancer may accelerate it dramatically. A research team led by Ying Chen, Chenglong Ma and colleagues at the Shanghai Artificial Intelligence Laboratory, in collaboration with clinical partners including Shanghai General Hospital, the Eastern Hepatobiliary Surgery Hospital and Stanford University School of Medicine, has unveiled SlideChat, a multimodal generative artificial intelligence assistant designed to interpret gigapixel-scale whole-slide images, the enormous digital scans that form the backbone of modern cancer diagnosis. Unlike earlier AI systems that analyze only small crops of tissue, SlideChat is built to reason over the entire slide, answering clinical questions and generating diagnostic-style reports across 31 cancer types. Expert pathologists who reviewed the system’s answers rated them as accurate, clinically relevant and, in several dimensions, superior to those of every competing model tested.

The technical hurdle the team set out to overcome is deceptively simple to state and extraordinarily difficult to solve. A single whole-slide image can contain billions of pixels, far exceeding the input capacity of conventional vision-language models. Most existing multimodal AI assistants in pathology, including widely cited systems adapted from general-purpose models, operate at the patch level, examining isolated regions of tissue. That approach works for tasks such as detecting mitotic figures or classifying small lesions, but it fails when a diagnosis depends on global tissue architecture. Determining tumor staging, for example, requires appreciating whether invasive cells have breached anatomical boundaries visible only when the entire slide is considered. The researchers demonstrated this limitation directly: in a challenging bladder cancer case, patch-level models misidentified the invasive stage, likely because they could not integrate spatial context across the slide, while SlideChat correctly analyzed the global architecture and reported the accurate pT3 stage.

SlideChat’s architecture combines three essential ingredients. First, it integrates a patch-level pathology encoder, which captures fine-grained cellular and subcellular detail, with a slide-level pathology encoder that aggregates information across the full specimen into a compact representation. Second, it connects these visual encoders to a pretrained large language model, allowing the system to translate visual evidence into fluent natural-language answers. Third, and perhaps most importantly, it is trained on SlideInstruction, a new dataset of 274,233 multimodal instruction samples that pair whole-slide images with diagnostic reports and question-answer pairs. The instruction-tuning paradigm, which proved transformative in general-purpose chatbots, teaches the model not merely to classify images but to interpret complex, realistic clinical queries and respond in the structured language that pathologists actually use.

Building the training data required considerable curation. The team drew on publicly available resources including The Cancer Genome Atlas and the Clinical Proteomic Tumor Analysis Consortium, whose whole-slide images and associated pathology reports are accessible through the National Institutes of Health data commons. Additional data came from the BCNB breast cancer cohort and the HISTAI dataset, while a retrospectively collected hepatobiliary cohort, gathered under approval from the Eastern Hepatobiliary Surgery Hospital, provided controlled-access material. Raw reports were parsed and cleaned, concise captions were extracted to teach image-language alignment, and instruction-style question-answer pairs were generated from the reports and labels. A rigorous quality-control pipeline then applied large language model filtering and pathologist verification to ensure that questions genuinely could not be answered without visual inspection of the slide, guarding against shortcuts that would let the model succeed through text alone.

The evaluation was among the most comprehensive ever assembled for slide-level pathology AI. The team created SlideBench, a benchmark spanning five cohorts and 31 cancer types, comprising 8,836 closed-ended questions, 129 open-ended questions and 3,149 whole-slide diagnostic reports. On closed-ended questions, SlideChat outperformed leading baseline models by 19.1 percentage points in accuracy, a margin the authors validated with two-sided Wilcoxon signed-rank tests and Benjamini-Hochberg correction across 1,000 bootstrap replicates. On report generation, judged by the Metric for Evaluation of Translation with Explicit Ordering, a standard automatic measure of textual overlap with reference documents, SlideChat exceeded the best baselines by 7.7 points. For open-ended questions, expert pathologists scored SlideChat highest across five evaluation dimensions, noting that its answers were the most diagnostically accurate and case-specific. In one representative differential-diagnosis case, SlideChat integrated key morphological features to support a refined melanoma diagnosis, while a leading general-purpose model produced a broad but weakly prioritized differential and competing medical models returned generic, weakly reasoned answers.

The comparison with general-purpose and specialist baselines was particularly revealing. Models such as GPT-4o, LLaVA-Med, Quilt-LLaVA and dedicated slide-level systems including HistoGPT and PRISM all trailed SlideChat on most tasks. In report generation case studies, SlideChat consistently produced accurate, structured reports that captured tumor type, anatomical location, invasion status, lymphovascular features, TNM staging and relevant histological details, closely matching reference clinical reports. HistoGPT, by contrast, sometimes produced anatomically inconsistent or diagnostically mismatched descriptions, while PRISM returned brief diagnoses lacking contextual or morphological explanation. In renal and lung carcinoma examples from the CPTAC cohort, SlideChat correctly identified tumor subtype, Fuhrman grade, pT2a staging, the absence of vascular and perineural invasion and negative margins, and even contextualized incidental benign findings such as emphysematous changes in the lung without drifting into irrelevant speculation.

The study also probes how the model thinks, offering an unusually transparent window into its behavior. Question-guided attention heatmaps show that SlideChat reallocates its visual focus depending on the query: when asked about cytology in an adenocarcinoma case, attention concentrates on nuclear detail, while a question about differentiation shifts attention to glandular architecture; in a breast carcinoma case, the model attends to fibrous stroma when asked about stromal reaction but to tumor nests when asked about cellular arrangement. Ablation experiments confirmed that both the slide-level encoder and the two-stage training procedure are indispensable to performance. Sensitivity tests in which Gaussian noise was injected into patches with high or low attention scores showed a monotonic drop in accuracy when meaningful regions were corrupted, indicating that the model relies on genuine visual content rather than artifacts or dataset shortcuts. When adapted patch-level foundation models were given whole-slide inputs through strategies such as patch voting, thumbnail downsampling or joint multi-patch input, SlideChat still outperformed them, often by statistically decisive margins.

Not everything in the evaluation was flattering, and the authors are candid about the model’s weaknesses. In multiturn conversational settings, SlideChat exhibited cross-turn inconsistency in some cases, at one point initially identifying a node-negative tumor and later contradicting that finding during prognostic assessment. The model also occasionally produced self-contradictory hallucinations, describing a tumor as confined to the epithelium while simultaneously reporting vascular invasion. These failure modes mirror the reliability challenges that afflict large language models generally, and their documentation in a rigorous pathology benchmark provides a concrete target for future work. The researchers also note that performance did not correlate strongly with the number of training samples per organ, suggesting that data scale alone does not determine competence and that data quality and task diversity matter substantially.

The potential applications extend beyond diagnosis. The authors highlight medical education as a promising use case, since a system that can answer natural-language questions about whole slides could serve as an interactive tutor for pathology trainees. Clinical decision support is another frontier: by integrating slide-level reasoning with conversational interaction, the assistant could help oncologists and pathologists rapidly extract staging, grading and prognostic information from complex specimens. Notably, SlideChat achieves performance competitive with specialized pathology foundation models such as CONCH, TITON-class slide encoders, CHIEF and Prov-GigaPath on breast cancer classification tasks involving tumor status and hormone receptor and HER2 status, despite being a general-purpose assistant rather than a task-specific classifier. The team has released the training and evaluation data on Hugging Face, the model source code on GitHub and the model weights publicly, a level of openness that could seed a new generation of slide-level pathology AI research.

The work arrives at a moment when computational pathology is transitioning from narrow, single-task algorithms to foundation models and generative assistants, and it addresses what many in the field identify as the central bottleneck: the gap between patch-level perception and slide-level clinical reasoning. By demonstrating that a single model can answer closed-ended questions, engage in open-ended diagnostic dialogue and generate expert-quality reports across a broad range of cancers, and by subjecting that model to pathologist review, extensive ablations and honest failure analysis, the study sets a new reference point for what clinically useful pathology AI looks like. Substantial work remains before such systems can enter routine practice, including prospective validation, regulatory review and the resolution of conversational reliability issues. But the trajectory is clear. The microscope, long the pathologist’s solitary instrument, is gaining a conversational partner, and that partnership may reshape how cancer is diagnosed, taught and understood.

Subject of Research: A multimodal generative AI assistant for whole-slide computational pathology across cancer types

Article Title: SlideChat is a multimodal generative artificial intelligence assistant for whole-slide computational pathology across cancer types

Article References: Chen, Y., Ma, C., Li, Q., Yan, F., Chen, Y., Li, T., Ye, J., Hu, M., Lin, Y., Li, Y., Wang, G., Xu, H., Dong, H., Wang, X., Xu, X., Zhou, Y., Zhu, X., Yang, S., Wang, X., … Ji, Y. (2026). SlideChat is a multimodal generative artificial intelligence assistant for whole-slide computational pathology across cancer types. Nature Cancer. https://doi.org/10.1038/s43018-026-01220-4

Image Credits: AI Generated

DOI: 10.1038/s43018-026-01220-4

Keywords: SlideChat, computational pathology, whole-slide imaging, multimodal AI, large language models, cancer diagnosis, digital pathology, instruction tuning, Nature Cancer, report generation, pathology AI, deep learning