The review, conducted by Hui Zong, Jiao Wang, and colleagues including senior author Bairong Shen, set out to map how well large language models perform on Chinese medical licensing examinations and related specialty tests. The team followed the PRISMA-ScR guidelines, the accepted standard for scoping reviews, and searched PubMed and Web of Science on June 12, 2025, using search terms that combined references to large language models with Chinese medical examinations. To be included, a study had to evaluate at least one large language model on a national-level Chinese medical examination and report quantitative performance metrics. Two reviewers independently screened the candidate papers and extracted the data, a double-check designed to reduce the risk that a single reader’s judgment would skew the synthesis.

What emerged from that screening was a body of evidence both rapidly growing and strikingly uneven. The final analysis included 14 studies encompassing 51 separate evaluation records spread across 8 different types of Chinese medical examinations. Beyond the core licensing exam, the models were tested on specialty assessments spanning critical care medicine, radiation oncology, and ultrasound medicine, domains that demand not just recall of textbook facts but the kind of layered clinical reasoning that experienced specialists apply under pressure. The examination years covered by the studies stretched from 2017 to 2024, and the review noted a clear shift over time toward using more recent exams, a sign that researchers are increasingly trying to avoid the contamination problem that haunts this field: the possibility that a model has already memorized the very questions it is being tested on, because those questions circulated in the internet-scale text used during training.

Nine large language models have been put through this Chinese examination gauntlet so far. The roster reads like a tour of the global and domestic AI landscape: OpenAI’s GPT-3.5, GPT-4, and GPT-4o; Baidu’s ERNIE; DeepSeek-R1; Alibaba’s Qwen-72B; the Baichuan2 models at 7 billion and 13 billion parameters; and DISC-MedLLM, a model specifically tuned for Chinese medical dialogue. This diversity matters. Chinese medical examinations are administered in Chinese, rich in domain-specific terminology, clinical vignettes, and culturally specific standards of care, so models trained primarily on English text face an additional hurdle. The inclusion of homegrown Chinese models alongside their American counterparts reflects a growing recognition that medical AI benchmarks cannot simply be transplanted from one language to another and assumed to mean the same thing.

The headline results are dramatic. On the 2024 Chinese National Medical Licensing Examination, GPT-4o achieved the highest score recorded in the reviewed literature: 552 points, equivalent to 92.00 percent, while DeepSeek-R1 followed closely with 523 points, or 87.20 percent. Both scores comfortably surpass the passing threshold for the examination. To put those numbers in perspective, the licensing exam is designed so that a substantial fraction of human candidates fail it in any given sitting; a model answering more than nine out of ten questions correctly is performing at a level that would place it among the stronger human test-takers. That a general-purpose chatbot, built without any specific intention of practicing medicine, can clear a national physician licensing barrier is a milestone that even optimistic researchers did not widely anticipate when GPT-3.5 first appeared.

The trajectory of improvement is as important as the endpoint. In an exploratory comparison drawn from the included studies, GPT-4 achieved a higher pass rate than GPT-3.5, and the difference was statistically significant when assessed with Fisher’s exact test, a statistical method suited to comparing proportions in small samples. This generational leap suggests that the same scaling forces driving progress in general AI, larger models, better training data, and refined reasoning techniques, are translating directly into medical competence. DeepSeek-R1’s strong showing is particularly noteworthy because that model was explicitly engineered to work through problems step by step, generating extended chains of reasoning before committing to an answer. Its success hints that structured reasoning, rather than sheer memorization, may be the key ingredient that allows language models to handle multi-step clinical questions, where a single symptom must be traced through differential diagnoses, laboratory interpretation, and treatment selection.

Yet the review is emphatic that these eye-catching scores should be read with caution, because the studies beneath them are far from uniform. The authors assessed study quality using a 12-item checklist covering dataset characteristics, how each language model was set up, and the evaluation methods employed. What they found were persistent inconsistencies across the literature: different studies used different datasets, some models were evaluated under opaque conditions with little disclosure of how prompts were constructed or how answers were scored, and the evaluation criteria themselves varied from paper to paper. In practical terms, this means that a score of 87 percent reported in one study and a score of 60 percent reported in another may not be directly comparable, because the underlying tests, question formats, scoring rules, and even the number of times each model was queried may all differ. The review also flags the issue of model transparency: when a commercial model’s architecture and training data are proprietary, researchers cannot rule out that exam content leaked into training, inflating performance in ways that have nothing to do with genuine clinical reasoning.

These methodological gaps are more than academic quibbles. The stakes of medical AI evaluation are enormous, because licensing examinations are not just tests; they are the social contract that certifies who may be trusted with human lives. If a language model’s apparent competence rests on contaminated data or lenient scoring, the danger is not that the model will be overrated in a paper, but that hospitals, educators, and regulators might make real decisions based on inflated numbers. Conversely, if evaluation is too fragmented, genuinely capable systems may be underestimated, slowing beneficial applications such as clinical decision support, medical education tutoring, and assistance for physicians in underserved regions. The review’s authors argue that the field needs standardized benchmarks, shared protocols, and transparent reporting so that each new model’s medical performance can be measured against a common yardstick rather than a shifting patchwork of ad hoc tests.

The authors also point toward multimodality as the next frontier. Real medicine is not a text-only discipline: a radiologist reads images, a dermatologist inspects lesions, an ultrasonographer interprets moving scans in real time. The Chinese examinations reviewed here are largely text-based, which means the current results capture only one slice of what physician competence entails. Models that can jointly process clinical text, medical images, laboratory data, and even patient histories will be far better probes of genuine clinical ability, and the review identifies multimodal capabilities as a priority for future research. Combined with transparent evaluation and clinically meaningful benchmarks, this direction could transform these examinations from a publicity benchmark for AI companies into a rigorous scientific instrument for measuring how close machines are actually getting to the practice of medicine.

What the PLOS Digital Health review ultimately documents is a turning point. In the span of a few years, large language models have gone from failing medical exams to surpassing the passing threshold of one of the world’s most demanding licensing systems, with GPT-4o and DeepSeek-R1 leading a generation of models that improve with each release. The 14 studies and 51 evaluation records synthesized by Zong, Wang, Shen, and their colleagues tell a story of rapid ascent shadowed by unfinished methodological work: datasets that need standardizing, models that need transparency, and evaluations that need to reach beyond text into the multimodal reality of the clinic. The machines have passed the written test. Whether they can be trusted at the bedside is the question the next generation of research, and the next generation of benchmarks, must now answer.

Subject of Research: Performance of large language models on the Chinese National Medical Licensing Examination

Article Title: Performance of large language models in the Chinese National Medical Licensing Examinations and Beyond: Scoping review

Article References: Zong, H., Wang, J., Cha, J., Wu, R., Hu, S., Zhao, Y., & Shen, B. (2026). Performance of large language models in the Chinese National Medical Licensing Examinations and Beyond: Scoping review. PLOS Digital Health, 5(10), e0001773. https://doi.org/10.1371/journal.pdig.0001773

Image Credits: AI Generated

DOI: 10.1371/journal.pdig.0001773

Keywords: large language models, GPT-4o, DeepSeek-R1, medical licensing examination, China, medical education, scoping review, clinical reasoning, AI benchmarks, multimodal AI, PLOS Digital Health, model evaluation