AI Can Grade Future Doctors, But a Landmark Review Says It Cannot Replace Them

Artificial intelligence has swept into medicine with dazzling speed, reading scans, drafting clinical notes, and even predicting patient deterioration before symptoms appear. But one of the most consequential places where AI is now being tested is quieter and arguably more high-stakes than any radiology suite: the examination hall, where the competence of future doctors is decided. A new systematic review published in BMC Medical Education by Murat Polat of Anadolu University and Engin Karadag of Akdeniz University offers the most methodically transparent synthesis to date of how AI is being used to assess medical students and trainees, and its verdict is a study in careful optimism. AI, the authors conclude, can genuinely make assessment faster, more individualized, and more timely. What it cannot yet do, on the strength of the available evidence, is prove that it measures clinical competence more validly or reliably than a seasoned human examiner, particularly when the stakes involve patient safety.

The review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses, or PRISMA 2020, the international standard for ensuring that evidence syntheses are conducted and reported without cherry-picking. The researchers searched six major electronic databases, PubMed/MEDLINE, Scopus, Web of Science, Embase, ERIC, and the Cochrane Library, for English-language records published between January 2019 and February 2026, combining search terms covering artificial intelligence, generative AI, machine learning, assessment, evaluation, and medical education. Two reviewers independently screened titles, abstracts, and full texts against pre-specified PICOS criteria, a structured framework that defines the population, interventions, comparators, outcomes, and study designs eligible for inclusion. The screening funnel was demanding: after deduplication, 1,247 records were reviewed, 187 full-text articles were assessed for eligibility, and only 62 studies ultimately met the inclusion criteria. That attrition alone signals how much of the current literature is preliminary, duplicative, or methodologically thin.

To gauge the quality of what survived, the team applied the Medical Education Research Study Quality Instrument, known as MERSQI, a validated appraisal tool that scores studies on design, sampling, data collection, and the validity of their outcome measures. The results were sobering. Most of the 62 included studies were small-scale feasibility or proof-of-concept reports, and few rose to the level of rigorous validation studies. In other words, the field is rich in demonstrations that AI tools can be made to work in a classroom or exam hall, but poor in evidence that they work well, consistently, and fairly across the settings where medical education actually happens. The authors judged the strength of the evidence descriptively and found it highly variable, a finding that should temper the enthusiasm of any administrator hoping to hand over grading to an algorithm tomorrow.

From the thematic synthesis, six major themes emerged, and together they sketch a remarkably complete map of the field. The first concerns AI-supported assessment tools and platforms, the software and models now being deployed to evaluate learners. The second maps AI use onto three established educational paradigms: Assessment of Learning, the summative exams that certify competence; Assessment for Learning, formative feedback that guides improvement; and Assessment as Learning, in which the act of assessment itself becomes a learning experience. The third theme addresses validity, reliability, and feasibility, the psychometric pillars on which any credible assessment must stand. The fourth catalogs medical-education-specific applications, including written examinations, objective structured clinical examinations, simulation-based assessments, and workplace-based assessments. The fifth confronts limitations head-on: algorithmic bias, transparency, equity, and data privacy. The sixth examines how the role of the medical educator is being redefined in an AI-augmented world.

The technical applications are genuinely impressive. In large-scale written examinations, machine learning systems can score open-ended responses, detect patterns in item performance, and flag problematic questions with a speed no human committee can match. In objective structured clinical examinations, the standardized, station-based assessments where students rotate through simulated clinical encounters, AI systems have been used to grade performance, sometimes analyzing video, audio, or text transcripts of student-patient interactions. Simulation-based assessments benefit from AI’s capacity to track a learner’s decisions in real time within virtual clinical environments, while workplace-based assessments, the evaluations of trainees performing with real patients, stand to gain from longitudinal competency tracking, in which algorithms aggregate performance data over months or years to reveal growth trajectories that a single supervisor’s snapshot would miss. The review found AI’s strongest showing precisely in these domains: written exams, OSCE grading, and long-term competency monitoring.

The advantages the review identifies cluster around three words: efficiency, individualization, and timeliness. AI can deliver feedback in seconds rather than weeks, which matters enormously in formative assessment, where the educational value of feedback decays rapidly with time. It can tailor feedback to individual learners at a scale impossible for faculty stretched across hundreds of students. And it can process volumes of assessment data that would otherwise go unanalyzed, surfacing weaknesses in curricula and in test design alike. For institutions running high-volume examinations, the economic and logistical appeal is obvious. But the review is equally clear about the ceiling: evidence that AI improves the validity or reliability of assessment is limited to narrowly bounded tasks. An algorithm may grade a multiple-choice exam flawlessly and still fail catastrophically when asked to judge clinical reasoning, professionalism, or the subtle communication skills that separate a competent physician from a dangerous one.

The limitations theme reads as a warning label for the entire enterprise. Algorithmic bias, the tendency of models trained on unrepresentative data to systematically disadvantage certain groups, is a direct threat to equity in a profession that must serve diverse populations. Transparency is another persistent concern: many AI systems operate as opaque black boxes, offering scores without explanations, which is untenable in high-stakes contexts where learners have a right to understand and contest their evaluations. Data privacy looms equally large, since medical education assessments capture sensitive recordings and performance data from students and standardized patients. The review emphasizes that these are not hypothetical worries but structural features of current technology that must be addressed through governance before deployment in consequential decisions.

Perhaps the most consequential conclusion is the one about human judgment. The authors position AI explicitly as a complement, not a substitute, for expert human evaluation. Medical educators, they argue, are not being automated out of existence; their role is evolving, shifting from the mechanical work of scoring toward the interpretive, ethical, and mentoring functions that no current system can perform. The review calls on educators, institutions, and regulators to develop validation frameworks, governance structures, and AI literacy programs before AI tools are safely deployed in high-stakes assessments. That tripartite agenda, validate, govern, educate the educators, is the review’s practical takeaway, and it reframes the question institutions should be asking. Not whether AI can grade, but whether it has been proven to grade fairly, accurately, and accountably in their specific context.

For a field moving faster than its evidence base, this review arrives at exactly the right moment. It confirms that AI’s promise in medical education assessment is real but bounded, that the literature is expanding rapidly yet remains dominated by feasibility studies rather than validation research, and that the ethical architecture needed to deploy these tools responsibly is still being built. The image of a future in which algorithms certify physicians is not here, and the evidence suggests it should not arrive without rigorous psychometric proof, transparent governance, and a firm commitment to keeping professional human judgment at the center of decisions that ultimately protect patients. What the next decade of research must deliver is what this review found scarce: high-quality validation studies demonstrating that AI assessment is not merely efficient, but trustworthy.

Subject of Research: Applications of artificial intelligence in medical education assessment

Article Title: Applications, advantages, and limitations of artificial intelligence in medical education assessment: a systematic review

Article References: Polat, M., & Karadag, E. (2026). Applications, advantages, and limitations of artificial intelligence in medical education assessment: a systematic review. BMC Medical Education. https://doi.org/10.1186/s12909-026-10545-8

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10545-8

Keywords: artificial intelligence, medical education, assessment, systematic review, OSCE, generative AI, machine learning, competency-based assessment, algorithmic bias, validity, reliability, BMC Medical Education