The review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses, or PRISMA 2020, the international standard for ensuring that evidence syntheses are conducted and reported without cherry-picking. The researchers searched six major electronic databases, PubMed/MEDLINE, Scopus, Web of Science, Embase, ERIC, and the Cochrane Library, for English-language records published between January 2019 and February 2026, combining search terms covering artificial intelligence, generative AI, machine learning, assessment, evaluation, and medical education. Two reviewers independently screened titles, abstracts, and full texts against pre-specified PICOS criteria, a structured framework that defines the population, interventions, comparators, outcomes, and study designs eligible for inclusion. The screening funnel was demanding: after deduplication, 1,247 records were reviewed, 187 full-text articles were assessed for eligibility, and only 62 studies ultimately met the inclusion criteria. That attrition alone signals how much of the current literature is preliminary, duplicative, or methodologically thin.

To gauge the quality of what survived, the team applied the Medical Education Research Study Quality Instrument, known as MERSQI, a validated appraisal tool that scores studies on design, sampling, data collection, and the validity of their outcome measures. The results were sobering. Most of the 62 included studies were small-scale feasibility or proof-of-concept reports, and few rose to the level of rigorous validation studies. In other words, the field is rich in demonstrations that AI tools can be made to work in a classroom or exam hall, but poor in evidence that they work well, consistently, and fairly across the settings where medical education actually happens. The authors judged the strength of the evidence descriptively and found it highly variable, a finding that should temper the enthusiasm of any administrator hoping to hand over grading to an algorithm tomorrow.

From the thematic synthesis, six major themes emerged, and together they sketch a remarkably complete map of the field. The first concerns AI-supported assessment tools and platforms, the software and models now being deployed to evaluate learners. The second maps AI use onto three established educational paradigms: Assessment of Learning, the summative exams that certify competence; Assessment for Learning, formative feedback that guides improvement; and Assessment as Learning, in which the act of assessment itself becomes a learning experience. The third theme addresses validity, reliability, and feasibility, the psychometric pillars on which any credible assessment must stand. The fourth catalogs medical-education-specific applications, including written examinations, objective structured clinical examinations, simulation-based assessments, and workplace-based assessments. The fifth confronts limitations head-on: algorithmic bias, transparency, equity, and data privacy. The sixth examines how the role of the medical educator is being redefined in an AI-augmented world.

The technical applications are genuinely impressive. In large-scale written examinations, machine learning systems can score open-ended responses, detect patterns in item performance, and flag problematic questions with a speed no human committee can match. In objective structured clinical examinations, the standardized, station-based assessments where students rotate through simulated clinical encounters, AI systems have been used to grade performance, sometimes analyzing video, audio, or text transcripts of student-patient interactions. Simulation-based assessments benefit from AI’s capacity to track a learner’s decisions in real time within virtual clinical environments, while workplace-based assessments, the evaluations of trainees performing with real patients, stand to gain from longitudinal competency tracking, in which algorithms aggregate performance data over months or years to reveal growth trajectories that a single supervisor’s snapshot would miss. The review found AI’s strongest showing precisely in these domains: written exams, OSCE grading, and long-term competency monitoring.

The advantages the review identifies cluster around three words: efficiency, individualization, and timeliness. AI can deliver feedback in seconds rather than weeks, which matters enormously in formative assessment, where the educational value of feedback decays rapidly with time. It can tailor feedback to individual learners at a scale impossible for faculty stretched across hundreds of students. And it can process volumes of assessment data that would otherwise go unanalyzed, surfacing weaknesses in curricula and in test design alike. For institutions running high-volume examinations, the economic and logistical appeal is obvious. But the review is equally clear about the ceiling: evidence that AI improves the validity or reliability of assessment is limited to narrowly bounded tasks. An algorithm may grade a multiple-choice exam flawlessly and still fail catastrophically when asked to judge clinical reasoning, professionalism, or the subtle communication skills that separate a competent physician from a dangerous one.

The limitations theme reads as a warning label for the entire enterprise. Algorithmic bias, the tendency of models trained on unrepresentative data to systematically disadvantage certain groups, is a direct threat to equity in a profession that must serve diverse populations. Transparency is another persistent concern: many AI systems operate as opaque black boxes, offering scores without explanations, which is untenable in high-stakes contexts where learners have a right to understand and contest their evaluations. Data privacy looms equally large, since medical education assessments capture sensitive recordings and performance data from students and standardized patients. The review emphasizes that these are not hypothetical worries but structural features of current technology that must be addressed through governance before deployment in consequential decisions.

Perhaps the most consequential conclusion is the one about human judgment. The authors position AI explicitly as a complement, not a substitute, for expert human evaluation. Medical educators, they argue, are not being automated out of existence; their role is evolving, shifting from the mechanical work of scoring toward the interpretive, ethical, and mentoring functions that no current system can perform. The review calls on educators, institutions, and regulators to develop validation frameworks, governance structures, and AI literacy programs before AI tools are safely deployed in high-stakes assessments. That tripartite agenda, validate, govern, educate the educators, is the review’s practical takeaway, and it reframes the question institutions should be asking. Not whether AI can grade, but whether it has been proven to grade fairly, accurately, and accountably in their specific context.

For a field moving faster than its evidence base, this review arrives at exactly the right moment. It confirms that AI’s promise in medical education assessment is real but bounded, that the literature is expanding rapidly yet remains dominated by feasibility studies rather than validation research, and that the ethical architecture needed to deploy these tools responsibly is still being built. The image of a future in which algorithms certify physicians is not here, and the evidence suggests it should not arrive without rigorous psychometric proof, transparent governance, and a firm commitment to keeping professional human judgment at the center of decisions that ultimately protect patients. What the next decade of research must deliver is what this review found scarce: high-quality validation studies demonstrating that AI assessment is not merely efficient, but trustworthy.

Subject of Research: Applications of artificial intelligence in medical education assessment

Article Title: Applications, advantages, and limitations of artificial intelligence in medical education assessment: a systematic review

Article References: Polat, M., & Karadag, E. (2026). Applications, advantages, and limitations of artificial intelligence in medical education assessment: a systematic review. BMC Medical Education. https://doi.org/10.1186/s12909-026-10545-8

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10545-8

Keywords: artificial intelligence, medical education, assessment, systematic review, OSCE, generative AI, machine learning, competency-based assessment, algorithmic bias, validity, reliability, BMC Medical Education

The study, published in BMC Nursing, was led by Xiaobei Ma and Xueyan Fan, who contributed equally, together with colleagues including Fengli Gao, Xin Liu, Yi Duan, and corresponding author Zhifeng Gao. Rather than relying on a generic nursing checklist, the team constructed a framework tailored specifically to the role of the nurse anesthetist, a profession in which errors in airway management, drug dosing, or vigilance during the post-anesthesia care phase can have immediate and serious consequences. The result is a hierarchical instrument comprising five domains, twenty second-level indicators, and one hundred individual scored elements, each anchored to explicit behavioral criteria.

Building such an instrument required a multi-stage methodology that blends structured expert consensus with quantitative weighting. The researchers began with a literature review and semi-structured interviews with experts, then ran a two-round Delphi consultation, a technique in which a panel of specialists independently rates proposed items and receives feedback between rounds until agreement stabilizes. To determine how much weight each domain should carry in the final score, they applied the analytic hierarchy process, a mathematical method that converts pairwise expert judgments into a consistent set of weights, checking that the judgments did not contradict one another through a consistency ratio threshold.

The preliminary measurement evaluation focused on content validity and reliability. Experts rated the relevance of each item, yielding a scale-level content validity index average of 0.95, very close to the perfect score of 1.0, while item-level modified kappa values, which correct expert agreement for chance, ranged from 0.831 to 1.000. To estimate inter-rater reliability, five nurses were independently scored by two prespecified evaluators at program entry; the total-score intraclass correlation coefficient using the two-way mixed-effects absolute-agreement single-measure definition, ICC(A,1), was 0.889, with a 95 percent confidence interval of 0.388 to 0.988. The authors are candid that the narrow confidence interval’s lower bound reflects the tiny reliability subsample, making this preliminary, low-precision evidence rather than comprehensive validation.

The longitudinal component is where the study becomes genuinely distinctive. The team retrospectively analyzed a census of all eligible nurses entering the training program between August 2016 and August 2025, defining T0 as program entry after pre-employment preparation and T1 through T6 as the completion of each of the first six months. Of 77 screened records, 68, or 88.3 percent, had complete T0 through T6 data and entered the analysis; five nurses resigned during the period and four had at least two consecutive missing monthly assessments. All analyses were performed in R version 4.5.2, with repeated-measures analysis of variance serving as the primary within-evaluator test of change over time.

The trajectory of improvement was striking. Mean self-assessment scores climbed from 35.95 at entry to 96.83 at month six, while faculty ratings rose from 29.20 to 95.32. After applying the Greenhouse-Geisser correction, which guards against violations of the statistical assumption of sphericity in repeated-measures designs, the time effects were highly significant for both perspectives, with P values below 0.001 and partial eta-squared values of 0.977 for self-ratings and 0.984 for faculty ratings, meaning that time explained nearly all of the variance in scores. In practical terms, new nurse anesthetists moved from roughly a third of the maximum score to near-ceiling performance within six months under this framework.

Perhaps the most psychologically interesting finding concerns the gap between self-perception and external judgment. At every single time point, self-ratings exceeded faculty ratings, but the size of that gap changed systematically. A participant-level mixed-effects model, which accounts for repeated observations nested within individuals, estimated the model-based difference between evaluator trajectories as largest at month two, at 9.51 points, and shrinking to just 1.50 points by month six. The time-by-evaluator interaction was significant in a likelihood ratio test yielding a chi-square statistic of 110.67 with six degrees of freedom and P below 0.001. The pattern suggests that new nurses begin with a degree of overconfidence relative to their supervisors’ assessments, then calibrate their self-appraisal as real clinical experience accumulates.

The framework also showed meaningful concurrent associations with established assessment tools. At month three, self-ratings correlated with written examination scores at a Spearman coefficient of 0.815 and faculty ratings at 0.582, while the corresponding correlations with objective structured clinical examination scores, the standardized practical tests widely used in medical education, were 0.580 and 0.558. By month six these associations remained positive, at 0.752 and 0.532 for the written examination and 0.689 and 0.340 for the OSCE. Notably, at program entry the correlations with the written examination were negligible, which makes sense: before any anesthesia-specific training, general knowledge tests say little about how a nurse will perform in the operating room, so the framework’s scores capture something the written test does not.

The authors also report an exploratory analysis in a separate prespecified five-person subset examining whether every individual improved from T0 to T6. All five showed positive change, but an exact two-sided sign test produced a P value of 0.0625, just shy of the conventional 0.05 threshold, a mathematical inevitability when only five participants are tested, since the smallest achievable two-sided P value for a sign test with five observations is exactly 0.0625. The team treats this honestly as suggestive rather than confirmatory, and the supplementary materials go further, documenting acceptability questionnaires completed by all 68 respondents, sensitivity analyses, and exploratory safety surveillance, an unusually transparent level of statistical disclosure for a single-centre education study.

What the study does not claim is perhaps as important as what it does. The authors explicitly state that their results provide preliminary, low-precision measurement evidence rather than comprehensive validation or causal proof that the training program itself drove the score increases, and they call for prospective multicenter evaluation before the framework is used for any high-stakes decisions such as hiring, promotion, or certification. Still, the implications are considerable. With anesthesia teams worldwide facing persistent staffing pressures, a validated, behaviorally anchored, month-by-month competency map could help educators target support precisely where new nurses lag, and the finding that self-assessment converges with faculty judgment over time offers a measurable signal of professional maturation. If larger multicenter studies replicate these trajectories, the humble scoring rubric developed in a Beijing hospital could become a template for how surgical specialties everywhere turn the anxious first months of a new clinician’s career into a quantified, coachable journey.

Subject of Research: Development and longitudinal evaluation of a competency assessment framework for newly recruited nurse anesthetists

Article Title: Development, preliminary measurement evaluation, and longitudinal application of a competency assessment framework for newly recruited nurse anesthetists

Article References: Ma, X., Fan, X., Gao, F., Liu, X., Duan, Y., & Gao, Z. (2026). Development, preliminary measurement evaluation, and longitudinal application of a competency assessment framework for newly recruited nurse anesthetists. BMC Nursing. https://doi.org/10.1186/s12912-026-05424-y

Image Credits: AI Generated

DOI: 10.1186/s12912-026-05424-y

Keywords: nurse anesthetists, competency framework, Delphi method, longitudinal assessment, mixed-effects model, content validity, inter-rater reliability, OSCE, nursing education, self-assessment, analytic hierarchy process, BMC Nursing

The study focused on the Objective Structured Clinical Examination, or OSCE, a cornerstone of medical training in which trainees rotate through stations and interact with standardized patients — trained actors portraying realistic clinical scenarios. In this case, 126 first-year internal medicine residents at NYU completed six communication-focused OSCE cases between 2019 and 2023, with 82 assessed in person and 54 assessed virtually. In each case, the standardized patients rated resident performance across three core communication domains: information gathering, relationship development, and patient education. Crucially, the ratings used a behaviorally anchored scale — “not done,” “partially done,” or “well done” — meaning each rating level corresponded to observable clinical behaviors rather than a vague global impression. This design choice mattered enormously for the analysis that followed.

Most studies comparing virtual and in-person OSCEs have stopped at overall performance scores, concluding broadly that the two formats produce similar results. The NYU team, led by Christine P. Beltran and Colleen Gillespie, argued that such aggregate comparisons can mask important differences. Two exams might yield identical average scores while individual checklist items behave differently across formats — for example, if a particular behavior is easier to demonstrate, or easier to notice, over video. To detect these hidden discrepancies, the researchers turned to the graded response model, a form of item response theory that estimates what they call item-level thresholds. Each threshold represents the amount of underlying communication proficiency a resident needs to move from one rating category to the next — from “not done” to “partially done,” or from “partially done” to “well done.” If a threshold is lower in one modality, that means residents need less actual skill to earn the same rating there.

The technical machinery is worth unpacking, because it represents a meaningful methodological advance for medical education assessment. In a graded response model, every checklist item is characterized by a discrimination parameter, describing how well the item distinguishes between residents of different ability levels, and a set of threshold parameters, marking the points on the latent proficiency continuum where the probability of achieving each successive rating crosses 50 percent. By fitting the model separately to virtual and in-person data and then comparing threshold estimates with Wald tests — a standard statistical procedure for testing whether estimated parameters differ significantly — the researchers could ask, item by item, whether “partially done” or “well done” meant the same thing in both settings. This stands in contrast to traditional approaches that compare mean scores, which assume that identical numbers imply identical underlying constructs.

The headline results were, in most respects, reassuring. Most residents demonstrated sufficient skill to receive “partially done” or “well done” ratings on the majority of communication items, regardless of format, and overall communication performance appeared broadly similar across modalities. For program directors worried that the pandemic-era pivot to virtual assessment diluted their exams, the message is largely encouraging: the global picture of resident communication competence looks comparable whether the encounter happens in an exam room or on a screen. But the item-level analysis told a subtler story. One specific behavior — “using words the patient understood and explaining jargon” — showed a statistically significant difference between modalities. Residents required a lower level of underlying proficiency to receive a “partially done” rating on that item in virtual encounters compared with in-person ones, with a Wald test statistic of z = 3.92 and an adjusted p-value of 0.001.

Why would explaining medical jargon in plain language be easier to credit over video? The authors are careful to lay out several non-exclusive explanations. It may reflect genuine modality-related variation in the skill itself: residents might consciously simplify their language when they cannot rely on physical props, printed materials, or the full repertoire of in-person cues, making plain speech more prevalent on screen. Alternatively, the difference may lie in rater scoring — standardized patients evaluating a video encounter may attend differently to language, or may be more generous when communication is constrained by technology. Both mechanisms could produce the same statistical signature, and the study design cannot fully disentangle them. What the finding does establish, however, is that at least one checklist item does not function equivalently across formats, which means raw scores from virtual and in-person exams are not perfectly interchangeable, even when their averages look the same.

The broader context makes the study timely. Telehealth has evolved from a pandemic stopgap into a permanent fixture of American medicine, and organizations such as the Association of American Medical Colleges have published telehealth competency frameworks urging training programs to teach and assess virtual care skills deliberately. Previous studies of virtual OSCEs — spanning physical medicine and rehabilitation residencies, pediatric high-stakes exams, dental education, and systematic reviews of implementation across the health professions — have generally reported acceptable feasibility and comparable overall performance, but few have probed whether individual rating categories mean the same thing across formats. Earlier psychometric work had already applied item response theory to OSCE data to explore rater effects and item discrimination, and researchers had documented differences in how satisfied patients feel about physician communication during telemedicine visits. The NYU study ties these threads together, using threshold analysis as a diagnostic for format comparability.

For the assessment community, the practical implications are concrete. Programs running mixed-modality OSCEs — increasingly common as telehealth training becomes standard — can use graded response modeling as a routine quality-assurance step, flagging items whose thresholds drift between formats before those items contaminate pass-fail decisions. The one flagged behavior in this study, avoiding jargon, is itself a natural candidate for targeted curriculum revision: if residents earn partial credit for plain language more easily online, virtual encounters may need explicit prompts or rubric anchors that distinguish genuine skill from modality-driven simplification. The authors also note that their sample, drawn from a single internal medicine program across six cases, limits generalizability, and that the modest virtual cohort of 54 residents constrains statistical power. The project, funded through the AAMC Competency-Based Education in Telehealth Challenge Grant, was reviewed by NYU’s institutional process and classified as educational quality improvement using fully anonymized, de-identified data, with the informed consent requirement waived accordingly. The work was previously presented as a poster at the 2025 Society of General Internal Medicine Annual Conference.

Perhaps the most durable contribution of the study is conceptual: it reframes what “equivalence” means in clinical skills assessment. Averages alone, the authors argue, can lull educators into false confidence, because two formats can converge on the same totals while individual behaviors are weighted differently by raters or expressed differently by trainees. Threshold analysis offers a way to interrogate whether an observed rating reflects the same level of underlying communication proficiency across settings — essentially asking not just whether residents score the same, but whether the same score certifies the same competence. As telemedicine cements its place in routine care, ensuring that the exams certifying tomorrow’s physicians measure what they claim, in every modality, is no longer a technical nicety. It is a patient-safety question, and this study provides a template for answering it one checklist item at a time.

Subject of Research: Item response theory analysis of internal medicine residents' OSCE communication skill ratings in virtual versus in-person examinations

Article Title: When “Well-Done” Is Not the Same: Item Response Theory Analysis of Medicine Residents’ OSCE Communication Ratings Across Modalities

Article References: Beltran, C. P., Nallamaddi, S., Wilhite, J. A., Hardowar, K., Hanley, K., Altshuler, L., Zabar, S. R., & Gillespie, C. (2026). When “Well-Done” Is Not the Same: Item Response Theory Analysis of Medicine Residents’ OSCE Communication Ratings Across Modalities. Journal of General Internal Medicine. https://doi.org/10.1007/s11606-026-10798-5

Image Credits: AI Generated

DOI: 10.1007/s11606-026-10798-5

Keywords: OSCE, item response theory, communication skills, medical residents, telehealth, virtual assessment, standardized patients, internal medicine, graded response model, medical education, assessment comparability, patient communication