AI Practice Patients Fail to Calm Medical Students Before First Standardized Patient Encounter

Medical educators at the University of Amsterdam have put one of the most seductive promises of generative artificial intelligence to a controlled test: can a chatbot playing a patient help first-year medical students walk into their first simulated consultation calmer, better prepared, and more confident? The answer, according to a quasi-experimental study of 383 students published in Frontiers in Education, is a sobering no — at least not in the form the researchers built. Yet the same study found that students who did try the AI patient liked it, and the authors argue the null result says more about how the tool was deployed than about the concept itself.

Communication skills training sits at the heart of every medical curriculum, and for good reason: effective patient-provider communication is fundamental to high-quality care. In the pre-clinical years, students rarely see real patients, so they rehearse with standardized patients — trained actors who portray specific cases. The problem is logistical and psychological. Standardized patient sessions are expensive and scarce, and for novices they can be genuinely stressful. Students are observed by peers and a trainer, often on camera, and a simulated consultation demands that they simultaneously retrieve biomedical knowledge, structure a conversation, and deploy a diverse toolkit of listening skills. Cognitive load theory suggests this combination can overwhelm working memory, leaving little capacity for actual learning.

Large language models appeared to offer an escape hatch. A generative AI ‘patient’ can hold open-ended, unscripted conversations, allowing unlimited private rehearsal in a low-stakes environment. Earlier studies showed such tools are feasible and that students enjoy them, but most relied on before-and-after designs without control groups, leaving the causal question open. The Amsterdam team — medical educators, communication trainers, a physician and a student assistant — set out to answer it properly.

Their tool was engineered with care. Built on the university’s secure AI platform running GPT-4o, it used a retrieval-augmented generation (RAG) framework: rather than letting the model improvise, the system pulled relevant details from a structured case file containing the patient’s demographics, medical history, medication use, social situation and symptoms. A behavioral prompt instructed the AI to behave like a standardized patient, for example by not volunteering more information than the student asked for. The case was designed to match the type and difficulty of the one students would face in class, and the prompt was refined iteratively over three months with pilot testing on 18 students.

The experiment exploited the natural split in the class schedule. Students taking their first standardized patient class in the first week of the three-week block formed the control group (182 students), while those scheduled in the following two weeks formed the intervention group (201 students) and were offered voluntary access to the AI patient alongside the standard e-learning. After their class, students completed a 31-item questionnaire measuring four primary outcomes: nervousness, preparedness, self-efficacy and self-perceived competence, all on seven-point scales. The researchers used Mann–Whitney U tests with Benjamini-Hochberg correction for false discovery rate, and their primary analysis was by intention-to-treat, comparing everyone allocated to the intervention with everyone in the control group regardless of actual use.

The results were unambiguous in their nullity. Of 180 students who completed the questionnaire, 49 of the 96 in the intervention group had actually used the AI patient — and most of them, 35, had done so only once. In the intention-to-treat analysis, every composite effect size was near zero, with Cliff’s delta ranging from −0.05 to 0.04 and all 95 percent confidence intervals including zero. The per-protocol comparison of actual users against controls fared no better, with deltas between −0.12 and 0.07. Not one of the 19 individual questionnaire items showed a statistically significant difference after correction. Median nervousness scores were identical at 4.0 in both groups.

The authors are candid about the statistical caveats. The per-protocol comparison included only 49 users, well below the 70 per group their power calculation required, meaning the study could only have detected effects of Cliff’s delta 0.29 or larger — the upper end of a small effect approaching medium. Smaller genuine benefits may have slipped through. But contextual factors loom larger. The presence of fellow students, a trainer and a camera creates social evaluation pressure that may independently drive anxiety, overshadowing any calming effect of prior text-based practice. And there is a deeper modality mismatch: an asynchronous, keyboard-driven conversation simply does not rehearse the real-time turn-taking, nonverbal reading and emotional responsiveness a face-to-face encounter demands.

The qualitative data sharpen that interpretation. Among users, the most appreciated aspect was simply the chance to rehearse — 14 of 34 respondents valued having practiced before meeting the standardized patient. But the improvement requests converged on fidelity and feedback: seven students found the AI’s answers too short, repetitive or unvaried, four wanted a voice or a visible face rather than text, four wanted more varied cases, and three noted the absence of any feedback after the conversation. Practice without correction, the authors observe, is repetition but not deliberate practice. Even the reasons for non-use are instructive: of 19 non-users who explained themselves, six cited lack of time and six cited technical or login problems — access barriers, not rejection of the concept.

Perhaps the most unsettling finding concerns self-assessment. Both groups rated their competence strikingly high, at levels inconsistent with what lecturers observe in earlier cohorts. The authors point to a well-documented phenomenon: novice learners systematically overestimate their abilities, and self-assessed competence correlates poorly with observed performance across health professions education. Ceiling effects and social desirability bias likely inflated the scores further. This is precisely why the team argues that future evaluations of AI patients should use blinded observer ratings with validated communication scales rather than self-report — and why an AI tutor that provides automated feedback could help calibrate inflated self-perceptions.

The study’s conclusion is neither a eulogy nor an endorsement. A single, voluntary, text-based session with a generative AI patient did not measurably reduce nervousness or improve self-assessed competence before a first standardized patient encounter. But 71 percent of users considered the tool a valuable addition to standard preparation, 63 percent wanted to use it again, and the format itself raised no red flags. The authors’ prescription is concrete: mandatory minimum engagement, multiple patient cases, automated feedback, and higher-fidelity multimodal interaction with voice and avatars. Repeated, effortful practice is what drives learning, they note — and if better-designed AI patients can sustain that practice at scale, the benefits that failed to appear here may yet emerge. The broader potential, they add, extends well beyond history-taking, to shared decision-making, team meetings and clinical reasoning.

Subject of Research: Effect of a generative AI virtual patient on medical students' communication skills preparation

Article Title: Use of a GenAI-patient for medical communication training for undergraduate medical students—a quasi-experimental study

Article References: Jongkind, R., Pool, J., Verberne, E., Meinema, J., Geerlings, S., & Henselmans, I. (2026). Use of a GenAI-patient for medical communication training for undergraduate medical students—a quasi-experimental study. Frontiers in Education, 11, Article 1933319. https://doi.org/10.3389/feduc.2026.1933319

Image Credits: AI Generated

DOI: 10.3389/feduc.2026.1933319

Keywords: generative AI, medical education, virtual patient, standardized patients, communication skills, GPT-4o, retrieval-augmented generation, self-efficacy, cognitive load theory, quasi-experimental study, simulation-based learning, medical students