Medical Students Cannot Tell AI-Written Exam Questions From Real Licensing Exam Items

Medical students sitting a tablet-based exam in Family Medicine could not reliably tell whether the multiple-choice questions in front of them had been written by a human examiner or generated by artificial intelligence, according to a cross-sectional study published in BMC Medical Education. The finding, drawn from 119 fifth-year undergraduate medical students at Saarland University in Germany, offers one of the most direct tests yet of whether large language models can produce assessment material indistinguishable from the real thing, and it suggests that the answer is essentially no, they cannot tell.

The study team, led by Philip Vogt and Fabian Dupont of the Department of Family Medicine at Saarland University, with additional affiliation to the Institute of General Practice and Family Medicine at Martin Luther University Halle-Wittenberg, set out to evaluate two things at once: whether students can distinguish between AI-generated multiple-choice questions and questions drawn from the National Licensing Exam, and whether the AI questions align with the course curriculum as perceived by the students themselves. The work was conducted within a real examination setting, not a laboratory simulation, which gives the results unusual ecological validity for medical education research.

Participants completed a tablet-based Family Medicine exam consisting of 60 multiple-choice questions in total. Half of the items, 30 questions, were generated by artificial intelligence using two prominent large language models, ChatGPT-4o and Gemini 1.5 Pro, working from digital learning materials. The other half, 30 questions, were authentic items from the National Licensing Exam. Before any question reached the students, an expert panel reviewed the AI-generated material and accepted 82 percent of the questions, making minor edits to retained items and removing those requiring major changes. This structured human-review process is central to how the researchers frame their conclusions about feasibility.

During the exam itself, students were asked after each item to indicate its source, AI or human-written, and to rate whether the question aligned with the course curriculum. Statistical analysis was performed using Jamovi 2.3.28.0. When the researchers compared correct attribution of AI-generated questions against correct attribution of National Licensing Exam questions, they found no significant difference between the two, with t(29.6) = -1.24, p = .225 for one comparison and t(29.6) = 1.18, p = .246 for the other. In plain terms, students were no better at spotting the machine-written questions than they were at correctly identifying the human-written ones.

The researchers also examined whether item difficulty played any role in recognition. Using Kendall’s tau-b correlation, they found no significant relationship between how difficult a question was and whether students correctly identified its source, with a p-value of .534. This is a technically important negative result, because it rules out the possibility that only the easiest or only the hardest AI questions gave themselves away. Recognition failure appears to be distributed across the difficulty spectrum rather than concentrated at either end.

Distractor analysis added another layer of technical scrutiny. Distractors are the incorrect answer options in a multiple-choice question, and their distribution across sources did not differ significantly when comparing ChatGPT, Google Gemini, and National Licensing Exam items, with χ²(2) = 2.61, p = .271 and U p > .05 for all pairwise comparisons. This suggests that the AI models produced incorrect answer options with a statistical profile similar to that of human examiners, at least as measured by how students engaged with them during the exam.

Item difficulty itself did not differ significantly between AI-generated and National Licensing Exam questions overall. However, an exploratory source-specific analysis revealed an overall difference among ChatGPT, Google Gemini, and National Licensing Exam items, with χ²(2) = 6.71, p = .035, and a specific difference between Google Gemini and National Licensing Exam items at p = .028. This exploratory finding hints that the two large language models may produce questions with slightly different difficulty characteristics, and that Gemini items in particular may have differed from the human-written benchmark in this sample.

Student-perceived curricular alignment did not differ significantly between AI-generated and National Licensing Exam questions, nor among ChatGPT, Google Gemini, and National Licensing Exam items. The researchers note that easier multiple-choice questions are generally perceived as more aligned with the curriculum, a relationship that held across the dataset. This perceptual finding matters because curriculum alignment is a key criterion for whether assessment material is educationally valid, and students judged the AI questions to be no less aligned with their course than the authentic licensing exam items.

The study received ethical approval from the Saarland Chamber of Physician’s Ethics Board on 24 June 2024, Germany, under reference Bu 234/20, and all procedures were conducted in accordance with the Declaration of Helsinki. Participation was voluntary, and informed consent for participation, anonymous data processing and storage for 10 years was obtained prior to participation. Open Access funding was enabled and organized by Projekt DEAL, and no external funding was received.

The authors conclude that students’ recognition did not differ between AI-generated multiple-choice questions and National Licensing Exam questions, and that AI-generated questions may provide a feasible approach to item drafting within a structured human-review process. The corresponding author is Fabian Dupont, who can be reached at fabian.dupont@uks.eu. The article was received on 28 May 2026, accepted on 11 September 2026, and published on 24 September 2026, carrying the DOI 10.1186/s12909-026-10410-8. Supplementary materials include an illustration of the prompting workflow for AI question generation, AI-generated question examples reviewed and edited for the final exam, and a before-and-after comparison of an AI-generated question following expert panel review.

Subject of Research: Artificial Intelligence

Article Title: Can students identify AI? – A cross-sectional quantitative study about AI recognition in tablet-based MCQ assessment among fifth-year undergraduate medical students at Saarland University, Germany

Article References: Vogt, P., Wolf, N., Jordan, S., Volz-Willems, S., Jäger, J., & Dupont, F. (2026). Can students identify AI? – A cross-sectional quantitative study about AI recognition in tablet-based MCQ assessment among fifth-year undergraduate medical students at Saarland University, Germany. BMC Medical Education, 26(1), Article 1519. https://doi.org/10.1186/s12909-026-10410-8

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10410-8

Keywords: students, identify, cross-sectional, quantitative, recognition, tablet-based, assessment, fifth-year, undergraduate, medical, Saarland, University