The study, conducted by Hanife Merva Parlak of the Department of Periodontology at Hacettepe University’s Faculty of Dentistry in Ankara, set out to answer a deceptively simple question: do question type and topic affect how well large language models answer periodontology questions? This matters because the debate over artificial intelligence in health science education has largely focused on accuracy scores, with less attention paid to the explanatory quality that determines whether a chatbot can actually teach rather than merely answer. A model that picks the correct option but offers a vague, inconsistent, or misleading rationale may be worse for learning than one that occasionally errs but explains itself clearly.

To investigate, Parlak assembled 134 multiple-choice questions drawn from the Dental Specialization Examination administered in Türkiye, the high-stakes test that dentists must pass to enter specialty training. The questions were fed to three widely used consumer-facing artificial intelligence systems: ChatGPT-5, Gemini 2.5 Flash, and Copilot. Each model’s responses were then assessed on two separate dimensions. The first was straightforward accuracy — did the model select the correct answer? The second was more nuanced: the quality and adequacy of the explanation accompanying each answer, judged on a Likert scale, a standard survey instrument that rates responses along a graded spectrum of quality.

The questions themselves were not uniform. As described in the study’s supplementary materials, they spanned different cognitive levels, following the classic taxonomy of educational objectives. Standard questions tested basic recall — the remembering level — such as memorized facts about periodontal disease classification. Information-based questions probed understanding, requiring the model to interpret and apply knowledge in a slightly transformed context. Case-based questions, the most demanding category, simulated clinical scenarios at the applying level, asking models to reason through patient presentations the way a periodontist would. The question pool also covered both basic science content and clinical periodontology, allowing the researcher to disentangle whether models perform differently on foundational biology versus applied clinical reasoning.

The headline finding on accuracy was, in a sense, a non-finding: the total accuracy rates of ChatGPT-5, Gemini 2.5 Flash, and Copilot were statistically similar. All three models have absorbed vast corpora of biomedical text, and by now the ability of frontier chatbots to answer multiple-choice dental questions at or near specialist level has been demonstrated repeatedly across medical specialties. What distinguished the models was something subtler. Gemini 2.5 Flash outperformed its rivals in the adequacy of its response explanations, a difference that reached statistical significance. In other words, when it came to articulating why an answer was correct, Gemini’s outputs were judged more complete and more useful than those of ChatGPT-5 or Copilot.

The pattern deepened when the analysis turned to question categories. Gemini provided more consistent explanations than the other models on basic science questions and on clinical periodontology questions alike, and it maintained that consistency across standardized and information-based question formats. Each of these differences was statistically significant. This suggests that Gemini’s advantage was not confined to one corner of the syllabus but reflected a more general capacity to produce coherent, pedagogically sound rationales across the breadth of periodontology content tested.

To move beyond simple group comparisons, the study employed regression analyses, statistical techniques that estimate how much each predictor variable contributes to an outcome while accounting for the others. The results were telling: question type and topic were both significantly related to explanation quality. Accuracy, by contrast, appeared comparatively robust to these factors. The models generally knew the right answer regardless of whether a question tested recall or clinical application, or whether it concerned the microbiology of periodontal pockets or the surgical management of mucogingival defects. But the clarity, depth, and consistency of the reasoning they offered in support of those answers fluctuated considerably depending on what was being asked and how.

This dissociation between accuracy and explanation quality carries real implications for how artificial intelligence should be integrated into health professional education. Multiple-choice examinations, including the specialty exams used to license and certify clinicians, reward the selection of a correct option. A student using a chatbot as a study aid, however, typically learns from the explanation, not the letter of the answer. If a model’s rationale is thin, inconsistent, or subtly wrong even when its final choice is right, the student may internalize flawed mental models that surface later at the chairside. Conversely, an explanation of high quality can transform a simple quiz question into a miniature tutorial, connecting the answer to underlying mechanisms, differential diagnoses, and treatment principles.

The findings also speak to the technical character of large language models. These systems generate answers by predicting likely text sequences based on patterns learned during training, rather than by consulting a verified knowledge base. Their factual accuracy on well-represented exam content can be excellent, because such content appears abundantly in textbooks, review articles, and question banks. Explanation quality, however, depends on the model’s ability to organize and verbalize reasoning in a way that is both correct and appropriately calibrated to the question’s cognitive demand — a harder task that varies more with the structure of the prompt. Case-based questions, which require integrating scattered clinical clues, appear to stress this capacity in ways that simple factual recall does not, and the regression results indicate that this stress shows up in explanation quality even when the final answer survives intact.

Parlak’s conclusion is measured. Large language models, the study finds, are promising tools for periodontology education, but they retain limitations that require improvement before they can be relied upon uncritically. The author also cautions that the findings should be interpreted within the constraints of the study design: the questions came from a single national specialty examination, were posed in Turkish, and the evaluation of explanation quality, though systematic, ultimately rests on structured human judgment rather than an objective gold standard. Consumer chatbot interfaces also evolve rapidly, meaning that any snapshot comparison reflects a particular moment in a fast-moving technological race. Still, the central message is likely to resonate well beyond periodontology. As educators worldwide weigh whether to embrace, restrict, or redesign assessment in the age of generative artificial intelligence, this study suggests the right question is not simply whether chatbots get the answer right, but whether they can explain it in a way that actually teaches. On that measure, the models are not yet interchangeable — and the format of the question matters more than anyone might have guessed.

Subject of Research: Comparative performance of large language models in answering periodontology multiple-choice questions

Article Title: Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study

Article References: Parlak, H. M. (2026). Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study. BMC Medical Education. https://doi.org/10.1186/s12909-026-10516-z

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10516-z

Keywords: large language models, artificial intelligence, periodontology, dental education, ChatGPT-5, Gemini 2.5 Flash, Copilot, multiple-choice questions, medical education, explanation quality, question type, regression analysis

The study, published in Discover Artificial Intelligence, was designed to mirror how ordinary people actually use these tools. On a single day, 30 June 2025, the team submitted all fifteen questions to each model through standard consumer web interfaces, exactly as a patient would, with no API access, no custom settings, and no prompt engineering. The questions themselves were drawn exhaustively from the official NHS page of common questions about pregabalin, a resource compiled and validated by clinical pharmacists for the general public. Each query came wrapped in a standardized instruction asking for a clear, factual, patient-appropriate answer that must include a warning if any potential danger existed. That explicit safety instruction is what makes the study’s central finding so striking.

Pregabalin was a deliberate and timely choice. The drug is a structural analogue of the neurotransmitter GABA, but it does not act on GABA receptors directly; instead it binds selectively to the alpha-2-delta-1 subunit of voltage-gated calcium channels, dampening the release of excitatory neurotransmitters. Its clinical footprint has expanded dramatically over the past decade. Global annual sales of gabapentinoids, the drug class that includes pregabalin, rose by 114.5 percent between 2012 and 2022, and by a staggering 180.9 percent in low- and middle-income countries, where specialist care is scarce and patients are especially likely to seek answers from digital platforms. Prescribing rates more than doubled over ten years, reaching 7.2 prescriptions per 100 patients in 2019, with the highest volumes among elderly patients juggling multiple medications. Off-label and recreational use have broadened the population asking questions even further.

To judge the answers, the researchers assembled a multidisciplinary panel of a rheumatologist, a neurologist, and a pharmacologist, each with at least five years of clinical experience, reflecting the three main specialties that prescribe and monitor pregabalin. The evaluators worked independently and blind to which model had produced each response, after a calibration session in which they jointly scored three practice answers to align their criteria. Six dimensions were assessed: warning compliance, accuracy, completeness, content safety, readability, and clinical relevance. The study followed the TRIPOD-LLM reporting guideline, an emerging standard for research involving large language models, and the team calculated inter-rater reliability using Fleiss’ kappa and intraclass correlation coefficients.

The headline numbers on accuracy look encouraging. ChatGPT-4o, Gemini 2.0 Flash, and Copilot each scored 0.933 for factual correctness, while Claude 3.5 Sonnet scored 0.867. Clinical relevance ranged from 0.867 to 1.000 across the four models. But the confidence intervals overlapped so extensively that no model could claim reliable superiority over any other, a point the authors emphasize repeatedly. With only fifteen questions and a single query per model, the differences fall well within the statistical noise. The researchers are candid about this: the point estimates are best read as an illustration of the evaluation framework rather than stable performance benchmarks, and the single-query design means the stochastic variability of these models, which can produce substantively different answers to identical prompts, was never characterized.

The safety findings are where the study bites. Every one of the fifteen questions was judged by panel consensus to carry a clinically relevant safety concern warranting a warning, and the prompt explicitly demanded one. Yet warning compliance scores ranged only from 0.467 for ChatGPT-4o and Gemini 2.0 Flash to 0.733 for Claude 3.5 Sonnet. In plain terms, the models ignored a direct safety instruction in roughly 27 to 53 percent of cases. The authors argue this is arguably more alarming than a simple failure to volunteer warnings, because it demonstrates that even explicit prompting cannot guarantee reliable safety communication. For a drug with dependence potential, meaningful interaction risks, and a heavy presence among polypharmacy patients, that gap between instruction and behavior is the study’s most clinically consequential result.

When the researchers dissected the errors, a clear pattern emerged. Oversimplification dominated, accounting for 56.3 percent of all recorded errors, followed by omission of essential information at 34.8 percent, with outright hallucinations making up 8.9 percent. Gemini 2.0 Flash logged the most errors at 32, while Claude 3.5 Sonnet logged the fewest at 25. The authors connect these patterns to plausible, though unproven, architectural explanations: token probability maximization may favor statistically common, simplified phrasing over clinically nuanced detail; attention-based architectures may underweight rare but critical content; and reinforcement learning from human feedback may optimize for fluency and consensus at the cost of completeness. Hallucinations, though rarest, remain the most dangerous failure mode, since a fabricated dosing recommendation or a fictitious drug interaction could translate directly into harm.

The study is notable as much for its transparency as for its findings. In an unusual move, the authors disclose that the rater-by-item scoring matrix and the original response corpus were not archived after analysis, that the rule used to collapse three raters’ scores into one per-question value was never recorded, and that formal paired statistical tests such as McNemar’s test therefore became impossible. Copilot’s characteristic inline citations may also have compromised the blinding protocol. Rather than presenting plausible reconstructions as fact, the team reports these gaps openly, framing the work as an exploratory evaluation whose primary contribution is the multidimensional framework itself and a model for honest documentation of methodological limitations in early-stage LLM research.

What should be done with these results? The authors recommend hybrid architectures that ground chatbot outputs in structured, validated pharmaceutical databases rather than relying on parametric knowledge alone, along with dedicated fine-tuning on pharmacovigilance corpora so that safety alerts are generated reliably rather than left to general instruction-following. They also propose uncertainty detection mechanisms that would signal a model’s limits and steer users toward professional consultation, and even a continuous error-monitoring system modeled on existing pharmacovigilance infrastructure. The limitations are real: one medication, fifteen questions, one query per model, English only, and no testing with actual patients. The team, based in Francophone West and Central Africa, already plans a follow-up in French, where the stakes may be even higher. For now, the message to patients is clear: chatbots can be a reasonable starting point for basic medication facts, but when it comes to the warnings that keep you safe, they still cannot be trusted to speak up.

Subject of Research: Evaluation of large language models answering patient questions about the medication pregabalin

Article Title: A structured exploratory multidisciplinary evaluation of four large language models responding to frequently asked patient questions about pregabalin

Article References: A structured exploratory multidisciplinary evaluation of four large language models responding to frequently asked patient questions about pregabalin. (n.d.). https://doi.org/10.1007/s44163-026-02394-7

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02394-7

Keywords: large language models, ChatGPT-4o, Copilot, Gemini, Claude, pregabalin, patient safety, medication information, warning compliance, hallucination, pharmacovigilance, TRIPOD-LLM