The research team, led by Amr A. Elgharib and colleagues, assembled a cohort of 162 patients aged 15 and over who presented with abdominal pain and were clinically suspected of having appendicitis between March 2024 and March 2025. All of them underwent appendectomy, and the final word on their diagnosis came from histopathological examination of the removed appendix, the definitive reference standard. The cohort was deliberately enriched: roughly 70 percent of patients turned out to have confirmed appendicitis, a prevalence far higher than in a general emergency department, which shapes how the results must be interpreted.

The researchers fed ChatGPT-5 structured clinical narratives for each patient, including age, sex, symptoms such as fever, nausea, vomiting and right lower quadrant pain, physical examination findings like rebound tenderness, and laboratory values including white blood cell count and neutrophil percentage. The prompt cast the model as a general surgery physician in an emergency department and, crucially, forbade it from using any established appendicitis scoring system such as Alvarado, AIR, RIPASA or AAS. Instead, the model had to rely on its own clinical reasoning, outputting either a diagnosis of acute or non-acute appendicitis and, for positive cases, a percentage probability stratified into low, intermediate or high risk.

The study was designed in two phases following the classic machine learning convention of a 70-30 split. In Phase 1, 114 cases were presented without any prior exposure to labeled examples, simulating how a clinician might use a readily available chatbot off the shelf. After the model’s predictions were recorded, the true diagnoses were revealed, and in Phase 2 the remaining 48 cases were used to test whether this in-context learning improved performance. The prompt format was fixed across all cases, and the model’s memory was cleared before each evaluation round to test its default behavior.

The headline numbers are eye-catching. In Phase 1 without randomization, ChatGPT-5 achieved 100 percent sensitivity, meaning it caught every single case of appendicitis with zero false negatives, alongside 81.6 percent specificity, a positive predictive value of 91.6 percent, a negative predictive value of 100 percent, and overall accuracy of 93.9 percent. The Alvarado score, by contrast, managed only 67.1 percent sensitivity, missing 25 true cases, though its specificity was higher at 92.1 percent, and its overall accuracy came to 75.4 percent. Statistical comparison using McNemar’s test for paired proportions confirmed that the model’s sensitivity and accuracy advantages were highly significant, with p-values below 0.001.

But the story took a twist when the researchers examined a subtle methodological trap known as order leakage. When cases are presented in a sequence, a language model can inadvertently exploit the ordering of the data rather than genuine clinical reasoning, artificially inflating its apparent accuracy. To probe this, the team re-ran the evaluations with the case order randomized. Sensitivity barely budged, remaining at 98.7 percent in Phase 1, but specificity collapsed dramatically from 81.6 percent to 36.8 percent, a difference that was highly statistically significant. In other words, part of the model’s apparent precision in ruling appendicitis in may have been an artifact of how the cases were sequenced in the prompt.

Phase 2 told a more balanced story. After in-context learning, the non-randomized model reached 100 percent sensitivity, 93.8 percent specificity, 97.0 percent positive predictive value, 100 percent negative predictive value and 97.9 percent accuracy, statistically indistinguishable from the Alvarado score’s 96.9 percent sensitivity and 93.8 percent specificity. Agreement between the two methods, measured with Cohen’s kappa, climbed from a moderate 0.503 in Phase 1 to an almost perfect 0.952 in Phase 2. The authors note that in-context learning produced a modest, non-significant gain in specificity, while sensitivity was largely unaffected, suggesting the model’s core ability to exclude appendicitis was robust from the start.

The findings sit within a growing but cautionary literature. Previous evaluations have found that ChatGPT’s answers to surgeon-designed appendicitis management questions were clinically pertinent but inconsistent, and that ChatGPT-3.5 performed significantly worse than clinicians for some acute abdominal conditions such as cholecystitis and diverticulitis. A systematic review of 29 studies on artificial intelligence in appendicitis highlighted wide heterogeneity in inputs, validation strategies and metrics. Imaging-based machine learning models reading CT scans have achieved sensitivities around 77 percent and specificities around 86 percent. The new study’s authors also point to broader evidence that data leakage has inflated reported performance in fields ranging from Parkinson’s disease detection to brain MRI classification, reinforcing their insistence on randomization as a guard against over-optimistic estimates.

The authors are careful about what their results do and do not license. Because the cohort consisted exclusively of surgically confirmed patients with high pre-test probability, the estimates cannot simply be generalized to the unselected emergency department population in whom appendicitis must be ruled in or out, and the study was limited to a single center, a single model and a single comparator, with newer scores like AIR and RIPASA omitted because the required data were not consistently available. ChatGPT-5, they conclude, cannot yet be relied upon to confirm acute appendicitis, given the Alvarado score’s higher specificity, and the findings should be regarded as hypothesis-generating pending prospective validation.

What the study does establish is a proof of concept with real clinical texture: a general-purpose language model, guided by a carefully engineered prompt and given nothing more than the routine clinical data already sitting in a patient’s chart, can match or exceed a century-old scoring heuristic at the task that matters most for safety, excluding the disease. The authors envision ChatGPT-5 as a clinical decision-support tool to assist professionals during early assessment, particularly in resource-limited settings, always under physician supervision and never as an autonomous diagnostician. They also warn explicitly against patient self-diagnosis, noting that false reassurance from a chatbot could delay timely medical evaluation. As large language models continue their march into medicine, this Egyptian study offers both an encouraging data point and a methodological warning: before trusting an AI’s diagnostic brilliance, check whether the cases were shuffled.

Subject of Research: Diagnostic accuracy of ChatGPT-5 compared with the Alvarado score for acute appendicitis

Article Title: Diagnostic performance of ChatGPT for acute appendicitis compared with the Alvarado score: a retrospective diagnostic accuracy study

Article References: Elgharib, A. A., Ghanem, M. I., Ibrahim, R. S., Elwakeel, N., & Shemes, A. (2026). Diagnostic performance of ChatGPT for acute appendicitis compared with the Alvarado score: a retrospective diagnostic accuracy study. Discover Artificial Intelligence, 6(1), Article 1352. https://doi.org/10.1007/s44163-026-02332-7

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02332-7

Keywords: ChatGPT-5, large language models, acute appendicitis, Alvarado score, diagnostic accuracy, artificial intelligence, clinical decision support, histopathology, order leakage, in-context learning, emergency medicine, prompt engineering

The study, conducted by Hanife Merva Parlak of the Department of Periodontology at Hacettepe University’s Faculty of Dentistry in Ankara, set out to answer a deceptively simple question: do question type and topic affect how well large language models answer periodontology questions? This matters because the debate over artificial intelligence in health science education has largely focused on accuracy scores, with less attention paid to the explanatory quality that determines whether a chatbot can actually teach rather than merely answer. A model that picks the correct option but offers a vague, inconsistent, or misleading rationale may be worse for learning than one that occasionally errs but explains itself clearly.

To investigate, Parlak assembled 134 multiple-choice questions drawn from the Dental Specialization Examination administered in Türkiye, the high-stakes test that dentists must pass to enter specialty training. The questions were fed to three widely used consumer-facing artificial intelligence systems: ChatGPT-5, Gemini 2.5 Flash, and Copilot. Each model’s responses were then assessed on two separate dimensions. The first was straightforward accuracy — did the model select the correct answer? The second was more nuanced: the quality and adequacy of the explanation accompanying each answer, judged on a Likert scale, a standard survey instrument that rates responses along a graded spectrum of quality.

The questions themselves were not uniform. As described in the study’s supplementary materials, they spanned different cognitive levels, following the classic taxonomy of educational objectives. Standard questions tested basic recall — the remembering level — such as memorized facts about periodontal disease classification. Information-based questions probed understanding, requiring the model to interpret and apply knowledge in a slightly transformed context. Case-based questions, the most demanding category, simulated clinical scenarios at the applying level, asking models to reason through patient presentations the way a periodontist would. The question pool also covered both basic science content and clinical periodontology, allowing the researcher to disentangle whether models perform differently on foundational biology versus applied clinical reasoning.

The headline finding on accuracy was, in a sense, a non-finding: the total accuracy rates of ChatGPT-5, Gemini 2.5 Flash, and Copilot were statistically similar. All three models have absorbed vast corpora of biomedical text, and by now the ability of frontier chatbots to answer multiple-choice dental questions at or near specialist level has been demonstrated repeatedly across medical specialties. What distinguished the models was something subtler. Gemini 2.5 Flash outperformed its rivals in the adequacy of its response explanations, a difference that reached statistical significance. In other words, when it came to articulating why an answer was correct, Gemini’s outputs were judged more complete and more useful than those of ChatGPT-5 or Copilot.

The pattern deepened when the analysis turned to question categories. Gemini provided more consistent explanations than the other models on basic science questions and on clinical periodontology questions alike, and it maintained that consistency across standardized and information-based question formats. Each of these differences was statistically significant. This suggests that Gemini’s advantage was not confined to one corner of the syllabus but reflected a more general capacity to produce coherent, pedagogically sound rationales across the breadth of periodontology content tested.

To move beyond simple group comparisons, the study employed regression analyses, statistical techniques that estimate how much each predictor variable contributes to an outcome while accounting for the others. The results were telling: question type and topic were both significantly related to explanation quality. Accuracy, by contrast, appeared comparatively robust to these factors. The models generally knew the right answer regardless of whether a question tested recall or clinical application, or whether it concerned the microbiology of periodontal pockets or the surgical management of mucogingival defects. But the clarity, depth, and consistency of the reasoning they offered in support of those answers fluctuated considerably depending on what was being asked and how.

This dissociation between accuracy and explanation quality carries real implications for how artificial intelligence should be integrated into health professional education. Multiple-choice examinations, including the specialty exams used to license and certify clinicians, reward the selection of a correct option. A student using a chatbot as a study aid, however, typically learns from the explanation, not the letter of the answer. If a model’s rationale is thin, inconsistent, or subtly wrong even when its final choice is right, the student may internalize flawed mental models that surface later at the chairside. Conversely, an explanation of high quality can transform a simple quiz question into a miniature tutorial, connecting the answer to underlying mechanisms, differential diagnoses, and treatment principles.

The findings also speak to the technical character of large language models. These systems generate answers by predicting likely text sequences based on patterns learned during training, rather than by consulting a verified knowledge base. Their factual accuracy on well-represented exam content can be excellent, because such content appears abundantly in textbooks, review articles, and question banks. Explanation quality, however, depends on the model’s ability to organize and verbalize reasoning in a way that is both correct and appropriately calibrated to the question’s cognitive demand — a harder task that varies more with the structure of the prompt. Case-based questions, which require integrating scattered clinical clues, appear to stress this capacity in ways that simple factual recall does not, and the regression results indicate that this stress shows up in explanation quality even when the final answer survives intact.

Parlak’s conclusion is measured. Large language models, the study finds, are promising tools for periodontology education, but they retain limitations that require improvement before they can be relied upon uncritically. The author also cautions that the findings should be interpreted within the constraints of the study design: the questions came from a single national specialty examination, were posed in Turkish, and the evaluation of explanation quality, though systematic, ultimately rests on structured human judgment rather than an objective gold standard. Consumer chatbot interfaces also evolve rapidly, meaning that any snapshot comparison reflects a particular moment in a fast-moving technological race. Still, the central message is likely to resonate well beyond periodontology. As educators worldwide weigh whether to embrace, restrict, or redesign assessment in the age of generative artificial intelligence, this study suggests the right question is not simply whether chatbots get the answer right, but whether they can explain it in a way that actually teaches. On that measure, the models are not yet interchangeable — and the format of the question matters more than anyone might have guessed.

Subject of Research: Comparative performance of large language models in answering periodontology multiple-choice questions

Article Title: Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study

Article References: Parlak, H. M. (2026). Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study. BMC Medical Education. https://doi.org/10.1186/s12909-026-10516-z

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10516-z

Keywords: large language models, artificial intelligence, periodontology, dental education, ChatGPT-5, Gemini 2.5 Flash, Copilot, multiple-choice questions, medical education, explanation quality, question type, regression analysis