ChatGPT configuration led expert ratings of dental radiology advice

In a controlled comparison of consumer-facing artificial intelligence tools asked to explain dental radiology and radiation-safety questions to patients, the selected ChatGPT configuration received the most favourable expert ratings for accuracy, relevance, safety and misinformation. The study, published in Frontiers in Oral Health, evaluated one response from each of three platform configurations for 20 English-language questions, producing 60 responses that were independently assessed by three oral and maxillofacial radiologists. The results apply to the particular interfaces, access tiers and single outputs tested, rather than establishing that one underlying AI model is generally superior.

The investigators used a standardized prompt instructing each system to answer as it would to a patient, in clear, simple and patient-friendly language. The tested products were an investigator-recorded, user-visible ChatGPT-5.5 configuration with its “Thinking” mode enabled, the free Gemini 3 Flash configuration and the free Microsoft Copilot 2026 configuration. Data collection took place in late April 2026 using the same device, browser and internet environment. A new chat was opened for every question, and chat history, memory and prior conversational context were disabled. Exactly one response was collected per platform and question; no follow-up prompts or repeated generations were made, so the study measured the specific outputs produced during that period rather than response stability or repeatability.

Twenty questions were selected from an initial pool of 57, following review by seven volunteer oral and maxillofacial radiologists who were not study authors and did not later score the AI responses. The final set covered radiation exposure, imaging frequency and necessity, radiation risks, paediatric imaging, pregnancy, cone-beam computed tomography, radiation protection, common misconceptions and professional consultation. The researchers also drew on international radiation-safety guidance and professional resources. Because original item-level classifications and detailed content-validity calculations had not been retained, the instrument was described as expert-reviewed rather than formally validated.

Each response was evaluated across seven domains. Accuracy, relevance, clarity, completeness, safety and triage appropriateness were scored from 1 to 5, with higher ratings indicating more favourable performance. Misinformation was scored from 0 to 4, with lower scores indicating less misinformation. Accuracy concerned whether stated information was factually correct against an item-specific evidence framework; completeness concerned whether essential information was omitted. Safety considered possible consequences of relying on the response, including false reassurance, unnecessary alarm, inappropriate avoidance of indicated imaging or delayed professional care. Clarity was judged by radiologists rather than tested in patients, so it represented expert assessment of apparent comprehensibility rather than measured understanding.

Question-level analysis treated the 20 questions as the primary paired experimental units, because three ratings of the same fixed response were correlated rather than independent observations. Friedman tests identified differences among the configurations in all six inferentially analysed domains, followed by two-sided exact sign tests for question-level pairwise comparisons. The analysis also examined ceiling effects, ties and inter-rater agreement. This approach showed that the tested ChatGPT configuration outperformed both alternatives in several respects, but the scale was compressed at the top for some domains, with numerous identical scores and very large effect estimates in comparisons based on only a few discordant questions.

ChatGPT received the highest accuracy ratings on all 20 questions, while Gemini and Copilot were tied on 19. All 60 individual ChatGPT ratings for accuracy were the maximum score of 5, compared with 59 scores of 4 and one score of 3 for Gemini and a score of 4 for every Copilot response. The same pattern appeared for safety: ChatGPT received 5 for all 60 ratings, whereas Gemini and Copilot each received a mixture of 3s and 4s and tied on all 20 questions. The question-level differences for ChatGPT versus both alternatives were significant after adjustment, but these ceiling effects also demonstrate that the rating instrument had little room to distinguish among the best outputs.

ChatGPT also received more favourable relevance ratings than Gemini on 18 questions and than Copilot on 19, with significant adjusted differences in both cases. It was rated more highly than Copilot for clarity on 16 questions and for completeness on all 20. Gemini, in turn, was rated more favourably than Copilot for clarity on 10 questions and for completeness on 19. The ChatGPT-Gemini differences for clarity and completeness were not significant: the platforms tied on 17 of 20 questions for completeness, while clarity produced 10 questions favouring ChatGPT, seven favouring Gemini and three ties. Thus, the study does not support a clean ranking across every quality dimension.

Misinformation ratings told a similar story. All 60 ChatGPT ratings were 0, the most favourable value on the reverse-scored scale, while Gemini and Copilot received a mixture of 0, 1 and 2. ChatGPT was rated as containing less misinformation than both alternatives on every question, whereas Gemini and Copilot did not differ significantly. A separate post hoc review, which did not alter the original scores, identified outdated or unqualified advice about lead aprons or thyroid collars in one of 20 ChatGPT responses, 12 of 20 Gemini responses and 11 of 20 Copilot responses. It also found claims of complete or near-complete protection from radiation in six Gemini responses and one Copilot response. Such claims are inconsistent with contemporary American Dental Association and American Academy of Oral and Maxillofacial Radiology guidance against routine shielding in dental radiography, including for children and pregnant patients, while local regulatory requirements still apply.

Several methodological boundaries sharply limit the findings. The platforms were not computationally equivalent: ChatGPT was used with reasoning enabled and paid access, whereas Gemini and Copilot were accessed through free tiers. Differences may therefore reflect access level, reasoning allocation, interface behaviour, retrieval functions or undisclosed backend routing, not just the models themselves. Platform names were concealed from evaluators, but retained headings and stylistic features might have allowed recognition, and masking success was not formally assessed. The study-specific rubric was not independently validated, response-specific reasons for scoring inconsistencies were not recorded, and the post hoc shielding analysis was not a blinded error-adjudication exercise. Only one question was suitable for assessing triage appropriateness: what a worried patient should do when a dentist recommends an x-ray. All three platforms received a median triage score of 4, but that single-question result was descriptive and cannot support broader comparisons.

Most importantly, the study evaluated what radiologists thought of written answers, not what happened to patients. It did not measure comprehension, trust, anxiety, willingness to accept imaging, care-seeking behaviour or clinical outcomes, and it did not test whether the advice changed decisions. High inter-rater agreement also requires caution: pooled single-measure intraclass correlation coefficients ranged from 0.696 for clarity to 0.976 for accuracy, but much of the agreement reflected large differences between platform configurations rather than reliable discrimination among responses within each platform. Repeated generations, equivalent configurations, verified blinding, more complex clinical scenarios, direct patient testing and formal adjudication of factual errors would be needed for stronger conclusions. For now, the results show that selected consumer AI systems can produce apparently understandable dental-radiology information, while also demonstrating why individual outputs should not be treated as a uniform or clinically validated substitute for professional advice.

Subject of Research: Comparative expert evaluation of patient-facing AI answers to dental radiology and radiation-safety questions.

Article Title: Expert-rated accuracy and safety of patient-facing dental radiology responses from three AI platform configurations: a comparative evaluation

Article References: Kapoor, D., Garg, D., Nirola, A., Sunil, M. K., Verma, P., & Tadakamadla, S. K. (2026). Expert-rated accuracy and safety of patient-facing dental radiology responses from three AI platform configurations: a comparative evaluation. Frontiers in Oral Health, 7, Article 1937905. https://doi.org/10.3389/froh.2026.1937905

Image Credits: AI Generated

DOI: 10.3389/froh.2026.1937905

Keywords: artificial intelligence, dental radiology, radiation safety, patient education, large language models, ChatGPT, Gemini, Microsoft Copilot, misinformation, oral health, Expert-rated, accuracy