AI oral screening tool passes its benchmark but fails real-world testing, audit finds

An artificial intelligence prototype designed to screen for oral health conditions scored well on its public benchmark, yet a detailed end-to-end audit revealed that the defects that mattered most to users lay not in the model itself but in the labels, evaluation software and deployment pipeline surrounding it. The audit, conducted by researchers at the University of Nottingham Malaysia and Universiti Malaya, examined a prototype combining a YOLOv8m detector trained to identify six oral conditions across 1,500 test images, a DenseNet169 histopathology network, and a ResNet18 router that decides which images reach the detector. The findings are published in Frontiers in Big Data.

The first problem the auditors uncovered was benchmark contamination. Under a pre-specified analysis plan, they found that near-duplicates of training images accounted for at least 17.3 percent of the test set and 17.1 percent of the validation set, figures the authors describe as lower bounds. Such duplication undermines the independence of the test partition, since images the model saw during training reappear in the data used to judge it. Yet when the researchers removed the duplicated test images and compared the result against 10,000 matched controls, the mean average precision at an overlap threshold of 0.5 changed by only +0.001, with a 95 percent confidence interval running from −0.012 to +0.014. In other words, contamination compromised the benchmark’s integrity without detectably inflating its headline metric.

The labels themselves proved far more troubling. Only 22 percent of duplicated photographs carried identical labels on both copies, and an exploratory analysis suggested the model had memorized training labels in the specific orientations seen during training rather than learning generalizable features. A separate check of the software environment revealed that a library update shifted the stricter mAP@0.5:0.95 metric by −0.012 even though the model’s predictions were identical, showing how evaluation software alone can alter reported results.

The most consequential gaps emerged when every test image was run through the actual application pipeline. The router rejected 23 percent of in-domain photographs and 51 percent of photographs showing ulcers, which meant that per-image sensitivity for ulcers was just 0.36 despite the benchmark reporting the highest recall of 0.91. A class-mapping defect, still live in a public demonstration as of September 2026, gave the wrong name to 73 percent of findings. Meanwhile, the histopathology classifier labeled every single image as positive, and its label was not a diagnosis in any clinical sense.

The authors conclude that evaluations of medical AI should verify partitions and labels, pin library versions, and test deployed systems end to end rather than relying on aggregate benchmark scores. The study is a brief research report of a single prototype audit, so its conclusions describe this system specifically, but its central lesson is general: the numbers that look impressive on a leaderboard may bear little resemblance to what a user actually sees on screen.

Subject of Research: An end-to-end audit of an AI-based oral health screening prototype revealing benchmark duplication, label errors and deployment defects.

Article Title: From benchmark to user: an end-to-end evaluation audit of an AI-based oral health screening prototype

Article References: Char, I. C. J., Sayed, A. S. A., Ichikawa, K., Sayed, Y. M. G., Bin Milhan, A., Sim, D. Y. Y., & Ramanathan, A. (2026). From benchmark to user: an end-to-end evaluation audit of an AI-based oral health screening prototype. Frontiers in Big Data, 9, Article 1972023. https://doi.org/10.3389/fdata.2026.1972023

Image Credits: AI Generated

DOI: 10.3389/fdata.2026.1972023

Keywords: artificial intelligence, oral health screening, benchmark contamination, data leakage, machine learning evaluation, model audit, digital public health, annotation quality, reproducibility, end-to-end validation, benchmark, user