AI Helps Radiologists Diagnose Tricky Salivary Gland Tumors in Landmark Prospective Trial

Telling two look-alike tumors apart on an ultrasound image may soon become a genuinely collaborative exercise between human expertise and machine intelligence. Researchers in South Korea have built an artificial intelligence system that does not simply spit out a diagnosis, but behaves more like a cautious colleague: it knows when it is confident, when to defer, and how to hand control back to the radiologist. In a rigorously designed prospective trial published in BMC Medicine, the system measurably improved the diagnostic accuracy of board-certified radiologists reading parotid gland ultrasound scans, offering one of the clearest demonstrations yet that human–AI teamwork can outperform either partner alone.

The clinical problem at the heart of the study is deceptively narrow but surprisingly stubborn. The parotid gland, the largest of the salivary glands, can develop two benign tumors that together account for the vast majority of non-cancerous parotid masses: pleomorphic adenoma and Warthin’s tumor. Both matter because their management differs, yet on ultrasound their appearances overlap enough that even experienced ultrasonographers can hesitate. Most AI tools pitched at this task have so far lived comfortably inside retrospective datasets, validated on images the algorithm had effectively already seen in development, and never tested in the messier environment where a real radiologist either accepts or ignores the machine’s suggestion. The Korean team, led from the Department of Biomedical Informatics at Korea University College of Medicine together with clinicians at Korea University Ansan and Anam Hospitals, set out to break that pattern.

Architecturally, the system is a hybrid pipeline built around a division of labor. The first stage uses a YOLO-family object detection model, which scans the ultrasound image to locate the suspicious lesion and delineate the region of interest. The second stage is a classification model, based on the ConvNeXt convolutional network architecture, that examines the localized region and renders a verdict: pleomorphic adenoma or Warthin’s tumor. Crucially, the pipeline is confidence-aware. When the detection model’s certainty about the region it has found falls below a pre-specified threshold, set at 0.80 in the final deployment configuration, the system falls back on the classification model to interpret the whole image instead. That fallback mechanism means the AI never has to commit to a localization it does not trust, a small piece of engineering humility with outsized consequences for performance.

The development data came from 949 ultrasound images collected at Korea University Ansan Hospital, with a completely separate external set of 427 images from Korea University Anam Hospital reserved for testing. Rather than assuming their architecture choices were optimal, the researchers explored 35 different combinations of detection and classification models to understand how the two components complement each other. The exploration revealed something subtle: the hybrid pipeline outperformed both of its standalone components in 22 of the 35 pairings on external data, and the benefit was most pronounced when the detection and classification models performed at a similar level. The best configuration achieved a macro-F1 score of 0.838, a balanced measure of precision and recall. When one component was dramatically stronger than the other, the hybrid’s advantage shrank, a finding that carries practical lessons for anyone assembling multi-stage medical AI systems.

The standalone results themselves underscore the diagnostic difficulty. On the external test set, the detection-first pipeline achieved an area under the receiver operating characteristic curve, or AUROC, of 0.811, while the classification model acting alone reached 0.919. AUROC is a standard gauge of a model’s ability to distinguish the two tumor types across all possible decision thresholds, with 1.0 representing perfect discrimination. But the authors argued that neither number alone answers the question that actually matters in a clinic: does the AI make radiologists better? That question demands a different kind of study entirely, one in which human observers read cases first unaided and then with the algorithm whispering over their shoulder, ideally without knowing which is which.

That is precisely what the two-phase observer-performance study delivered. Three board-certified radiologists read 107 external patient cases, three reads per case for a total of 321 interpretations, first on their own and then with AI assistance available through an interface the radiologists could edit. The statistical machinery, a generalized linear mixed model, was chosen because it can properly handle the fact that each radiologist read each case and the repeated measures are entangled. The verdict: pooled diagnostic accuracy climbed from 76.3 percent to 81.6 percent when AI was in the loop, an odds ratio of 1.54 with a 95 percent confidence interval of 1.12 to 2.13 and a p-value of 0.008, meaning the improvement is unlikely to be statistical noise. Just as strikingly, the three readers converged toward each other, with Fleiss’ kappa, a measure of inter-reader agreement that ranges from 0 to 1, rising from 0.475 to 0.611. In plain terms, the AI did not just make the radiologists more accurate; it made them more consistent with one another.

Perhaps the most revealing results concern how the human readers actually behaved, because selective trust is the difference between collaboration and automation bias. The radiologists were not rubber stamps. They followed the AI’s correct predictions 76.2 percent of the time but adopted its incorrect predictions only 51.1 percent of the time, a statistically significant gap that suggests genuine discrimination rather than blind deference. The benefits also concentrated where they were most needed: on harder cases, the gain from AI assistance was larger, which is exactly the profile one wants from a decision-support tool. The system’s explainability layer, using techniques such as Grad-CAM heatmaps to show which image regions drove the classification and D-RISE explanations for the detection model, gave readers visual evidence they could weigh against their own reading of the scan.

There is also an honest asymmetry in the data that the authors did not hide. Sensitivity for Warthin’s tumor jumped by 18.1 percentage points under AI assistance, while sensitivity for pleomorphic adenoma slipped by 9.3 percentage points. The pattern tracks the AI’s own prediction profile: the machine was better at spotting Warthin’s tumors, and readers absorbed that strength along with that weakness. It is a candid illustration that AI assistance reshapes the whole diagnostic landscape rather than uniformly lifting it, and it flags a priority for future iterations of the system, which should aim to rebalance per-class performance before wider deployment.

Why does this particular trial matter beyond the parotid gland? Because prospective, reader-based evaluations of medical AI remain rare, even as algorithms multiply in the radiology literature. A model can post a dazzling AUROC on a curated dataset and still fail to change a single clinical decision. By pre-specifying a single deployment configuration, testing on external cases from a different hospital, and measuring outcomes that track actual human decisions, this study models the kind of evidence regulators, clinicians, and patients should be demanding. The authors were careful to frame the result as support for the system’s potential as a decision-support tool that operates alongside the interpreting physician, not as a replacement, and to call for further evaluation in real-world clinical workflows where scanner variation, workflow pressure, and patient diversity all conspire to degrade performance.

The trajectory from here is familiar but demanding. The system was built and validated within a single national health system, with consent waived for the retrospective components and institutional review board approval at both hospitals, and the deployment threshold of 0.80 was fixed in advance rather than tuned on the test set. Broader multi-center validation, monitoring for performance drift as imaging hardware and patient populations change, and studies of how radiologists interact with editable AI interfaces over months rather than one sitting are the logical next steps. But the core finding stands on its own: when an AI system is designed to know the limits of its confidence and to keep the radiologist firmly in command, the combination reads salivary gland ultrasound better than either the machine or the human alone. That may be the most viral idea in medical AI right now, not that algorithms will replace doctors, but that carefully engineered humility in software can make human judgment measurably, and provably, sharper.

Subject of Research: Human–AI collaboration for AI-assisted differentiation of benign parotid tumors on ultrasound

Article Title: Human–AI interaction in parotid ultrasound diagnosis: a prospective two-phase observer-performance study

Article References: Human–AI interaction in parotid ultrasound diagnosis: a prospective two-phase observer-performance study. (n.d.). https://doi.org/10.1186/s12916-026-05301-0

Image Credits: AI Generated

DOI: 10.1186/s12916-026-05301-0

Keywords: artificial intelligence, parotid ultrasound, pleomorphic adenoma, Warthin's tumor, deep learning, radiology, observer-performance study, decision support, confidence-aware AI, ConvNeXt, YOLO, BMC Medicine