Why Expert Disagreement Could Be the Missing Key to Trustworthy Medical AI

When a radiologist traces the outline of a tumor on a CT scan, that contour is not a fact of nature. It is a judgment, shaped by training, experience, and the specific instructions given for the task. Yet in the datasets used to train and test medical AI, that judgment is almost always flattened into a single binary mask: a rigid grid of pixels labeled either inside or outside the target. Two researchers at the University of Oxford, Sophie Fischer and Irina Voiculescu, argue in a new Perspective article in Frontiers in Imaging that this seemingly innocuous act of simplification is quietly undermining the reliability of medical segmentation models, and that the uncertainty discarded at the annotation stage carries a hidden value the field can no longer afford to ignore.

The problem begins with the nature of the task itself. Segmentation, the process of identifying which pixels in an image belong to a given structure such as an organ or a lesion, is foundational to medical image analysis. But many of the structures clinicians are asked to delineate do not have crisp, well-defined boundaries. A tumor may fade gradually into healthy tissue. An organ’s edge may be ambiguous at certain resolutions. When an expert draws a contour around such a target, they are making a series of informed decisions under uncertainty, weighing visual evidence against prior anatomical knowledge about shape, topology, and adjacency to neighboring structures. The binary mask that results from this process captures only the final verdict, not the deliberation behind it.

Fischer and Voiculescu describe the consequences of this loss with a concept they call annotation-induced evaluation uncertainty. When multiple distinct expert annotations are plausible for the same image, the reported performance of a segmentation model becomes conditional on whichever particular annotation happens to be used for evaluation. A model might score highly against one expert’s contour and poorly against another’s, not because the model’s quality changed, but because the reference standard shifted. In a safety-critical domain such as medical imaging, where these evaluations inform decisions about clinical readiness, the authors argue that this source of variance deserves the same scrutiny that has been applied to model uncertainty over the past decade.

The asymmetry is striking. Researchers have developed a rich toolkit for quantifying the uncertainty of model predictions: Bayesian approaches that treat network weights as probability distributions, Monte Carlo Dropout as a computationally practical approximation, spatial probability maps that convey where a model is confident and where it is not, and probabilistic architectures such as the Probabilistic U-Net, PHiSeg, and more recently CaliDiff, which learn distributions of plausible segmentations from multi-annotator data. Methods for assessing whether a model’s stated confidence matches its actual accuracy, known as calibration, have also matured considerably. But on the other side of the comparison, the reference annotation remains a single, rigid binary mask, stripped of the very uncertainty that the model is being asked to express.

The sources of annotation uncertainty are varied and well documented. Some are inherent to the imaging modality, such as low contrast that makes a boundary genuinely difficult to perceive. Others relate to the annotation protocol, where unclear instructions can be reduced through better guidance and visual examples. Still others stem from the properties of the target itself: an organ like the lung has a constrained location and shape that annotators can agree on with relative confidence, while a pathological target like a lung nodule may vary so much in appearance that even its presence can be disputed between experts. Annotator-specific factors, including individual preferences and levels of expertise, add further layers. In practice, the observed variability reflects a mixture of these inherent and avoidable sources, and while the latter can be minimized, perfect alignment between annotators is unrealistic.

Current approaches to handling this variability tend to compress it further rather than preserve it. When multiple annotations exist, they are often fused into a single consensus segmentation using methods such as majority voting or the STAPLE algorithm, which irreversibly removes information about the extent and spatial distribution of disagreement. When only one annotation is available per image, inter-annotator variability is typically summarized as a single scalar value, such as an average agreement score across a dataset. The authors point out that this scalar is indicative of potential variance in evaluation scores but cannot provide exact bounds or image-specific information. A model achieving 90 percent agreement against a test set with 75 percent inter-annotator agreement should not be interpreted as exceeding expert performance, because the test set score is itself conditional on the particular annotations chosen.

The scale of the problem is illustrated by a recent review of 241 medical image analysis challenges, which found that 74 percent did not report inter- or intra-annotator variability at all. This scarcity is partly economic: medical data and expert annotations are expensive to acquire, and commissioning multiple annotations per image multiplies that cost. When complete multi-annotator datasets are not viable, variability is reduced to a summary statistic or omitted entirely. Publicly available multi-annotator datasets therefore represent critical resources, and recent initiatives such as the CURVAS challenge demonstrate how they can support uncertainty-aware benchmarking that compares distributions rather than individual mask pairs.

The authors also examine the limitations of the metrics themselves. Overlap-based measures such as Dice and Intersection-over-Union have become standard practice, but they carry a well-known weakness for small or complex structures: because distance from the boundary is not considered when penalizing incorrectly classified pixels, slight differences at the edge can disproportionately affect the overall score. Boundary-based measures offer a complementary perspective, and frameworks such as Metrics Reloaded guide researchers toward selecting appropriate combinations. Yet even with a well-chosen set of measures, without information on the extent and distribution of plausible annotations, the metrics cannot distinguish between genuine segmentation errors and acceptable differences that fall within the range of expert variability.

The consequences become vivid in concrete scenarios. In a benchmark with high annotator disagreement, a model could be perfectly recreating the segmentations of one expert while being evaluated against equally plausible segmentations from another. If the presence of a tumor is disputed between annotators and the test set happens to include an annotation from an expert who chose to discard it, then a model that correctly identifies the tumor, matching the judgment of an expert who included it, will be penalized as erroneous. This is not a hypothetical concern. The authors note that many such cases can be found in existing multi-annotator datasets of pathological targets, including the LIDC-IDRI database of lung nodules and datasets for multiple sclerosis lesion segmentation.

The path forward, as Fischer and Voiculescu see it, requires a shift in how the field thinks about annotation. Rather than treating annotator variability as noise to be mitigated or overcome, they argue it should be recognized as rich information about annotation quality, segmentation difficulty, and the context needed for trustworthy evaluation. Future benchmarks should acquire multiple independent annotations, document annotation protocols clearly, and preserve rather than eliminate expert disagreement. Research by Joskowicz and colleagues suggests that two to three annotators often capture only a fraction of the variability found in larger groups, so the number of annotators must be chosen carefully relative to the task. Cost-efficient methods to approximate the distribution of plausible annotations, analogous to how MC Dropout approximated expensive Bayesian inference, represent one promising direction. Ultimately, the authors contend that if segmentation models are expected to represent uncertainty, then evaluation must account for uncertainty on both sides of the comparison, and that progress will depend not only on building more capable models but on building better benchmark representations that honor the full complexity of expert judgment.

Subject of Research: Annotation uncertainty and its impact on the evaluation of medical image segmentation models

Article Title: The hidden value of uncertainty in medical segmentation

Article References: Fischer, S., & Voiculescu, I. (2026). The hidden value of uncertainty in medical segmentation. Frontiers in Imaging, 5, Article 1965816. https://doi.org/10.3389/fimag.2026.1965816

Image Credits: AI Generated

DOI: 10.3389/fimag.2026.1965816

Keywords: medical image segmentation, annotation uncertainty, inter-annotator variability, model evaluation, binary masks, probabilistic segmentation, benchmark datasets, uncertainty quantification, clinical AI, radiology, deep learning, Frontiers in Imaging