The study focused on a real medical physics problem. Ultrasound scanners are the workhorses of modern hospitals—real-time, low-cost and free of ionising radiation—but they must be regularly quality-assured throughout their lifespan. Medical physicists perform this task by scanning test objects known as phantoms, which offer predictable and controlled imaging environments. The Edinburgh Pipe Phantom (EPP), a purpose-built device containing wall-less, fluid-filled pipes suspended in tissue-mimicking material, allows physicists to measure the resolution integral, a single dimensionless figure-of-merit representing the overall imaging performance of a greyscale ultrasound scanner. The conventional method requires an operator to manually identify the most superficial and deepest visible extent of each pipe—a time-consuming and potentially observer-dependent process that automation could dramatically accelerate.

The team trained a U-Net architecture with a ResNet-18 backbone, a convolutional pairing selected because it had achieved the best performance in the authors’ previous comparison of segmentation models. The network accepted single-channel greyscale ultrasound images and generated two-class logits for pipe and background, trained with cross-entropy loss with label smoothing and an Adam optimiser, with the ResNet-18 encoder initialised with weights pretrained on ImageNet. To facilitate uncertainty estimation, a two-dimensional dropout layer with a probability of 0.3 was applied independently to each encoder feature map prior to decoding, allowing the researchers to run 20 stochastic forward passes at inference time and measure pixel-wise variance across predictions—an approach known as Monte Carlo dropout, which can be interpreted as approximate Bayesian inference.

The dataset consisted of 162 images acquired from the Edinburgh Pipe Phantom using ten different ultrasound probes across three scanners: a GE Logiq E10, a Mindray A20 and a Samsung R20. A trained operator followed the conventional resolution integral measurement protocol, acquiring superficial and deep visibility measurements for each visible pipe as separate, independently optimised images. Half of the images were randomly selected and flipped horizontally to represent the two possible orientations in which an operator may hold the probe, and each image was cropped to a square region containing the target pipe and resized to 256 × 256 pixels. Ground truth binary masks were created by manually segmenting the pipes of interest using VIA annotation software, with the resolution integral values from manually segmented images differing from corresponding visual measurements by only 0.9 ± 0.3%—well within the previously reported 3% repeatability of resolution integral measurements.

Performance was assessed using stratified five-fold cross-validation, with the 162 images divided into five mutually exclusive outer folds stratified according to the individual ultrasound probe used for acquisition, so that the relative contribution of each of the ten probes was approximately balanced across the five folds. The model achieved an overall Dice score of 0.81 ± 0.03 and a 95th-percentile Hausdorff distance (HD95) of 11.98 ± 3.39 pixels, with held-out performance remaining broadly consistent across folds. A single forward pass required just 0.003 seconds on an NVIDIA GeForce RTX 4090 GPU workstation and 0.036 seconds on a CPU-only laptop, compared with approximately 30 seconds for a manual image measurement—an approximately 99.88% reduction in the segmentation component of the workflow.

But the raw accuracy numbers tell only part of the story. Seven segmentation errors were identified among the 162 held-out images: three complete failures in which no foreground mask was predicted at all, and four non-empty predictions in which the incorrect pipe was segmented. The complete failures occurred at a rate of 1.85%, and while they could be deterministically identified from the model output—simply by checking whether the predicted mask was empty—the subtler wrong-pipe errors proved far more challenging to catch prospectively.

To understand the model’s internal behaviour, the researchers deployed a battery of interpretability tools. Predictive entropy, derived from the SoftMax probabilities, was used as an approximation of data-related ambiguity, while Monte Carlo dropout variance served as an approximation of model-related, or epistemic, uncertainty. Input-gradient saliency maps were generated by backpropagating the gradient of the summed foreground-class score with respect to the input image, and Grad-CAM heatmaps were applied to the final residual block of each of the four encoder stages and to the last decoder block. Calibration was assessed using the Brier score—the mean squared difference between predicted foreground probability and observed binary label—and the expected calibration error (ECE), which compares average predicted probability with observed frequency across ten probability bins.

The results delivered a striking verdict on the relative value of these measures. The Dice score was strongly negatively associated with the Brier score (Spearman ρ = −0.96) and the ECE (ρ = −0.72), indicating that images with better segmentation overlap also tended to show closer agreement between predicted probabilities and reference labels. HD95 was positively associated with the Brier score (ρ = 0.82), so larger boundary errors tended to occur in images with larger probability errors. In contrast, predictive entropy and Monte Carlo dropout variance showed only weak associations with segmentation performance (|ρ| < 0.4), meaning neither uncertainty approximation functioned as a strong image-level correlate of segmentation quality. The mean Brier score was 0.17 ± 0.02 and the mean ECE was 0.18 ± 0.02. Mean Monte Carlo dropout variance was below 0.01 in every fold, indicating limited variation across stochastic passes under the selected dropout configuration—though the authors caution this should not be interpreted as proof that the model had negligible epistemic uncertainty.

The visual analyses provided complementary but necessarily limited information. In selected multi-pipe images, elevated entropy, variance or activation overlapped both target and non-target structures, including in complete failures. In the erroneous example, the model segmented the non-target pipe while elevated uncertainty and activation also overlapped parts of the intended target region. Earlier encoder maps retained finer spatial detail, whereas deeper encoder maps were coarser due to progressive downsampling, and the final decoder map reflected the reconstructed segmentation features. Across folds, attention localisation showed the strongest mean positive association with Dice score in the final decoder block (mean ρ = 0.60), followed by encoder stage 2 (mean ρ = 0.51), although fold-to-fold variability was substantial and no single layer provided a uniformly strong association in every fold. Crucially, the authors emphasise that these maps represent sensitivity or activation, not causal explanations of the decision process.

An exploratory image-level quality-control analysis then tested whether poor-quality outputs could be flagged automatically for review. Morphology-based indicators—bounding-box fill fraction, predicted area deviation and component count deviation—were evaluated alongside uncertainty-based mean entropy, with poor segmentation quality retrospectively defined as Dice < 0.50 and/or HD95 > 20 pixels among non-empty predictions. Component count and predicted area deviation methods both achieved a sensitivity of 1.00 by identifying all 23 poor-quality cases, while boundary box fill fraction and mean entropy identified 22. However, these indicators showed insufficient specificity to support their use as prospective quality-control criteria in the present dataset. Of the four wrong-pipe errors, only the component count deviation method identified all cases, and specificity was poor (< 0.1) in all methods except predicted area deviation, which achieved 0.58.

The authors are careful to frame their conclusions within the scope of what was actually tested. The cross-validation design balanced images from each probe across folds and therefore evaluated new images acquired within the known probe and scanner distribution; it did not test generalisation to an entirely unseen probe, scanner or phantom. The dataset was small and contained controlled phantom images rather than clinical ultrasound scans, lacking anatomical variability, pathology, patient motion and many acquisition artefacts. Only one architecture was evaluated, and external validation on a separate phantom or scanner was not performed. Reference masks were produced by one operator from measurements acquired by another, and while agreement in derived resolution integral values was high, independent boundary-level inter-rater reliability was not assessed.

For the intended medical-physics workflow, the researchers envision a future quality-control interface that displays the predicted pipe mask and flags outputs with an empty or implausible foreground region for manual review before the resolution integral is calculated. Probability calibration could be monitored retrospectively on labelled quality-assurance datasets, but it would not be available as an image-specific reference-free check in routine use. Uncertainty maps may help developers investigate recurring failure patterns, although presenting them directly to users would require evidence that they improve decisions rather than simply increasing cognitive burden. The framework should therefore be viewed as a development and audit tool for EPP automation, not as a validated clinical decision-support system.

The study’s broader message resonates far beyond the Edinburgh Pipe Phantom. As deep learning pushes further into computer-aided diagnostics to alleviate pressure on healthcare professionals, growing concern surrounds the trustworthiness of machine learning predictions and the clinical implications of errors—a concern exacerbated by the black-box problem, the inability to understand how a model reached its prediction. This research demonstrates that building genuinely trustworthy medical AI requires more than impressive accuracy metrics: it demands multimodal evaluation combining segmentation performance, calibration, uncertainty approximations and attention analyses, each interpreted within its proper limits. Calibration metrics, the study suggests, may deserve more attention than the uncertainty approximations that have dominated recent interpretability discussions. And simple deterministic checks—like verifying that a predicted mask is not empty—may catch the most catastrophic failures that sophisticated uncertainty methods miss. Before broader or deployable reliability claims can be made, however, independent validation, alternative architectures and uncertainty methods, and prospective threshold evaluation are all required. The path to trustworthy medical AI, this study shows, runs through honest accounting of what our evaluation tools can and cannot tell us.

Subject of Research: Beyond segmentation accuracy: understanding model reliability and failure modes in ultrasound image analysis

Article Title: Beyond segmentation accuracy: understanding model reliability and failure modes in ultrasound image analysis

Article References: Bricout, A., Hardman, D., Inglis, S., Pye, S., & Moran, C. M. (2026). Beyond segmentation accuracy: understanding model reliability and failure modes in ultrasound image analysis. Frontiers in Medical Technology, 8, Article 1918432. https://doi.org/10.3389/fmedt.2026.1918432

Image Credits: AI Generated

DOI: 10.3389/fmedt.2026.1918432

Keywords: Beyond, segmentation, accuracy, understanding, model, reliability, failure, modes, ultrasound, image, analysis, scientific research