Calibration failure and limited-label recalibration of retinal image classifiers under dataset shift

A retinal-image classifier that appeared nearly flawless on its home dataset lost much of its accuracy when tested on images from a different collection, while its probability estimates became poorly aligned with observed outcomes. Five EfficientNet-B0 models trained for binary detection of diabetic retinopathy on APTOS 2019 achieved an area under the receiver operating characteristic curve, or AUC, of 0.997 ± 0.002 on the internal APTOS test set. On the external Messidor-2 dataset, however, their AUC fell to 0.665 ± 0.066 and accuracy declined from 0.981 ± 0.008 to 0.639 ± 0.021.

The study shows why a model can retain some ability to rank cases without assigning reliable probabilities, and why recalibrating those probabilities does not by itself repair weak external discrimination. This was a cross-dataset external-validation study, not a prospective clinical evaluation. The models were trained for five epochs with AdamW, a learning rate of 3 × 10−4, batches of 32 and class-imbalance-weighted binary cross-entropy.

The experiment was repeated across five random seeds, and the checkpoint with the lowest APTOS validation loss was retained for each. APTOS 2019 contained 3,662 labeled images, while Messidor-2 provided 1,744 gradable images. Of the target images, 523 formed a calibration pool and 1,221 were reserved for final testing.

All images were resized to 224 × 224 pixels and normalized with ImageNet statistics. The endpoint combined grade 0 with all higher grades as “any retinopathy,” rather than using the clinically common threshold for referable diabetic retinopathy, so the findings are methodological rather than estimates of screening performance. Before interpreting the strong internal result, investigators audited the APTOS splits for repeated images.

Exact matching of decoded RGB arrays identified 48 cross-split groups containing 97 images. A perceptual-hash screen followed by manual review added five non-exact pairs judged to be the same or a near-identical retinal capture. Altogether, 51 connected groups contained 105 images distributed across training, validation and test sets.

Eleven groups had discordant original grades, including one with a different binary label. Removing all 27 affected test images left 523 source test images and did not materially change the result: uncalibrated AUC remained 0.997 ± 0.002 and accuracy was 0.983 ± 0.006. The audit therefore argues against the recognized overlap explaining the high internal score, but it could not establish patient-level independence because the available table contained no patient or eye identifiers.

Probability calibration asks a different question from discrimination: whether a predicted risk corresponds to the observed frequency of the outcome. On Messidor-2, the uncalibrated positive-class probabilities were poorly aligned with outcome frequencies, producing equal-width and equal-mass 15-bin expected calibration error, or ECE, values of 0.266 ± 0.077 and 0.262 ± 0.077. Applying a temperature fitted on APTOS validation data changed almost nothing.

Across paired comparisons on the same 1,221 target test images, source-domain temperature scaling reduced equal-mass ECE by 0.0010 and the Brier score by 0.0006. Although these tiny differences were statistically detectable because the same images were reused, their magnitude was negligible, and the high-confidence error rate at a 0.9 threshold did not change significantly. Calibrating on small labeled samples from the target dataset was more effective.

In repeated sampling, researchers drew 200 random subsets at each of four sizes—25, 50, 100 and 200 labeled Messidor-2 images—with every subset containing both classes. The procedure produced 16,000 successful fits across five model seeds, four sample sizes, 200 draws and four methods: temperature scaling, Platt scaling, beta calibration and isotonic regression. Every fitted calibrator was evaluated on the same disjoint target test set.

With temperature scaling, mean equal-mass ECE declined from 0.108 at 25 labels to 0.085 at 200, mean Brier score improved from 0.233 to 0.227, and mean negative log-likelihood fell from 0.667 to 0.646. A constant predictor based only on the positive prevalence in each calibration subset was used as a reference; temperature scaling achieved a positive mean Brier skill score of 0.080–0.084 across the sample sizes. The best method depended on how much target data were available.

At 25 labels, temperature scaling had the lowest mean Brier score, negative log-likelihood and equal-mass ECE among the four calibrated approaches, while the more flexible methods were less stable. At 200 labels, beta calibration performed best, with a mean Brier score of 0.221, equal-mass ECE of 0.061, negative log-likelihood of 0.634 and Brier skill score of 0.104 relative to the prevalence-only predictor. Its mean accuracy was 0.651, compared with 0.639 for temperature scaling, while mean AUC ranged from 0.659 to 0.665 across methods.

These comparisons describe this experiment rather than establishing a universal ranking. The flexible calibrators could alter ranking or create ties, unlike positive temperature scaling, which preserves the order of the original logits and therefore leaves AUC, accuracy and the 0.5 decision threshold mathematically unchanged. The threshold results showed why “high-confidence error” cannot be interpreted in isolation.

In the fixed 100-label temperature calibration, the high-confidence error rate at a 0.9 threshold fell from 0.220 to 0.001. Yet coverage at that threshold also fell from 0.667 to 0.005: rather than making confident predictions correct, recalibration compressed probabilities toward 0.5 until almost no test images were retained as high confidence. Across all five seeds and 6,105 seed–image predictions, the uncalibrated model covered 4,075 images at the 0.9 threshold and made 1,342 errors, corresponding to conditional error of 0.329 with a 95% confidence interval of 0.315–0.344.

The 100-label temperature model covered only 30 images and made five errors, but its conditional error of 0.167 had a wider interval of 0.056–0.347, and three of five seeds had no covered image. The observed reduction in high-confidence error therefore mainly reflected the collapse in high-confidence coverage, not the removal of classification mistakes. Risk–coverage curves supplied an additional check on what recalibration could accomplish.

The curves for raw predictions, source temperature scaling and all target temperature fits overlapped because multiplying logits by a positive temperature changes their magnitude but not their order. Mean selective error was 0.311 at 60 percent coverage and 0.334 at 80 percent coverage in every temperature condition. Calibration changed the numerical confidence scale but did not improve which cases were retained at a given coverage.

The optimization audit also revealed instability in the smallest target samples: 69 of 4,000 repeated temperature fits reached the expanded upper temperature boundary of 403.43, including 55 of the 25-label fits, whereas none did at 200 labels. These boundary solutions define confidence-compressing mappings, but their fitted temperatures should not be treated as precise finite estimates. The observed shift included a difference in positive-label prevalence: 48.7 percent in the APTOS test set, 39.0 percent in the Messidor-2 calibration pool and 42.8 percent in the Messidor-2 test set.

The datasets also came from different collection sources, making acquisition-related differences plausible, but the study lacked harmonized camera, demographic and patient-level metadata needed to separate prevalence effects from other forms of covariate shift. A scalar temperature has no intercept and is not a dedicated correction for label shift. The analysis was restricted to one model family, one dataset pair, one binary endpoint and four calibration methods, and it did not test prospective workflows, subgroup calibration, referral decisions or patient safety.

Its central result is that calibration, coverage, conditional error, discrimination and optimizer diagnostics should be reported together: limited target labels can improve probability estimates, but they do not turn a weakly transferring retinal classifier into a clinically ready system.

Subject of Research: External calibration and limited-label recalibration of retinal image classifiers under dataset shift

Article Title: Calibration failure and limited-label recalibration of retinal image classifiers under dataset shift

Article References: Rahat, S. R. U. I., Shaikh, M. H., Sorna, I. J., & Mahajan, M. (2026). Calibration failure and limited-label recalibration of retinal image classifiers under dataset shift. Discover Artificial Intelligence, 6(1), Article 1549. https://doi.org/10.1007/s44163-026-02520-5

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02520-5

Keywords: retinal imaging, diabetic retinopathy, probability calibration, dataset shift, external validation, temperature scaling, beta calibration, EfficientNet-B0, confidence thresholds, risk–coverage analysis, Calibration, failure