The problem is known as spurious correlation: a feature that is not genuinely predictive of the target becomes associated with it in the training data. A bird classifier might learn that water backgrounds indicate one species and land backgrounds indicate another, while a face classifier might associate blond hair with one gender. Such patterns can produce high average accuracy during training and testing while masking serious failures in underrepresented combinations. When a new example breaks the accidental relationship, performance can fall sharply. This makes average accuracy alone an incomplete measure of whether a model has learned the intended task.

The study builds on UnLearning from Experience, or ULE, a method introduced by the same research group. ULE trains two neural networks in parallel. The first, called the biased model, learns through standard empirical risk minimization, meaning it minimizes average training loss in the conventional way. The second, called the debiased model, is trained to solve the same classification task while using different features. It is encouraged to do so through a loss based on the gradients, or saliency maps, of the two networks: the debiased model receives a signal to direct its attention away from the features emphasized by the biased model.

That design is motivated by the “simplicity bias” hypothesis, which holds that neural networks tend to learn easier features before more complicated ones. If shortcut features are easier to learn than the task-relevant characteristics of birds, cars, faces or language, an early-stage network may be especially likely to use them. The researchers calculate the gradient of each model’s output with respect to its input and use the mean squared error between the negated biased-model gradient and the debiased-model gradient as a feature-saliency term. A separate cross-entropy term still requires the debiased network to predict the correct label, while a hyperparameter called lambda controls the balance between classification and saliency matching.

Weighted UnLearning from Experience, abbreviated wULE, changes how this combined loss is applied to individual examples. Earlier ULE treats every training item alike, implicitly assuming that all contain a shortcut. That can be counterproductive: forcing the two models to diverge on examples with no spurious correlation may prevent the debiased model from learning useful behavior that the biased model has already captured correctly. The new formulation therefore introduces two positive scaling factors. Alpha can increase the weight assigned to the classification loss for estimated non-spurious examples, while beta can increase the saliency loss for examples likely to contain spurious features. The authors describe wULE as a generalization of ULE, for which the two new factors can reduce to the original weighting scheme.

Because ground-truth group annotations are generally unavailable in practical use, the researchers tested two ways of estimating which training examples contain spurious correlations. A JTT-based method treats examples correctly classified by the biased network after an early training epoch as the spurious set, on the assumption that shortcut examples will already have been learned. A SPARE-based method instead clusters the biased model’s logits and assumes that the majority cluster represents examples with spurious correlations. In both cases, the biased network is frozen after the selected epoch and the debiased model continues training. Experiments on Waterbirds showed that both estimators worked best when the biased model was frozen early. The SPARE-based estimate was consistently stronger than the JTT estimate, producing better worst-group accuracy and remaining more useful at later freezing points, after the biased model had begun fitting the training set more comprehensively.

The evaluation covered Waterbirds, CelebA, UrbanCars and MultiNLI. Waterbirds tests whether a classifier can identify landbirds and waterbirds when backgrounds do not match the usual correlation. CelebA asks models to predict blond or non-blond hair when gender is correlated with the label. UrbanCars classifies cars as urban or country while varying both background and a co-occurring object, allowing researchers to test simultaneous shortcuts. MultiNLI is a natural-language inference task involving entailment, neutral and contradiction relationships; negation words in hypotheses were treated as a potentially spurious attribute. The principal metric was worst-group accuracy, which records performance in the least successful subgroup, alongside overall average accuracy. Group labels were used for evaluation and hyperparameter selection, not to identify spurious examples during wULE training.

On Waterbirds, both estimated variants of wULE significantly improved worst-group accuracy over ULE and approached results obtained by an “oracle” version that used ground-truth information to identify spurious examples. Across the broader comparison with previously published methods, wULE ranked among the strongest approaches: the authors placed it second on Waterbirds, UrbanCars and MultiNLI, and fourth on CelebA when using the same ResNet-50 backbone used for like-for-like comparisons. The SPARE-based variant produced the reported wULE results in that comparison. Its worst-group and average accuracies were also relatively close, which the authors interpret as evidence that it was less dependent on majority-group shortcuts than methods showing a larger gap between those measures. These are benchmark results rather than evidence of performance in uncontrolled production settings.

Additional analyses examined how the method responds to errors, model capacity and the choice of loss. Randomly corrupting the estimated spurious set caused worst-group accuracy to decline, but at corruption levels resembling the estimation errors observed in practice, performance remained near the results obtained by the two practical estimators. Increasing the emphasis on non-spurious examples through alpha and on saliency separation for spurious examples through beta generally improved results in the tested range, while the saliency term received somewhat more weight than classification. Replacing mean squared error with mean absolute error slightly reduced worst-group accuracy but marginally increased average accuracy, so the researchers retained mean squared error for their main analysis. Tests combining ResNet-18, ResNet-50 and ViT-H-14 backbones showed that a more expressive debiased model generally improved performance, whereas the biased model’s architectural complexity had less influence.

Explainable-AI heatmaps provided a qualitative view of the learned behavior. On UrbanCars, the standard biased model often assigned high saliency to the background or the accompanying object as well as to the car. Models trained with wULE concentrated more strongly on the car and gave less attention to the two misleading attributes. On Waterbirds, the debiased model increasingly focused on the bird as training proceeded, while the biased model continued to emphasize the background. The researchers also showed that the method need not carry an extra inference-time burden: both networks are trained in parallel only for the first k epochs, the biased model is then discarded, and only the debiased model is used for predictions. Overall, the results support weighted treatment of likely shortcut examples as a practical extension of ULE, while the reliance on early-training behavior, estimated groups, pretrained backbones and benchmark-specific tuning remain important limits on how broadly the findings can be generalized.

Subject of Research: Machine-learning method for improving robustness to spurious correlations in image and language models

Article Title: Weighted UnLearning from Experience for Improved Spurious Correlation Robustness

Article References: Mitchell, J., Martinez-del-Rincon, J., & McLaughlin, N. (2026). Weighted UnLearning from Experience for Improved Spurious Correlation Robustness. Machine Learning, 115(11), Article 251. https://doi.org/10.1007/s10994-026-07164-1

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07164-1

Keywords: spurious correlations, machine learning, robust AI, deep learning, out-of-distribution generalization, UnLearning from Experience, image classification, natural language inference, gradient saliency, benchmark datasets, Weighted, UnLearning