AI models learn when to say ‘I don’t know’ through conformalized selective regression

Should a prediction model always deliver a prediction? A new study argues that the answer is no, and it offers a concrete mathematical recipe for deciding when a model should stay silent. In a paper published on 29 July 2026 in the open-access journal Discover Data, Anna Sokol, Nuno Moniz and Nitesh V. Chawla introduce Conformalized Selective Regression, or CSR, a framework that lets regression models abstain from predicting when uncertainty runs high, and pairs that mechanism with a standardized yardstick for judging how well any selective model walks the line between accuracy and coverage.

Selective regression, also known as the reject option, is not a new idea. The authors trace its origins seven decades back, to early proposals that allowed classifiers and regressors to withhold answers in situations of considerable uncertainty. Yet the field, they argue, has been stuck on a particular way of measuring that uncertainty. Most existing approaches rely on distribution-based proxies, above all conditional variance, which estimates how much the predicted value of a target variable can vary for a given input. That quantity is easy to compute and easy to threshold, but it carries a blind spot that the new paper puts at the center of its argument.

The blind spot is model bias. Conditional variance describes the spread of the data around a conditional expectation, but it says nothing about the specific errors a particular trained model tends to make. A model can be systematically overconfident in certain regions of the input space, and a variance-based uncertainty signal will not notice. The authors contend that this neglect of model-specific biases can lead to overconfidence in predictions and to underestimating the limitations of selective regression models, precisely the failure modes that the reject option is supposed to guard against. Their remedy is to replace the distributional proxy with a signal that is calibrated to the model itself.

That signal comes from conformal prediction, a framework for uncertainty quantification that produces statistically valid prediction intervals without assuming anything about the underlying data distribution. Conformal methods build calibrated prediction regions around each predicted value at a user-specified confidence level, with a guarantee that the true value falls within the region with a certain probability. The guarantee rests on exchangeability and on a calibration set: conformity scores computed on held-out calibration data are used to set a quantile threshold, and the resulting coverage condition is designed to hold for any joint distribution of features and responses and any sample size. The framework has found applications ranging from image classification to healthcare diagnostics.

CSR builds on conformalized quantile regression, an existing technique that adapts prediction intervals to heterogeneous data. The procedure splits the data into training and calibration sets, fits two conditional quantile functions on the training set, one for the lower end of the interval and one for the upper end, trained with a quantile-based loss such as the pinball loss, and then computes conformity scores on the calibration set to quantify how far the intervals miss. Those scores adjust the estimated quantiles so that the final interval respects the designated coverage criterion. Crucially, the calibration step also absorbs whatever biases are inherent in the lower and upper quantile models, which is what makes the resulting uncertainty measure model-specific rather than purely distributional.

The selective mechanism itself is deliberately simple. The width of the conformalized interval serves as the uncertainty measure: a wider interval signals higher uncertainty, a narrower one signals confidence. The model outputs a prediction only if the interval width falls below a threshold; otherwise it rejects. Sweeping that threshold across its range produces the characteristic trade-off curve of selective regression, in which coverage, the proportion of predictions actually delivered, rises as the model accepts more uncertain cases and error rises with it. For visualization purposes, the authors use the mean value of the interval as the point prediction.

The second contribution addresses a quieter but arguably just as consequential problem: evaluation. The most common metric in this literature, the area under the curve, or AUC, of the error-versus-coverage trade-off, does not necessarily identify the most effective model. The authors illustrate with hypothetical regression models in which one model dominates another across most coverage levels, yet the smaller AUC does not reveal which model sits closest to the ideal operating point of zero error at full coverage. A model with a smaller AUC can still offer a more optimal trade-off between accuracy and coverage, and the field lacks a consensus on how to assess that balance.

Their proposed fix is a normalized distance-based framework. For each model, they compute the normalized mean squared error at each coverage level, normalizing by the maximum MSE across all models, and then locate the point on that curve closest to the ideal point of zero normalized error and full coverage. The Euclidean distance from that point to the ideal becomes a single-number summary of the model’s trade-off quality. The authors argue that this measure is more straightforward than alternatives such as the Pareto Frontier, which is suited to multi-objective problems but does not reduce to an easily ranked value, whereas the Euclidean distance simplifies threshold setting and direct comparison. Because all methods share the same underlying predictions at full coverage, differences in the metric reflect only the effectiveness of each rejection strategy.

The experimental evaluation tested CSR against two state-of-the-art baselines on four well-known datasets: COMPAS, Communities, Insurance and LSAC, split 70 percent training, 10 percent calibration and 20 percent test, with multiple underlying regressors including Random Forest, XGBoost and quantile neural networks. The first baseline enforces fairness through sufficiency in feature representation, calibrating mean and variance and rejecting when monotonic selective risk criteria are unmet. The second is a plug-in epsilon-predictor with a reject option that thresholds the conditional variance function, estimated semi-supervised with k-nearest neighbors, run with k set to 10 and repeated 100 times for stability. CSR was run with a fixed confidence parameter alpha of 0.05, and performance was evaluated across threshold values spanning 0 to 100 percent coverage.

The results, the authors report, favor CSR on both research questions. Across all four datasets, with the best Random Forest model, CSR demonstrated the best trade-off between error rates and coverage, maintaining lower normalized MSE values than either baseline. When the analysis was restricted to higher coverage levels of 0.8, 0.85, 0.9 and 0.95, chosen because practical applications rarely tolerate rejecting more than a modest share of predictions, CSR outperformed both comparison models in more than 80 percent of cases. To probe robustness further, the team ran the same comparison on an additional 25 regression datasets drawn from earlier work: CSR achieved top performance in 80 percent of them, 20 datasets, while the two baselines led in 8 percent and 12 percent respectively, two and three datasets. Lower AUC values in the accompanying comparison table also indicated better performance for the proposed method in that analysis. The authors caution that the coverage guarantee of conformal prediction holds under exchangeability and is not estimated directly, and they frame future work around different underlying evaluation metrics and scenarios in which predictive performance and coverage carry different weights. The source code and processed datasets are publicly available, and the authors suggest the approach could matter wherever unreliable predictions carry real costs, from healthcare outcome estimates to credit risk assessments.

Subject of Research: A conformal prediction framework for selective regression that lets models abstain from uncertain predictions and balances error against coverage

Article Title: Conformalized selective regression

Article References: Sokol, A., Moniz, N., & Chawla, N. V. (2026). Conformalized selective regression. Discover Data, 4(1), Article 14. https://doi.org/10.1007/s44248-026-00113-2

Image Credits: AI Generated

DOI: 10.1007/s44248-026-00113-2

Keywords: selective regression, conformal prediction, uncertainty quantification, reject option, machine learning, prediction intervals, model reliability, coverage, evaluation metrics, quantile regression, artificial intelligence, Conformalized