New Shrinkage Method Exposes Hidden Gaps in AI Prediction Intervals Across Groups and Time Horizons

Every machine-learning forecast that matters in the real world comes with an implicit promise: the prediction interval wrapped around it will contain the truth most of the time. A public-health agency forecasting child mortality, a bank estimating portfolio risk, or an engineer predicting how many years a battery has left all rely on intervals that should cover the realised value, say, ninety percent of the time. Conformal prediction has become the standard tool for keeping that promise, because it can turn any forecasting model, no matter how complex, into one with calibrated intervals using nothing more than a held-out calibration set. But a new study published in Machine Learning with Applications by Jitong Zou, Enmiao Feng, Pengfei Li and Gaike Wang reveals a troubling blind spot in that guarantee, and proposes an elegant statistical fix that borrows ideas from a classical technique developed half a century ago.

The problem is that conformal coverage is marginal, meaning it is an average over the entire population and over all forecast horizons. A system can be perfectly calibrated on paper while failing spectacularly in the places that matter most. The authors illustrate this with simple arithmetic: if ninety percent of records belong to a majority group covered at ninety-four percent, and ten percent belong to a minority covered at just fifty-four percent, the overall coverage is exactly ninety percent, yet the minority is under-covered by more than thirty points. The same illusion applies across time. An interval procedure that covers one-year-ahead forecasts at ninety-nine percent and five-year-ahead forecasts at eighty-one percent still averages out to the nominal target, while long-horizon users are left systematically exposed. Prior work has documented gaps of more than thirty percentage points between race groups on real clinical prediction tasks, and has shown that conformal sets can even reduce accuracy for some demographic groups while improving it on average.

The obvious remedy is to calibrate separately for every combination of group and horizon, a strategy known as stratification. Each cell of the resulting grid gets its own threshold derived from its own calibration scores. But this is where the mathematics turns hostile. The finite-sample-corrected empirical quantile that conformal prediction requires is only defined when a cell contains at least the ratio (1−α)/α calibration scores, which amounts to nine points at the ninety percent level and nineteen at the ninety-five percent level. In realistic settings those thresholds are routinely violated. With seven demographic groups, nine forecast horizons, and a long-tailed group distribution whose smallest stratum holds roughly 1.5 percent of records, a calibration set of five thousand records leaves the smallest group with only about eight scores per cell, just below the cutoff. At stricter coverage levels, roughly half of the smallest-group cells fall below the threshold even with ten thousand records. When the quantile is undefined, the naive stratified interval becomes infinitely wide, which is useless for deployment.

Zou and colleagues frame this as a joint failure mode that neither marginal methods nor purely stratified ones address, and they answer it with a structured-shrinkage framework rooted in empirical Bayes statistics, the family of techniques made famous by Efron and Morris in the 1970s. Their primary method, called HACC, short for Horizon-Aware Conformal Calibration, replaces each cell’s raw empirical quantile with a weighted blend of that quantile and a structured anchor. The anchor is multiplicative: it assumes the residual scale in a given group-horizon cell approximately factorises into a group effect times a horizon effect, divided by the overall scale. This is the classical multiplicative-heteroscedasticity model from econometrics, and it means the anchor for a cell can be computed from pooled quantities, the group-level quantile times the horizon-level quantile divided by the marginal quantile. Well-sampled cells keep most of their local signal, while sparse cells lean heavily on the anchor, with the blending weight determined by the cell’s sample size and a shrinkage strength selected by cross-validation.

The theoretical contribution is a finite-sample perturbation bound showing that shrinking the threshold can degrade per-cell coverage by at most a quantity proportional to the shrinkage weight times the distance between the empirical quantile and the anchor. Crucially, a second theorem extends this bound to data-dependent shrinkage strengths, including the cross-validated value computed on the same calibration data, without requiring sample splitting or union bounds. The argument works because the perturbation analysis is pointwise in the calibration sample and never invokes the distribution of the selected strength, so the algorithm as actually implemented coincides with the algorithm as analysed. For cells too sparse to yield a defined quantile at all, the method falls back to the anchor, and coverage in those cells is governed entirely by how well the anchor approximates the true cell quantile.

The empirical evaluation is unusually thorough. On a synthetic benchmark with seven groups of wildly unequal sizes, horizon-dependent variance inflation, heavy-tailed residuals and covariate drift, the authors ran thirty paired seeds comparing their methods against six conformal baselines. The four marginal methods, including split conformal prediction, conformalized quantile regression, weighted conformal prediction and adaptive conformal inference, all hit the marginal target but posted minimum-group coverage between sixty-eight and seventy-two percent and worst-cell coverage as low as forty-five percent. The naive fully stratified method produced infinite-width intervals in twenty-three to thirty-three percent of seeds. HACC and its globally smoothed sibling, AdditiveFE, which fits a log-additive two-way model to the entire quantile matrix, achieved the highest worst-cell coverage and the most uniform coverage matrices among all finite-width methods, with no cell below eighty-four percent in the stress scenario.

The real-data experiment is perhaps the most compelling part. The authors applied their framework to the Our World in Data panel of under-five child mortality, forecasting the log-change in mortality for 179 countries across five horizons from one to ten years ahead, grouped by continent crossed with fertility tier, yielding fifteen intersectional groups and seventy-five group-horizon cells. Here the results were stark. The stratified baselines covered the smallest strata at only twenty-eight to thirty-one percent minimum-group coverage, with worst cells at zero to six percent, even though every cell was technically above the finite-sample threshold, because per-cell quantile estimates were simply too noisy. Every member of the shrinkage family improved on them by wide, bootstrap-significant margins at comparable interval width. Remarkably, on this panel the improvement grew monotonically with the amount of pooling: the anchor-only estimator reached seventy-two percent minimum-group coverage, outperforming even the cross-validated HACC, whose selection objective turned out to be blind to the sparsest cells because they contributed too few points per validation fold.

That last finding carries a practical lesson the authors state candidly. When the sparse cells are too small to be validated, cross-validation cannot see the benefit of shrinking them, and a stronger default, such as a large fixed shrinkage strength, the AdditiveFE variant, or the anchor itself, is the safer choice. They also tested robustness by deliberately misspecifying the anchor with sinusoidal interactions and localised variance shocks. HACC kept its worst-cell coverage within one point of its well-specified value, while Mondrian-style group stratification collapsed under the shock, and the cross-validation procedure adapted by shrinking less when the anchor fit poorly. A rolling-origin protocol, in which only labels observed before each deployment year could be used, confirmed that the ordering of methods survives genuine temporal drift, though with smaller margins, and showed that restricting calibration to the most recent five years of labels was the first-order remedy for drift, with shrinkage composing on top of it.

The implications reach well beyond global-health statistics. The authors point out that long-horizon degradation forecasting in engineering prognostics, such as battery state-of-health prediction and remaining-useful-life estimation, shares exactly the structure their method targets: residual distributions that widen with the forecast horizon and heterogeneous sub-populations, whether cell chemistries or engine units, that are unevenly represented in calibration data. Deep forecasters like the Temporal Fusion Transformer deliver strong point accuracy but no coverage guarantee of their own, and the conformal layers wrapped around them are only marginally calibrated. Because HACC is post-hoc and model-agnostic, it can sit on top of any such architecture, and it runs in a fraction of a second on a laptop CPU after the base model is fitted.

The study is also honest about its limits. The coverage bounds are marginal over the calibration draw and control only the magnitude of the shrinkage perturbation, not its sign. Equalised coverage is only one fairness criterion among several, and recent work shows that even balanced-coverage conformal sets can produce disparate downstream impact. The real-data evidence rests on a single panel with a fixed calendar split, and the group attribute must be observed at prediction time. Still, the core message is one that any organisation auditing a forecasting system should internalise: a marginal coverage report can conceal exactly the failures an audit exists to find, and the per-cell coverage matrix, not the headline number, is what deserves scrutiny. Structured shrinkage offers a principled, theoretically grounded way to make those matrices both visible and reliable, even when the cells that matter most are the ones with the least data.

Subject of Research: Sparse group- and horizon-conditional coverage auditing in conformal prediction via structured empirical-Bayes shrinkage

Article Title: Horizon-aware group-conditional conformal prediction: Post-hoc shrinkage for sparse multi-horizon coverage auditing

Article References: Zou, J., Feng, E., Li, P., & Wang, G. (2026). Horizon-aware group-conditional conformal prediction: Post-hoc shrinkage for sparse multi-horizon coverage auditing. Machine Learning with Applications, 26, Article 101038. https://doi.org/10.1016/j.mlwa.2026.101038

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.101038

Keywords: conformal prediction, prediction intervals, coverage auditing, empirical Bayes, shrinkage estimation, multi-horizon forecasting, group-conditional coverage, distribution-free guarantees, quantile regression, fairness in machine learning, child mortality forecasting, uncertainty quantification