Stop trusting single accuracy scores: why ML models need distributional evaluation

Every machine learning practitioner knows the ritual: train a model, run it on a held-out test set, and report a single number — an accuracy of 0.91, a root mean squared error of 10.4. That number then travels through papers, dashboards, and boardroom slides as if it were a definitive statement about the model. A new study argues that this ritual is quietly misleading us. Machine learning training is a stochastic process, shaped by random weight initializations, random data splits, random augmentation, and randomized hyperparameter search, and a single performance figure is just one draw from an unknown underlying distribution. Treating it as the whole story, the authors contend, is like judging a drug trial from a single patient.

The research, published in Machine Learning with Applications by Christoph Lehmann and Yahor Paromau, proposes a deceptively simple reframing: view ML training as a controlled scientific experiment with repeated measurements. By varying one confounding factor at a time — the train–test split, the initial weights, the data augmentation scheme, the dropout configuration, or the hyperparameter optimization procedure — while holding everything else fixed via random seed control, each training run becomes a sample from the distribution of a target metric of interest. Repeating the run with different seeds yields repeated measurements, and with them the tools of classical inferential statistics become applicable: quantiles, confidence intervals, and explicit uncertainty quantification.

The scale of the empirical work is unusual for this kind of methodological study. The authors ran roughly 500 to 1,000 seed-controlled training repetitions for each source of variation across three real-world use cases: image classification of Simpsons characters with convolutional neural networks, the standard CIFAR10 benchmark with a VGG11 architecture, and a regression task predicting the critical temperature of superconductors using deep neural networks and gradient boosting trees. The effort consumed approximately 19,000 GPU hours on NVIDIA A100 and H100 accelerators, producing empirical performance distributions dense enough to serve as near-ground-truth references against which small-sample statistical methods could be validated.

The core statistical move is a shift from means to quantiles. A quantile answers a risk-oriented question that a mean cannot: the 25% quantile of accuracy tells you the accuracy falls below that threshold in only a quarter of runs, while the 90% quantile of RMSE tells you the error exceeds that value in just one run out of ten. The authors draw an explicit parallel to Value-at-Risk in financial regulation, where a quantile of the loss distribution defines the threshold exceeded only with a small pre-specified probability. For high-stakes applications — medicine, banking, criminal justice — this threshold language maps naturally onto acceptance criteria: a model can be required, in advance, to keep its error below a bound with a stated probability, something a mean-based evaluation cannot certify.

Quantiles, however, are harder to estimate than means. They are sensitive to the shape of the distribution, and the more extreme the quantile level, the more data is needed. The study characterizes this trade-off precisely. Exact nonparametric confidence intervals, built from order statistics and the binomial distribution, exist only when the sample size clears a minimum that grows steeply toward the distribution’s tails: estimating a 10% quantile with 90% confidence requires at least 22 runs, while a 1% quantile demands 230. Asymptotic intervals based on the normal approximation of sample quantiles have their own requirements, and the authors show a clever sign transformation — analyzing the negated metric — that converts a costly lower-tail quantile into a cheaper upper-tail one, cutting the required sample size for a 10% quantile from 42 to 25 runs.

For situations where even those minimums cannot be met, the authors evaluate a semiparametric bootstrap that extrapolates into the distribution’s tails using the spacing of the outermost order statistics. This method remains applicable where the closed-form intervals are undefined, at the price of extrapolation that may not match the true tail behavior. Across 42 experimental settings and a simulation study spanning skewed Beta distributions, normal mixtures, and Gumbel-distributed errors, the asymptotic nonparametric interval emerged as the practical default, offering a favorable balance of coverage, interval length, and simplicity. Bootstrap intervals remained a usable fallback for very small samples, though with coverage dropping to around 0.85 and noticeably wider intervals.

The practical payoff is demonstrated with two worked examples. In the Simpsons classification task, two training configurations — one varying hyperparameter optimization, one varying data augmentation — had nearly identical mean accuracies of 0.874 and 0.872, indistinguishable by any mean-based comparison. But their 10% quantile confidence intervals, estimated from just 25 runs each, did not overlap at all: [0.870, 0.873] for hyperparameter optimization versus [0.860, 0.868] for data augmentation. The hyperparameter-optimized configuration was substantially more resistant to producing poor runs, a fact invisible in the means. Against a deployment requirement that accuracy not drop below 0.87 in more than 10% of cases, only one configuration qualified — a statistically grounded model selection decision that point estimates could never support.

The regression example on superconductors showed the complementary value of the framework. A deep neural network and a gradient boosting tree differed clearly in mean RMSE, 10.47 versus 9.59, and their confidence intervals for the mean, the 10% quantile, and the 90% quantile were all cleanly separated with similar widths. That uniform pattern, the authors note, is the empirical signature of a simple location shift between the two distributions — and its absence of tail-specific effects is itself informative, justifying a mean-based comparison in this case. Across 2,000 repeated paired draws, the quantile intervals separated the two models in 92 to 99 percent of draws at n=25, while the mean separated them essentially always.

The study also distills concrete guidance for practitioners working under realistic compute budgets. With fewer than 10 repetitions, only a t-interval for the mean is advisable. Between 10 and 15 runs, exact nonparametric intervals for the quartiles become feasible, with a maximum recommended confidence level of 0.9. At 25 runs, asymptotic intervals with the sign transformation extend coverage to the 10% and 90% quantile levels, and at 50 or more runs, extreme quantiles such as the 5% and 95% levels come within reach. Interval lengths themselves carry diagnostic value: a median interval markedly longer than the mean’s t-interval signals low density near the center and hints at bimodality, while asymmetric tail intervals reveal skewness. The authors also caution that metrics like accuracy are formally discrete, computed on a grid set by test-set size, and recommend inspecting the fraction of distinct values in a sample before trusting quantile estimates.

What emerges is a bridge between two communities that rarely speak. Statisticians have long treated experimental outcomes as random variables deserving of distributional analysis; machine learning practice, optimized for leaderboard positions and single benchmark numbers, has largely resisted that view. This work shows that the statistical machinery is not merely compatible with modern deep learning — it is affordable. Even 10 to 25 seed-controlled runs, a modest addition to any training budget, unlock tail behavior, stability assessment, and requirement-driven evaluation that no point estimate can provide. As models move into domains where a single bad run can mean a misdiagnosis or a denied loan, the authors’ message lands with force: the average is not the answer, and the distribution is the evidence.

Subject of Research: Distributional uncertainty quantification in machine learning performance evaluation using quantile estimation and confidence intervals

Article Title: Beyond point estimates: Distributional uncertainty in machine learning performance evaluation

Article References: Lehmann, C., & Paromau, Y. (2026). Beyond point estimates: Distributional uncertainty in machine learning performance evaluation. Machine Learning with Applications, 26, Article 101033. https://doi.org/10.1016/j.mlwa.2026.101033

Image Credits: AI Generated

DOI: Not provided

Keywords: machine learning, model evaluation, uncertainty quantification, confidence intervals, quantiles, statistics, reproducibility, deep learning, risk assessment, benchmarking, hyperparameter optimization, experimental design