The heart of Gaussian process training lies in its hyperparameters, the knobs that define how the model believes data points relate to one another. A kernel function encodes assumptions about smoothness, correlation length, signal strength and noise level, and these quantities are typically learned by maximizing the marginal likelihood, a quantity that balances how well the model fits the data against how complex it becomes. For decades this procedure has been regarded as statistically principled and, under ideal Gaussian assumptions, it genuinely is. What Sedai and Medda show, however, is that the estimators produced by this procedure possess what statisticians call an unbounded influence function, a formal certificate that a single extreme observation can exert arbitrarily large leverage on the fitted kernel parameters.

The influence function is a classic concept from robust statistics, introduced by Frank Hampel in the 1970s as a diagnostic for measuring how much an infinitesimal speck of contamination at any point in the data distribution shifts an estimator. If the influence function is bounded, no single observation can dominate the result no matter how extreme it is. If it is unbounded, a lone outlier can drag the estimate anywhere. The authors adapt this framework to the Gaussian process setting with a subtle twist: because the joint log marginal likelihood couples all observations through the kernel covariance matrix, it does not decompose into a sum of independent per-point terms the way classical M-estimators do. Their solution is a leave-one-in contamination construction, in which the perturbation is represented by the predictive log-likelihood of an added data point conditional on the observed sample.

From this construction they derive a closed-form expression for the influence function of the hyperparameter estimator, factoring it into two interpretable pieces: the global curvature of the objective, captured by the Hessian of the log marginal likelihood, and a local score contribution from the contaminated point. That local score depends on the residual between the contaminated response and the model’s predictive mean, on the predictive variance at the contaminated input, and on where the input sits relative to the observed design. When the input lies far from the training data, the kernel coupling weakens and the geometric influence attenuates, though variance-related parameters such as signal and noise variance remain affected even then.

The central result is stark. Under the standard Gaussian likelihood, the contamination score grows quadratically in the magnitude of the response, so the influence function diverges as the outlier’s value tends toward infinity. The authors prove this formally in a proposition requiring only mild regularity conditions, and they show that the conclusion is automatic whenever the model includes a signal variance or noise variance parameter, which virtually every practical Gaussian process does. They further demonstrate that the result survives log-parameterizations and automatic relevance determination kernels, closing the escape hatches that practitioners might hope would soften the blow. In Hampel’s terminology, maximum likelihood training of Gaussian processes is nonrobust by construction.

Crucially, the paper does not stop at diagnosis. The authors extend the influence function framework to the robust alternatives that have long circulated in the Gaussian process literature, including heavy-tailed likelihoods such as the Student-t and Laplace distributions, and robustified objectives built on the Huber loss and its smooth cousin, the Pseudo-Huber or Charbonnier loss. For each case they establish the conditions under which the influence function becomes bounded. The unifying principle is elegant: if the derivative of the pointwise loss with respect to the mean prediction remains bounded over all residuals, then the contamination score is bounded, and so is the influence function, provided the local curvature of the objective stays finite and nonsingular. This connects Gaussian process training directly to the classical theory of M-estimation, recasting it as a kernelized, probabilistic analogue of robust regression.

The empirical validation is unusually thorough. The authors first verify that their analytic influence function matches an independent automatic-differentiation implementation to machine precision, that it is insensitive to the small ridge regularization used for numerical conditioning, and that it reproduces the true first-order response of the optimizer to small data perturbations as measured by finite differences. On a synthetic one-dimensional benchmark with two hundred points, they then compare Gaussian, Student-t, Huber and Pseudo-Huber training under single-point stress tests and controlled contamination. The scatter of per-point influence norms against whitened residuals tells the story visually: the Gaussian fit reaches large influence values at modest residual sizes, while Huber and Pseudo-Huber accommodate residuals several times larger within a comparable influence range, with Student-t sitting in between.

Perhaps the most surprising empirical finding concerns curvature. Clipping the score function, as Huber does, turns out to be necessary but not sufficient for stable hyperparameter behaviour. Because Huber’s score is flat in the tail, contaminated points contribute no data curvature to the Hessian, and with enough contamination the smallest eigenvalue of the curvature matrix can collapse toward zero or below, destabilizing the local solve and sometimes amplifying influence beyond even the Gaussian fit. The Student-t likelihood, whose score redescends toward zero while retaining curvature, preserved positive curvature in every contaminated run and produced the smallest hyperparameter drift, whereas Huber and Pseudo-Huber lost positive curvature in most runs. The lesson is that robustness depends on both suppressing the direct pull of outliers and maintaining a well-conditioned training objective.

The practical implications ripple outward. In experiments on real datasets including Kin8nm and superconductivity data, the authors show that robust objectives cost essentially nothing when the Gaussian model is appropriate, matching point accuracy and calibrated coverage after a single variance calibration. When contamination or heavy tails appear, however, the Gaussian fit degrades fastest, compensating by inflating its noise estimate and widening its intervals, while Student-t and Pseudo-Huber keep hyperparameters stable and uncertainty honest. The authors also verify that their influence analysis carries over to sparse variational Gaussian processes, the scalable variant used with large datasets, finding that the analytic influence function closely predicts the optimizer’s actual movement.

For a field that increasingly deploys Gaussian processes in safety-critical emulation, environmental modelling and Bayesian optimization, the message is hard to ignore. The fragility of marginal-likelihood training is a structural property of the estimator, not an artefact of finite precision or poor implementation, and it can be measured, predicted and repaired. Sedai and Medda’s work gives practitioners both the warning and the toolkit: check the influence structure of your fitted model, prefer heavy-tailed likelihoods when outliers are plausible, and remember that a bounded score alone will not save you if the curvature of your objective quietly disappears. Robustness, it turns out, is a two-part bargain, and Gaussian process training has only recently learned to sign it.

Subject of Research: Robustness analysis of Gaussian process hyperparameter estimation using influence functions

Article Title: Influence functions for Gaussian process hyperparameter estimation

Article References: Sedai, A., & Medda, F. (2026). Influence functions for Gaussian process hyperparameter estimation. International Journal of Data Science and Analytics, 22(1), Article 316. https://doi.org/10.1007/s41060-026-01285-5

Image Credits: AI Generated

DOI: 10.1007/s41060-026-01285-5

Keywords: Gaussian processes, influence functions, hyperparameter estimation, robust statistics, marginal likelihood, outliers, Student-t likelihood, Huber loss, M-estimation, Bayesian inference, kernel methods, machine learning

The heart of the problem is mathematical. For the simple textbook models used in introductory statistics courses, Bayes factors can be computed exactly. But real experiments in cognitive science typically involve linear mixed-effects models with random effects for subjects and items, hierarchical structure, and multiple fixed effects. For these models, the Bayes factor requires evaluating an integral over all possible parameter values, an integral that has no closed-form solution. Researchers therefore depend on numerical approximations. One popular strategy, implemented in the R package bridgesampling, first estimates the posterior distribution of each competing model using Markov-chain Monte Carlo (MCMC) sampling and then uses those posterior samples to approximate the marginal likelihood, the quantity whose ratio defines the Bayes factor. Every step of this pipeline introduces potential error, and until now there has been no systematic way to know whether the final number on the screen corresponds to the true Bayes factor or to a biased approximation of it.

Schad and Modrák attacked this question with a technique called marginal simulation-based calibration, or marginal SBC, which the authors had previously developed. The logic is elegant. The researchers define a prior probability for each competing hypothesis, for example a 50 percent chance that the null model and the alternative model are each true. They then repeat a simulation many times: sample a model from that prior, sample parameter values from the parameter priors, generate artificial data from the sampled model, and compute the Bayes factor for the simulated data using the method under scrutiny. If the Bayes factor computation is correct, then the average posterior model probability recovered across all the simulations should match the prior model probability that was used to generate the data. If the average posterior drifts away from the prior, the estimator is biased, either liberally, exaggerating evidence for effects, or conservatively, understating it. The authors supplemented this test with reliability diagrams, which visualize whether posterior probabilities correctly predict which model actually generated each simulated dataset.

The team applied this procedure to three experimental designs that dominate cognitive psychology and psycholinguistics. The first was a 2×2 repeated measures design inspired by the two-step decision-making task used to study model-based and model-free reinforcement learning, with random effects for subjects only. The second was a Latin square design with a single two-level fixed factor and crossed random effects for both subjects and items, analyzed with a Bayesian generalized linear mixed model with a lognormal likelihood. The third was a full 2×2 Latin square design with crossed random effects for subjects and items, testing two main effects and their interaction. In each case, the null model was created by setting one fixed effect to exactly zero, and the full model was compared against it using the brms package for model fitting in combination with bridgesampling for the Bayes factor computation, using the warp3 variant of the bridge sampler and 10,000 MCMC iterations with 2,000 warm-up iterations.

The results split cleanly along one diagnostic line: whether or not the bridge sampling algorithm issued a warning message. In design 1, the algorithm flagged 122 of 1,200 model fits, roughly 20 percent, with the message that the log marginal likelihood could not be estimated within the maximum number of iterations and that the estimate might be more variable than usual. For the simulations without warnings, the calibration checks were reassuring. Bayesian t-tests comparing the average posterior model probability against the prior found default Bayes factors exceeding 10 in favor of no difference, and the reliability diagrams showed posterior probabilities aligned with the true generating model. In other words, when the software stayed silent, the Bayes factors were accurate. When a warning appeared, however, the picture darkened: the reliability diagrams deviated visibly from the diagonal, indicating miscalibrated and biased estimates.

The pattern repeated across the other designs, sometimes more dramatically. In the Latin square design with crossed random effects, only 3 percent of fits produced warnings, and the Bayes factors proved accurate. But in the full 2×2 design with crossed random effects, warnings appeared in 37 percent of simulations, and the consequences were serious. When the prior probability of the null hypothesis was set high at 0.8, the biased estimates pushed posterior model probabilities for the alternative hypothesis too high, producing liberal tests that could declare evidence for an effect even when the true effect was zero. When the researchers flipped the setup and gave the alternative hypothesis a prior probability of 0.8, the bias reversed direction, yielding conservative tests that could hide genuine effects. The culprit was not a systematic upward or downward distortion but extreme noise: the variance of the estimated log marginal likelihoods ballooned in the warned simulations, and floor and ceiling effects then pushed posterior probabilities toward 0.5, making the resulting Bayes factors essentially uninformative.

The authors also probed whether the problem could simply be computed away. Increasing the MCMC sample size from 10,000 to 40,000 iterations reduced the warning rate only marginally, from 37 percent to 32.5 percent, a change well within expected sampling variability. Running 1,000 SBC simulations instead of 200, however, sharpened the conclusions considerably: with that statistical power, the evidence for accuracy in warning-free simulations grew strong, with default Bayes factors above 10 favoring no bias, while the evidence for bias in warned simulations became overwhelming, with Bayes factors exceeding 10,000. A re-analysis across varying numbers of SBC runs showed that evidence for bias in warned simulations emerged quickly, with fewer than 100 simulation runs, whereas evidence for accuracy accumulated more slowly. The practical lesson is that researchers can detect dangerous Bayes factors relatively cheaply, but confirming that a computation is clean requires a somewhat larger investment of simulations.

What causes the warnings in the first place? The authors found that warned datasets were precisely those that forced Stan’s Hamiltonian Monte Carlo sampler to take very small steps and explore deep trajectories, indicating a highly irregular posterior geometry. Problematic cases tended to involve very small fitted residual standard deviations or random effect standard deviations close to zero, creating weak non-identifiability between the overall effects and the subject-level random effects. In such situations, the authors suggest, reparameterizing the model can help: dropping the random effect for the interaction, or imposing sum-to-zero constraints on random intercepts and slopes, resolved the warnings in some cases. But these fixes are not guaranteed, and the authors are candid that researchers facing persistent warnings may need to turn to alternative approaches altogether, such as the Savage-Dickey method, the BayesFactor package, region of practical equivalence decisions, or posterior credibility intervals, after verifying that those alternatives are themselves accurate for the analysis at hand.

The broader message is both a validation and a caution. On the one hand, the study provides real evidence that brms and bridgesampling deliver accurate Bayes factors for the factorial designs that dominate experimental psychology and psycholinguistics, provided the computation proceeds without complaints. On the other hand, it demolishes the comfortable assumption that software output can be taken at face value. A warning message that many users skim past, or dismiss as a minor numerical hiccup, turns out to mark the boundary between calibrated evidence and numbers that are biased, highly variable, and potentially capable of steering scientific conclusions in the wrong direction. The authors argue that marginal SBC should become a routine part of the Bayesian workflow: before trusting Bayes factors from any particular combination of design, model, and priors, researchers should check the computation’s accuracy through simulation. It is a modest additional burden, but one that could spare the field a great deal of misplaced confidence in numbers that look precise and are anything but.

Subject of Research: Accuracy of Bayes factor-based null hypothesis tests in linear mixed-effects models assessed with simulation-based calibration

Article Title: How accurate are Bayes factor-based null hypothesis tests? A simulation study

Article References: Schad, D. J., & Modrák, M. (2026). How accurate are Bayes factor-based null hypothesis tests? A simulation study. Behavior Research Methods, 58(10), Article 282. https://doi.org/10.3758/s13428-026-03168-w

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03168-w

Keywords: Bayes factors, null hypothesis testing, simulation-based calibration, bridge sampling, linear mixed-effects models, brms, Bayesian statistics, marginal likelihood, psycholinguistics, factorial designs, MCMC, statistical software