The core mathematical observation is deceptively simple. In kernel ridge regression, the prediction for the training data can be written as a weighted sum of the eigenvectors of the kernel matrix, where each weight is the projection of the target vector onto that eigenvector, scaled by the corresponding eigenvalue divided by the eigenvalue plus the regularization parameter. This spectral decomposition means that whatever the model learns is confined to the span of the kernel’s eigenvectors, and the contribution of each direction depends jointly on how strongly the target aligns with it and how large its eigenvalue is. A direction with a huge eigenvalue but poor alignment contributes almost nothing, while a weakly aligned trailing eigenvector can quietly dominate the error.

From this decomposition, the authors derive a striking result about reconstruction, the model’s ability to reproduce its own training labels. They prove that as long as the kernel matrix has sufficient rank, the reconstruction error can be made arbitrarily small simply by shrinking the regularization parameter, an algebraic fact that holds regardless of the data distribution. In high-dimensional settings, near-zero training loss is therefore nearly always achievable. The researchers demonstrate this empirically with synthetic targets built from eigenvectors of a linear kernel: once the input dimension exceeded roughly 500, reconstruction errors collapsed toward zero across the board. The practical implication is unsettling for practitioners who treat a low training loss as evidence of a healthy model, because the training error carries almost no predictive power for test performance.

The study goes further and shows that near-zero reconstruction error is not even a reliable sign of overfitting. In experiments on MNIST images of the digits four and nine, the team constructed learning targets aligned either with the top eigenvectors of the kernel matrix or with its trailing eigenvectors. When the target was aligned with the leading directions, both training and test predictions were nearly perfect. When it was aligned with the trailing directions, the model still reproduced the training data almost exactly, yet its test performance collapsed. The same pattern held on CIFAR10, FashionMNIST, and SVHN2, confirming that the gap between training and test behavior is governed by which part of the eigenspectrum the target inhabits, not by the training loss itself.

To explain these observations, the authors turn to matrix perturbation theory, treating the kernel matrix built from any finite sample as a perturbed version of a hypothetical optimal kernel matrix of the same size. Using classical tools such as the Davis-Kahan theorem, they derive an explicit finite-sample bound on the generalization error expressed in terms of three quantities: the alignment of the learning targets with the eigenvectors, the magnitude of the eigenvalues, and the gaps between consecutive eigenvalues. Strong generalization, the theorem says, requires increasing any of these. Targets aligned with the top eigenvectors benefit from all three at once, because leading eigenvalues are large, well separated, and estimated most accurately from finite data.

This perturbation perspective yields a reinterpretation of the familiar bias-variance tradeoff. In their framing, bias arises when the top eigenvectors cannot represent the learning targets, while variance arises when the targets must be represented by trailing eigenvectors, whose sample estimates are unstable and noisy. Overfitting in kernel ridge regression, on this view, is essentially a failure of the kernel to distinguish information from noise, either because the data are fundamentally too noisy or because the kernel choice is suboptimal. The finding echoes recent work on benign overfitting, but grounds it in concrete spectral quantities that can be computed from a single training set.

Perhaps the most provocative claim concerns regularization. The ridge parameter, long regarded as the main lever for controlling model complexity in the non-parametric setting, turns out to have minimal influence on generalization for most learning targets. Because leading eigenvalues are typically large relative to any reasonable regularization strength, the parameter barely changes their weights. It matters only when targets align with trailing eigenvectors, precisely the regime where generalization is already poor. The authors summarize this bluntly: regularization is more about salvaging bad performance than improving good performance, and its primary role is mitigating numerical instability rather than shrinking the hypothesis space. Their experiments confirm the pattern, showing that stronger regularization often increases generalization error except in the trailing-eigenvector regime.

The paper also delivers a practical payoff in the form of a simple empirical risk estimator, built from the projections of the target onto each eigenvector weighted by the inverse eigenvalue gaps. When benchmarked against the Kernel Alignment Risk Estimator of Jacot and colleagues and the eigenlearning estimator of Simon and colleagues on MNIST and FashionMNIST, the new estimator tracked the true testing error most closely in the leading eigendirections, roughly up to index 250 for MNIST and 300 for FashionMNIST. Existing estimators significantly underestimated the risk in trailing eigendirections, where the new bound, though loose, at least captured the upward trend. The authors note that sharper eigenvector perturbation bounds could improve the estimator further.

Methodologically, the work distinguishes itself from prior analyses by avoiding Gaussian assumptions on the sampling operator, an assumption that recent critiques have shown to be unrealistic in practice, and by focusing on the finite-sample regime rather than asymptotic limits. The comparison with a hypothetical optimal training set, chosen to minimize misalignment with the actual sample, allows the authors to state their bounds without distributional assumptions. They also highlight a connection to empirical observations in deep learning, noting that well-generalized neural networks tend to have weight matrices with isolated eigenvalues, consistent with the emphasis on eigenvalue gaps in their theory.

The authors are candid about limitations. Their analysis assumes noiseless labels, does not handle missing data, and produces bounds that may not be tight, since they are built from a chain of inequalities. Even so, the message is likely to resonate widely: what a kernel machine learns is determined not by how well it fits its training data, but by whether the structure of the data and the kernel conspire to place the target in the stable, well-estimated part of the spectrum. For a field increasingly concerned with understanding why overparameterized models generalize at all, that reframing turns the spotlight away from the loss curve and toward the geometry of the eigenspectrum.

Subject of Research: Eigen-alignment and generalization in kernel ridge regression

Article Title: On Kernel Eigen-Alignments of KRR: Reconstruction and Generalization

Article References: Liu, Y., Fokoue, E., Lange, R., & Krutz, D. (2026). On Kernel Eigen-Alignments of KRR: Reconstruction and Generalization. Machine Learning, 115(10), Article 238. https://doi.org/10.1007/s10994-026-07177-w

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07177-w

Keywords: kernel ridge regression, eigenvectors, generalization, machine learning, kernel alignment, matrix perturbation theory, regularization, overfitting, learning theory, spectral bias, eigenvalues, risk estimation

The core difficulty is easy to state but hard to solve. In classical binary classification, an algorithm sees both labeled positives and labeled negatives and learns a decision boundary between them. In PU learning, negative labels simply do not exist. Instead, the algorithm receives a small set of confirmed positives and a large unlabeled pool. Because the unlabeled pool is dominated by negatives but contaminated with positives, treating every unlabeled example as negative injects label noise into training. Traditional approaches either try to identify reliable negatives from the unlabeled set before training, or they incorporate assumptions about the class prior, the underlying proportion of positive examples in the data. Both strategies become brittle when the dataset is severely imbalanced, meaning positives may account for only one to a few percent of all examples.

Zavitsanos and Paliouras observed that in imbalanced PU data, the mathematics of existing risk estimators works against the practitioner. The estimators used by state-of-the-art methods, such as the unbiased PU estimator (uPU) and its non-negative successor (nnPU), contain a term weighted by the positive class prior. When that prior is tiny, the contribution of the few labeled positives nearly vanishes, and the training objective is dominated by the unlabeled data. The model therefore learns to be excellent at recognizing negatives, and mediocre at catching the rare positives that matter most. Matters worsen under the Probabilistic Gap assumption, a realistic refinement of labeling theory in which positive examples that resemble negatives are the least likely to have been labeled. The very positives a model most needs to learn from are the ones most likely to be hidden in the unlabeled pool.

The researchers’ answer is a new empirical risk estimator they call iFPU, for imbalanced focused PU learning. The key ingredient is focal loss, a function originally developed in computer vision for dense object detection, where positive targets are similarly swamped by an enormous number of easy negatives. Focal loss reshapes the standard cross-entropy objective by multiplying each example’s loss by a factor that shrinks as the model’s confidence in the correct answer grows. Easy examples, which the model already classifies correctly with high probability, contribute almost nothing to the gradient. Hard examples, sitting near the decision boundary, receive exponentially amplified weight, controlled by a focusing parameter gamma. The result is that training concentrates its capacity on exactly the boundary cases that define the Probabilistic Gap.

Incorporating focal loss into a PU risk estimator is technically delicate. The authors derive a non-negative risk estimator that combines three terms: the focal loss of the labeled positives treated as positive, an unbiased correction subtracting the focal loss the positives would incur if treated as negative, and the focal loss of the unlabeled pool treated as negative. A max operator clamps the correction at zero to prevent the notorious problem of negative empirical risk, which causes overfitting in neural networks trained with earlier unbiased estimators. When the correction term goes negative within a training mini-batch, the method performs a step of gradient ascent instead, deliberately nudging the model away from overfitting that batch. The whole procedure slots into standard stochastic optimization, requires no preprocessing, resampling, or manipulation of the data, and can be attached to essentially any classifier trained by cost minimization, from multilayer perceptrons to gradient-boosted trees.

The authors also supply a rigorous theoretical analysis. They prove that the population-level iFPU risk is identical to the focal classification risk under full supervision, for any labeling mechanism, by virtue of a mixture identity between the unlabeled distribution and the class-conditional densities. They further characterize the bias that appears when labeled positives are not selected completely at random, showing that this bias depends only on the labeling propensity and is orthogonal to the choice of loss function. Under the SCAR assumption, they establish an estimation error bound that vanishes at the standard statistical rate, proportional to the inverse square root of the number of labeled positives and unlabeled examples combined, using Rademacher complexity tools. Crucially, they show focal loss is Lipschitz continuous and bounded on a restricted score range, which makes these guarantees possible. Although focal loss itself introduces a deliberate bias by reweighting errors, it remains classification-calibrated, meaning the Bayes-optimal classifier under focal loss coincides with the Bayes-optimal classifier under zero-one loss.

Empirically, the method was tested on 14 publicly available benchmark datasets for imbalanced binary classification, spanning positive rates from roughly 1 to 14 percent. The experimental design was deliberately demanding: positive examples were progressively hidden in the unlabeled pool at rates of 25, 50, and 75 percent, under both the SCAR and the more realistic SAR labeling assumptions, generating 840 experimental runs in total. Notably, the authors avoided hyperparameter tuning altogether, using the recommended default gamma of 3, because tuning in PU settings is itself fraught with assumptions due to the absence of negatively labeled validation data. Under SCAR, iFPU outperformed the neural risk estimators uPU, nnPU, and i-NNPU, and when paired with an XGBoost classifier it matched or exceeded strong competitors including the two-step NNIF anomaly-detection method, PU Hellinger Decision Trees, the label-bias estimation method LBE, and the SAREM expectation-maximization framework, trailing only slightly behind the PU Hellinger Random Forest ensemble.

The picture shifts decisively in favor of iFPU under the more realistic SAR assumption, where positives resembling negatives are less likely to be labeled. Here the competing methods degrade noticeably, while iFPU maintains its performance, and in the hardest scenario, with only 25 percent of positives labeled, it ranks first overall, surpassing the PU Hellinger Random Forest by five percentage points in PR-AUC. Statistical tests confirmed significant differences among methods, and paired comparisons showed iFPU significantly outperforming all alternatives in the most challenging configuration. A sensitivity analysis demonstrated that the method remains robust even when the class prior is misspecified by factors of two or four in either direction, degrading meaningfully only when the prior is severely underestimated. On the three most imbalanced datasets, Cover, Poker, and Satellite, iFPU showed clearly higher mean and median PR-AUC than its strongest rival.

To demonstrate real-world value, the researchers applied their method to financial misstatement detection, a problem where PU data arise naturally. Auditors and regulators typically discover accounting misstatements years after reports are filed, and often only a fraction of misstatements have been identified when a model is trained. Using data on publicly traded US companies spanning 2000 to 2014, with 47,086 firm-year records described by 28 financial indices and derived accounting features, the team simulated realistic detection delays in which approximately 40 percent of positive training labels were missing at training time. Misstatements, whether deliberately concealed frauds or subtle errors resembling normal accounts, fit the Probabilistic Gap assumption perfectly. Models built on a TabTransformer architecture with gated MLP modules and equipped with the calibrated iFPU risk achieved R-precision scores roughly three times higher than prior baselines such as RUSBoost, and outperformed both earlier specialized models and the PU Hellinger Random Forest, setting a new state of the art in this application.

The significance of this work extends beyond any single benchmark. By combining a principled risk-estimation framework with a loss function engineered for imbalance, the authors show that PU learning can be made practical in exactly the conditions that dominate high-stakes applications, from disease gene identification to fraud detection, where positives are rare, partially labeled, and deceptively similar to negatives. The method’s plug-and-play compatibility with modern classifiers, its robustness to prior misspecification, and its theoretical grounding distinguish it from heuristic preprocessing pipelines. The authors point to extensions toward semi-supervised and multi-class settings, integration with pre-trained tabular foundation models such as TabPFN, and output calibration via temperature scaling as promising future directions. For now, iFPU offers practitioners a rare commodity in weakly supervised machine learning: a method whose assumptions match reality rather than convenience.

Subject of Research: Positive-unlabeled machine learning from highly imbalanced datasets

Article Title: Focused PU learning from imbalanced data

Article References: Focused PU learning from imbalanced data. (n.d.). https://doi.org/10.1007/s10618-026-01264-1

Image Credits: AI Generated

DOI: 10.1007/s10618-026-01264-1

Keywords: PU learning, imbalanced classification, weakly supervised learning, focal loss, risk estimation, machine learning, fraud detection, financial misstatement detection, XGBoost, SCAR assumption, SAR assumption, data mining