Why Machines Overfit: New Survey Brings Order to the Chaos of Regularization

Every practitioner of machine learning knows the eerie moment when a model that seemed flawless in training stumbles on data it has never seen. The network has memorized rather than learned. The remedy for this affliction, regularization, has spawned thousands of papers and countless tricks, yet the field has grown faster than its theoretical foundations. A new survey published in the journal Machine Learning by Yaxing Huang, Chenhe Zhang, Renjun Xu and Qinghai Zhang, spanning institutions from Xinjiang University to Zhejiang University, confronts that imbalance directly. The work, which appeared in Volume 115 of the journal as article number 244, does something unusually ambitious for a survey: it first rebuilds the mathematical ground floor of regularization from first principles, and only then organizes the sprawling modern literature of loss-based regularizers into a coherent map.

The theoretical core of the paper begins with a definition that reaches back a century. Regularization, the authors argue, is best understood as the solution of ill-posed problems, a concept introduced by the French mathematician Jacques Hadamard in 1923. A problem is well-posed when a solution exists, is unique, and depends continuously on the input data. Learning from finite, noisy datasets fails all three tests at once: many different functions can fit the same training examples, tiny perturbations in the data can swing the fitted model wildly, and nothing guarantees a stable answer. From this vantage point, regularization is not an afterthought bolted onto training but the very mechanism that converts an impossible inverse problem into a tractable one.

To make this concrete, the survey examines the effectiveness of Tikhonov regularization in linear systems, the setting where everything can be worked out exactly. Tikhonov regularization, introduced by the Russian mathematician Andrey Tikhonov and familiar to statisticians as ridge regression through the 1970 work of Hoerl and Kennard, adds a penalty on the squared size of the solution to the fitting objective. Using the singular value decomposition, the authors show how the penalty suppresses the directions in which the data are least informative, damping the tiny singular values that would otherwise amplify noise into enormous errors. The analysis makes precise, in a setting free of neural network mysteries, exactly what a penalty term buys: a controlled trade of bias for stability.

The most consequential theoretical contribution is a consistency principle for the design of regularization methods, proposed to hold in both linear and nonlinear systems. The principle addresses a question that has quietly haunted the field: when a penalty helps in one architecture or task, when can we trust it to help in another? Rather than treating each regularizer as an isolated empirical trick, the survey argues that sound regularization should behave consistently with the underlying structure of the learning problem, so that the guiding principles established rigorously for linear systems carry over, in appropriately adapted form, to the nonlinear models that dominate modern practice. This reframing gives researchers a design criterion, not merely a menu of options.

With that foundation laid, the second half of the paper delivers what may become its most-used artifact: a comprehensive taxonomy of recent loss-based regularizers, the family of methods that modify the training loss itself rather than the architecture, the data pipeline, or the optimization algorithm. The taxonomy summarizes the prominent features of these regularizers, and it draws together strands of work that rarely appear in the same review. Classical sparsity penalties such as the lasso of Tibshirani and its nonconvex successors, including the minimax concave penalty and transformed l1 approaches, sit alongside group and exclusive sparsity formulations, low-rank and nuclear norm methods, and total-variation style penalties inherited from image processing.

The survey also gives careful attention to regularizers designed specifically for the deep learning era, where the classical capacity-control story falters. It covers penalty terms targeting the Lipschitz constant of networks, Jacobian and gradient-norm penalties that smooth the model’s response to its inputs, and sharpness-aware minimization, the influential technique that steers optimization toward wide, flat basins of the loss landscape. Work on flat minima stretching back to Hochreiter and Schmidhuber in 1997, and the finding of Dinh and colleagues that sharp minima can nonetheless generalize, illustrates the kind of unresolved tension the taxonomy is built to expose. Adversarial perturbation regularizers, persistent-homology-based topological penalties, mixup-style data interpolation losses and dropout all find their place in the same organized structure.

What makes this organization more than bibliographic housekeeping is the diagnosis embedded in the abstract: despite the numerous methods developed in a fast-growing literature, there is a lack of theoretical understanding and rigorous analysis of regularization. The survey is candid that the explosion of techniques has outpaced the theory explaining why they work and when they fail. By anchoring every entry in the taxonomy to the ill-posed-problems formulation and the consistency principle, the authors offer researchers a shared vocabulary for comparing methods that were previously justified by unrelated intuitions, from Bayesian priors to geometric smoothness to robustness against adversarial attack.

The paper does not rest on analysis alone. The authors report results of benchmark tests that confirm the effectiveness of representative loss-based regularizers, providing an empirical check that the theoretical framework identifies methods which genuinely deliver generalization gains in practice. The numerical experiments and the data supporting the manuscript are available from the authors upon request, in line with the journal’s transparency expectations. Co-first authors Huang and Zhang divided the labor, with Huang conducting the literature search, summarizing each surveyed method and producing the tables and figures, while Zhang carried out the mathematical proofs. Xu led conceptualization and the taxonomy, and Qinghai Zhang contributed the mathematical reasoning and manuscript writing, with support from the Fundamental Research Funds for the Central Universities.

The timing of the survey, received in 2023, revised in March 2026 and published in October 2026 after more than two and a half years of review and refinement, reflects a field in urgent need of synthesis. Modern models are trained on massive datasets with billions of parameters, and the celebrated generalization of such overparameterized networks famously defies classical bias-variance intuition, a puzzle highlighted by Belkin, Neyshabur, Zhang and their colleagues in influential analyses cited in the survey. Understanding which penalties push a model toward solutions that transfer, and which merely reshape the training landscape, has become a question with enormous economic weight, affecting everything from medical imaging systems to the reliability of language models deployed to hundreds of millions of users.

For the research community, the payoff of this work is likely to be felt in two ways. First, the rigorously analyzed linear case provides a template: students and researchers can now trace, step by step, how a penalty reshapes a solution through the singular spectrum, and carry that intuition into nonlinear settings with a stated principle rather than a vague analogy. Second, the taxonomy turns the design of new regularizers from an art into an engineering discipline with a consistency criterion to satisfy. If the authors are right that regularization should be treated as the principled resolution of ill-posedness, then the next generation of methods will not be judged solely by leaderboard margins on benchmark datasets, but by whether their penalties encode the right structural assumptions about the problem. In a discipline where the gap between memorization and understanding is the central mystery, bringing mathematical order to that gap may prove the most valuable regularization of all.

Subject of Research: Loss-based regularization methods for improving model generalization in machine learning

Article Title: A Survey on Loss-based Regularization for Model Generalization

Article References: Huang, Y., Zhang, C., Xu, R., & Zhang, Q. (2026). A Survey on Loss-based Regularization for Model Generalization. Machine Learning, 115(10), Article 244. https://doi.org/10.1007/s10994-026-07139-2

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07139-2

Keywords: regularization, model generalization, machine learning, Tikhonov regularization, ill-posed problems, loss functions, sparsity, deep learning, consistency principle, empirical risk minimization, taxonomy, benchmarking