Adam Optimizer’s Neural Network Training Regimes Mapped Through Observable Measurements

Adam, the adaptive optimization algorithm that underpins much of modern deep learning, has long been treated as something of a black box whose internal settings are tuned by habit rather than understanding. A new open-access study published in Neural Processing Letters now offers one of the most detailed empirical pictures yet of how the optimizer’s two central hyperparameters shape the global behavior of neural-network training, revealing a hyperparameter landscape that is fragmented, sensitive and, at its boundaries, strikingly complex.

The research, carried out by S. Sveleba, I. Katerynchuk, I. Kunyo and M. Dufanets of Ivan Franko National University of Lviv together with N. Sveleba of the European University in Kiev, introduces an observable-based framework for mapping training regimes in the two-dimensional space defined by the exponential decay coefficients of Adam’s first and second gradient moments, commonly written as β₁ and β₂. These coefficients control how quickly the optimizer’s running estimates of the gradient mean and gradient variance are forgotten, and they are almost always left at their default values in practical work. The new study treats them instead as axes of a map on which the qualitative character of training can be charted.

The central idea of the paper is to characterize neural-network optimization not by the raw trajectory of millions of network weights, which is impossible to visualize or interpret directly, but through low-dimensional observable fields derived from the training-error dynamics of individual neurons. Optimization is framed as an iterative parameter-update process evolving in a high-dimensional parameter space, and the behavior of that process is then projected into a small set of measurable quantities that capture how training errors evolve, how stable they are over time, how aligned different parts of the network are, whether the dynamics are anisotropic, and how the training process unfolds over time.

Rather than relying on any single diagnostic, the authors combine seven complementary empirical descriptors into a composite regime map. Each point of the (β₁, β₂) plane is assigned a regime label based on these observables, and the resulting maps reveal a strikingly fragmented hyperparameter landscape. Four broad classes of behavior emerge: stable regimes in which training proceeds smoothly, oscillatory regimes in which the error dynamics swing without settling, slow-learning regimes in which progress is made only at a glacial pace, and irregular regimes in which no consistent pattern can be discerned. The practical implication is that small changes in the moment coefficients can move a network from one qualitatively different training regime to another, even though the loss surface and the data remain identical.

The experiments span datasets of increasing complexity, beginning with simple printed-digit patterns and progressing through Fashion-MNIST, CIFAR-10 and CIFAR-100, using multilayer neural-network architectures of varying width and depth. This staged design allows the researchers to trace how the regime structure responds to the difficulty of the learning problem and to the capacity of the model. The overall trend is consistent: as dataset complexity and network capacity grow, the coherent stable regions of the hyperparameter map shrink, and training becomes increasingly sensitive to the precise values of the Adam moment coefficients. In other words, the harder the task, the narrower the range of settings under which training behaves predictably.

Perhaps the most intriguing quantitative result concerns the geometry of the boundaries between regimes. Using a box-counting analysis, a standard technique for estimating the fractal dimension of a set by covering it with boxes of decreasing size, the authors measured the complexity of the regime boundaries in the (β₁, β₂) plane. For several of the configurations they investigated, the estimated dimensions approach D ≈ 1.9, a value close to 2, the dimension of the plane itself. In practical terms, this means the boundaries are highly irregular and nearly space-filling at the available numerical resolution: they wind through the hyperparameter space so intricately that almost any region of the map is close to a transition.

The authors are careful to frame this result correctly. They interpret the near-2 dimensions as empirical measures of boundary complexity rather than as evidence of exact mathematical fractality. The measurements are made at a finite numerical resolution, and infinitely fine structure cannot be confirmed. Even so, the finding suggests that the transitions between training regimes are not smooth, well-behaved curves but ragged, convoluted interfaces, echoing the kind of complexity seen in phase-boundary phenomena in other iterative dynamical systems. For practitioners, the message is sobering: the boundary between a stable and an oscillatory training regime may be dense and tangled, so a hyperparameter point that appears safe at coarse resolution may sit close to a transition.

A natural objection to any such mapping exercise is that the observed structures might be artifacts of the observation window. If the regime maps were computed from only a short burst of training, the classification of a given (β₁, β₂) point might reflect transient behavior rather than the genuine long-term character of the optimization process. The authors addressed this concern directly with robustness experiments repeated over 300, 500 and 1,000 optimizer steps. The results show that the large-scale organization of all seven observable fields remains largely preserved across these windows, with the principal changes concentrated near the transition boundaries. This persistence indicates that the detected regime structures are reproducible and are not merely artifacts of short optimization histories.

The methodology itself is notable for what it avoids. The framework is explicitly empirical and observable-based: it does not attempt to derive regime structure from theoretical assumptions about the loss landscape, nor does it depend on specialized instrumentation of the optimizer. Instead, it treats the training process as a dynamical system and reads its state through a small set of measurements, in the way a physicist might characterize an unknown system through its observable responses. This makes the approach portable: the same seven descriptors can be computed for any network architecture, dataset and training setup, allowing different optimization regimes to be visualized and compared on a common footing.

For the deep-learning community, the study offers both a diagnostic tool and a conceptual reframing. On the tool side, the regime maps make it possible to identify comparatively stable regions of the hyperparameter space before committing to a full training run, which could be valuable in settings where training is expensive or where reproducibility is critical. On the conceptual side, the work underscores that the influence of optimizer hyperparameters is not a smooth, monotonic effect but an organization of qualitatively distinct regimes, whose boundaries can be so complex that they dominate the practical experience of tuning.

The work also fills a genuine gap in the literature. Although adaptive optimization algorithms are fundamental to modern deep learning, the global organization of training regimes induced by optimizer hyperparameters has remained insufficiently understood, and the influence of the Adam moment coefficients on the stability and qualitative behavior of learning has not previously been systematically investigated through parameter-space regime mapping. By treating the (β₁, β₂) plane as a dynamical phase diagram, the Lviv group has turned a neglected corner of the hyperparameter space into a map with structure, boundaries and measurable geometry.

The paper, published open access on 31 August 2026 after being received on 25 April and accepted on 15 August, carries the title “Dynamic Regime Maps of Neural Network Training Under the Adam Optimizer: An Observable-Based Empirical Analysis.” The research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors. The authors suggest that the framework provides a complementary, observable-based perspective on the complex behavior of adaptive neural-network optimization, one that may guide both future theoretical work on why these regime boundaries are so intricate and practical work on choosing hyperparameters that sit safely inside the stable regions of the map.