The research team, led by Mohammad Akram of the Islamic University of Madinah together with Umar Ishtiaq, Khaleel Ahmad, and Ioan-Lucian Popa, proves that convergence survives under a far more permissive condition called enriched nonexpansivity. A map in this class is allowed to be expansive, to stretch distances, as long as it becomes nonexpansive once it is averaged with the identity operator. Formally, an operator T is b-enriched when a specific inequality holds that bounds how much T can expand points, with the amount of permissible expansion absorbed by averaging. The enriched class strictly contains the nonexpansive one, so every previously certifiable denoiser still qualifies, but an entire new territory of expressive networks opens up alongside it. Crucially, the enrichment constant b is not merely an abstraction; it becomes a computable number that a program can extract from a trained network.

The heart of the paper is a reduction, and it has a satisfying geometric flavor. The authors show that when the composite Plug-and-Play operator is enriched, an inertial Mann or Ishikawa iteration, classical fixed-point schemes that move only part of the way toward the operator’s output and add a momentum-like correction, becomes term by term identical to an ordinary iteration on a hidden nonexpansive operator called the generator. The inertial correction turns out to be exactly the quantity that cancels the overshoot introduced by the expansion. In other words, the rigidity is not imposed on the network at all; it is moved off the network and onto the iteration, where averaging repairs the loss of structure automatically. The fixed points of the expansive composite and its tame generator are the same set, so the convergent iteration lands on a genuine Plug-and-Play solution.

Turning this into usable software required a certificate, a way of checking for any given trained denoiser whether the enrichment hypothesis actually holds. The team’s main characterization theorem states that a smooth operator is b-enriched precisely when its displacement, the difference between the identity and the operator, is infinitesimally cocoercive with a modulus tied to b. The smallest admissible enrichment constant is one less than the supremum of a Rayleigh-type quotient of that displacement, a ratio that measures how far the displacement moves a direction relative to how much of that motion is aligned with the direction itself. This quotient is exact rather than merely sufficient, it is attained on non-scalar operators, it remains bounded near the identity where coarser bounds degenerate, and it can be evaluated through Jacobian-vector products that automatic differentiation supplies at the cost of roughly one network evaluation each.

The numbers that emerge are striking. On a linear surrogate problem in 120 dimensions, constructed so that every constant is available in closed form, the exact certificate widened the admissible step-size window by about 31 percent compared with the standard field-of-values bound, and the computed values matched the ground truth. The surrogate itself made the point vividly: an expansive operator with a spectral norm above one, admitted by no nonexpansive condition, was certified with a finite enrichment constant, and averaging it yielded a generator whose norm measured just below one. On real image restoration tasks with trained denoisers, the team measured the certified constant for each composite operator individually, using Lanczos iterations and bisection built on the same differentiable primitives.

The headline experimental result concerns Gaussian deblurring. A denoiser trained with a relaxed Lipschitz target produced a composite operator with a Lipschitz constant of 2.08, a value that no nonexpansive, firmly nonexpansive, or contractive-residual condition in the existing literature would accept. Yet its symmetric-part maximum stayed below one, and the exact characterization certified it with an enrichment constant of just 0.60, more than fifteen times tighter than the field-of-values relaxation for the same operator. The gap is a direct measurement of non-normality: the deep network expands strongly in directions poorly aligned with the residual, something the pair of conventional spectral quantities cannot see. Most remarkably, this expansive operator restored images within four hundredths of a decibel of a certified nonexpansive composite of identical architecture, meaning the larger, more expressive class was reached at no measured cost in quality.

Certification alone would be a curiosity without control over the iteration itself, so the authors built a safeguarded step-size controller on top of the theory. Any adaptive policy that stays inside the certified window preserves the convergence guarantee, whatever the policy has learned, so speed can be trained while safety is decided by the certificate. Within that architecture they derived a closed-form greedy relaxation, a Barzilai-Borwein-type step for the fixed-point residual, which roughly halved the iteration count of the constant-step scheme in both the surrogate and the image experiments. An amortized policy trained by unrolled differentiation through the solver matched the greedy rule without ever violating its guarantee, and the team verified empirically that stepping beyond the certified window by a factor of two caused divergence, confirming that the certificates mark a real stability boundary rather than an overly cautious one.

The framework also proved robust in ways that matter for practice. Convergence rates degrade gracefully, with the enrichment constant entering the residual bound as a simple known factor while the order of convergence is untouched. The theory tolerates denoiser evaluation errors of summable norm, covering finite-precision arithmetic and early-stopped inner solvers, which is exactly the situation encountered in real pipelines. A splitting-agnostic extension carries the entire construction beyond forward-backward splitting to Plug-and-Play ADMM and other schemes, since all that is ever required is a finite enrichment constant for the fixed-point operator of the scheme, checkable through the same eigenvalue computations.

The authors are careful about scope. The certificate is first order and measured on a finite sample, so the reported guarantees hold on the region actually certified rather than the whole image domain without further work. On the linear surrogate, an unrelaxed Picard iteration that carries no guarantee happened to be the fastest method, a reminder that acceleration claims here are comparisons among safeguarded schemes. The restoration study used small residual networks on modest image crops, so the reconstruction numbers are not a bid for state-of-the-art performance. What the results establish is arguably more consequential: the enlarged operator class is genuinely occupied by networks the existing guarantees reject, and using it costs nothing measurable in quality.

The deeper significance of the work may lie beyond imaging. Wherever a learned operator is iterated to a fixed point, in deep equilibrium models, learned optimizers, or unrolled reconstruction networks, the same tension between expressivity and guaranteed convergence reappears. The transfer-return mechanism offers a uniform recipe: certify the composite as enriched, transfer to the nonexpansive generator, iterate with the inertial correction, and transfer back. Whether that recipe extends to those settings remains an open conjecture, but the authors have made it checkable, reducing the entire question to eigenvalue computations that automatic differentiation already supports. In a field where convergence proofs have traditionally been bought by constraining networks into submission, the idea that the mathematics can instead bend around the network is a quiet revolution with loud implications.

Subject of Research: Convergence theory and certification for enriched-nonexpansive Plug-and-Play image restoration with learned denoisers

Article Title: Enriched-nonexpansive Plug-and-Play image restoration

Article References: Akram, M., Ishtiaq, U., Ahmad, K., & Popa, I.-L. (2026). Enriched-nonexpansive Plug-and-Play image restoration. Frontiers in Applied Mathematics and Statistics, 12, Article 1945678. https://doi.org/10.3389/fams.2026.1945678

Image Credits: AI Generated

DOI: 10.3389/fams.2026.1945678

Keywords: Plug-and-Play restoration, enriched nonexpansive mappings, fixed-point iteration, inertial Mann iteration, denoisers, image deblurring, super-resolution, convergence guarantees, cocoercivity, automatic differentiation, spectral certification, ADMM

The core idea of zeroth-order optimization is old and intuitively appealing. Rather than computing a gradient analytically, one estimates it by perturbing the parameters slightly in random directions and observing how the loss changes. If nudging a weight upward makes the model worse and nudging it downward makes it better, the estimator knows, roughly, which way to push. Two function evaluations per step, in the so-called two-point estimator popularized in modern language model work such as the MeZO algorithm, are enough to produce a usable descent direction. The catch is statistical. A random direction in a space of millions or billions of dimensions is almost certainly nearly orthogonal to the true gradient, so the difference in loss between two nearby points along that direction is dominated by noise. The variance of the resulting gradient estimate scales with the dimension of the perturbation space, and in parameter-efficient fine-tuning, where the trainable set may still contain millions of parameters, that variance can be severe enough to make training crawl or stall.

Parameter-efficient fine-tuning itself was supposed to solve the scale problem. Techniques such as adapters, prefix-tuning, prompt tuning, BitFit and the now-ubiquitous low-rank adaptation known as LoRA freeze the bulk of a pretrained model and train only small inserted modules or subsets of weights. This slashes memory for optimizer states and makes adaptation feasible on modest hardware when gradients are available. But as Zhang and Yuan point out, even a reduced trainable set frequently reaches into the millions of parameters, and a zeroth-order estimator perturbing all of them simultaneously inherits variance proportional to that count. The reduction that makes PEFT attractive under backpropagation does relatively little for the statistical burden of gradient-free methods. Previous lines of work have attacked the problem from several directions, including sparse perturbations, transferable static sparsity, low-rank structures, curvature-aware estimates, preconditioned and accelerated variants, and random subspace methods that restrict perturbations to a randomly chosen subset of coordinates. The new paper builds on this momentum but changes where the compression happens.

The proposed framework, called Projected Latent Subspace Zeroth-Order Optimization, or PLS-ZO, restricts perturbations not to a subset of coordinates but to a deliberately constructed low-dimensional latent subspace, and it does so at the level of individual weight matrices rather than the flattened parameter vector. The authors construct hierarchical low-rank random projections that map each trainable matrix into a controllable latent space of much lower dimension. Random projection is a classical tool from linear algebra and randomized numerical computation: a high-dimensional vector multiplied by a random matrix retains, with high probability, the geometric information that matters, compressed into far fewer coordinates. By applying this idea hierarchically across the matrices of the model, PLS-ZO creates a latent coordinate system in which the optimization actually takes place. Perturbations are drawn in this latent space, and the difference of loss values between perturbed and unperturbed forward passes yields an estimated gradient with variance governed by the latent dimension rather than by the original parameter count.

A crucial and somewhat distinctive design choice follows from this. Many subspace methods treat the projection as a way to generate cheaper perturbations but still run the optimizer in the original high-dimensional coordinates. PLS-ZO instead performs the main optimizer operations, including momentum accumulation, preconditioning and the bookkeeping of optimizer states, directly in the low-dimensional latent coordinates. Momentum in particular benefits: an exponential moving average of past gradient estimates is a powerful variance reducer, but maintaining and applying it over millions of coordinates is exactly the kind of overhead zeroth-order methods are supposed to avoid. Working in latent space keeps those state tensors small and makes preconditioning, which rescales update directions to account for uneven curvature, computationally practical. The result is an optimizer whose memory footprint and per-step arithmetic scale with the chosen latent dimension, a hyperparameter the practitioner controls, rather than with the full size of the trainable parameter set.

One subtlety of any random-subspace approach is that the subspace must eventually change. A single fixed random projection risks missing important directions of variation in the loss landscape for the entire run, so PLS-ZO periodically refreshes its projections, drawing new random bases in which optimization continues. Naively, each refresh would force the optimizer to discard its accumulated momentum and preconditioner statistics, throwing away the very variance reduction that makes the method work, or would require expensive re-projection of large state tensors. The authors introduce what they call a lazy subspace refresh strategy, which defers and amortizes the cost of regenerating projections, together with a principled state migration rule that projects the latent optimizer states from the old basis into the new one. In effect, the optimizer’s memory of its trajectory survives the change of coordinates, so long-horizon training remains stable instead of repeatedly restarting from a cold state.

Beyond the algorithmic engineering, the paper contributes theory. The authors prove that the bias and variance of the two-point zeroth-order estimator under their scheme are governed by the latent subspace dimension rather than the original parameter dimension, formalizing the intuition that compressing the perturbation space compresses the noise. They further establish a nonconvex convergence guarantee for the setting in which the random subspaces are periodically refreshed, showing that the algorithm converges to a stationary point at a rate consistent with classical zeroth-order analysis despite the moving coordinate system. Nonconvex guarantees of this kind matter because the loss surfaces of fine-tuned neural networks are emphatically nonconvex, and a method whose theoretical behavior degrades under basis changes would be difficult to trust in practice. The migration rule and refresh schedule are precisely the components that make the analysis go through.

On the empirical side, the authors report extensive experiments demonstrating that PLS-ZO consistently achieves highly effective fine-tuning performance under the parameter-efficient setting. The evaluation draws on widely used natural language understanding benchmarks of the kind standard in this literature, including datasets such as SST-2, BoolQ, WiC, WinoGrande and MultiRC that have anchored comparisons among gradient-free fine-tuning methods, applied to robustly pretrained encoder models in the RoBERTa family. The paper’s data availability statement notes that no new datasets were generated or analyzed, meaning the contribution lies in optimization methodology rather than data collection. The experiments position PLS-ZO against the backdrop of a rapidly crowding field: recent work has introduced low-rank zeroth-order fine-tuning, temporally low-rank perturbations, gradient-aligned projections, minimum-variance two-point estimators, Hessian-informed optimizers, quantized zeroth-order training, and learned optimizers, all racing to make forward-pass-only adaptation competitive with full backpropagation.

The significance of this line of research extends well beyond academic benchmarks. As large models are deployed at the edge, embedded in privacy-sensitive pipelines, or offered as closed services, the assumption that a practitioner can differentiate through the model is increasingly fragile. Zeroth-order fine-tuning requires only the ability to run the model forward and read off a loss, which makes it compatible with API-based adaptation, on-device training under tight memory budgets, and models whose weights are frozen by policy as much as by architecture. Methods like PLS-ZO attack the central statistical obstacle, dimension-induced variance, with a control knob, the latent dimension, that lets practitioners trade a small amount of expressive freedom for large gains in estimate quality and compute. If the theoretical promise holds up under wider independent testing, the practical upshot is that adapting a frontier-scale model could become feasible for organizations with neither the hardware nor the internal access that today’s fine-tuning workflows assume, a shift that could meaningfully democratize who gets to specialize the most powerful models in existence.

Subject of Research: Zeroth-order optimization for parameter-efficient fine-tuning of large language models

Article Title: PLS-ZO: projected latent subspace zeroth-order optimization for parameter-efficient fine-tuning

Article References: Zhang, X., & Yuan, S. (2026). PLS-ZO: projected latent subspace zeroth-order optimization for parameter-efficient fine-tuning. International Journal of Machine Learning and Cybernetics, 17(10), Article 488. https://doi.org/10.1007/s13042-026-03317-9

Image Credits: AI Generated

DOI: 10.1007/s13042-026-03317-9

Keywords: zeroth-order optimization, parameter-efficient fine-tuning, large language models, random subspace, low-rank projection, black-box optimization, gradient-free methods, LoRA, latent subspace, convergence guarantees, machine learning, optimization