In a study published in the journal Quantum Machine Intelligence, Oliver Sefrin of the German Aerospace Center (DLR) in Ulm, together with Manuel Radons and Lars Simon of Bundesdruckerei GmbH in Berlin and Sabine Wölk of DLR and the Center for Integrated Quantum Science and Technology at Ulm University, report the first systematic investigation of a hybrid quantum-classical reinforcement learning agent in non-stationary environments. The work, published as an open-access research article in Volume 8 of the journal, marks a departure from earlier studies of the so-called hybrid agent for quantum-accessible reinforcement learning, which had until now been tested only in stationary settings where the rules of the game never change.

Reinforcement learning, or RL, is the branch of machine learning in which an agent learns by trial and error, interacting with an environment that is formally described as a Markov decision process. The agent observes states, takes actions, collects rewards, and gradually adjusts its policy — the probability with which it chooses each action in each state — to maximize cumulative reward. From mastering Atari games and Go to fine-tuning today’s large language models through reinforcement learning from human feedback, the paradigm has become one of the most consequential tools in artificial intelligence. But in realistic scenarios, from traffic management to industrial control, the environment itself moves: lanes close, demand fluctuates, targets relocate. A policy that was optimal yesterday may be worthless today, and well-trained agents often fare worse after a sudden change than untrained ones, because what they have learned actively works against them.

The hybrid learning agent at the center of the new study, introduced in earlier work by Hamann and Wölk, offers a provable speed advantage in a special class of learning problems. It combines a classical RL algorithm with Grover-type amplitude amplification, a quantum technique that boosts the probability of sampling rewarded action sequences quadratically faster than random classical sampling. In the quantum part of the algorithm, actions and percepts are encoded in orthonormal quantum states, and the environment’s response is modeled as a unitary that produces a phase-kickback oracle marking rewarded action sequences with a sign flip. Applying Grover iterations to a superposition of action sequences weighted by the agent’s current policy raises the chance of measuring a rewarded sequence according to the well-known sine-squared amplification law, and the resulting candidate is then verified in a classical episode of play. The payoff is a quadratic reduction in the number of episodes needed to find rewarded behavior — but only for deterministic, strictly episodic tasks, such as the classic Gridworld maze, where the episode length is fixed and transitions are deterministic.

The catch, the researchers recognized, is that speed can become a liability. “In such dynamic learning scenarios, a well-trained agent often performs worse after a sudden change of the environment compared to an untrained or a barely trained agent,” the authors note in the paper. A fast-learning quantum agent that has converged hard onto one reward structure can find itself trapped, its confident policy pointing away from the new source of reward. To be competitive outside the laboratory, the hybrid agent needed a way to forget.

The team’s solution is elegantly simple: they paired the hybrid agent with Projective Simulation, a tabular RL algorithm developed by Hans Briegel and Gorka De las Cuevas, which stores experience in a network of clips connected by weighted edges. Projective Simulation includes a natural dissipation mechanism — a forgetting parameter that continuously pulls the edge weights back toward their initial values, driving the policy gradually toward uniform exploration. Two modifications were required to make the hybrid version work in a changing world. First, because executing k rounds of amplitude amplification consumes 2k+1 interaction episodes, the quantum agent updates its policy less frequently than a classical one; the researchers derived an equivalent update rule that scales the dissipation strength so that the hybrid agent forgets at the same rate as its classical counterpart, and they proved the equivalence formally by induction. Second, the agent’s internal lower-bound estimate of its success probability — needed to calibrate the number of Grover iterations — had to be maintained adaptively, purging previously rewarded action sequences from memory whenever they were discovered to be no longer rewarded after an environment change.

The testbed was a modified Gridworld with a moving target. In the first scenario, the reward moves along a fixed path within each episode, synchronously with the agent’s own steps, adding time dependence within episodes while keeping the overall reward structure stable. In the second, more demanding scenario, the reward path itself is swapped for a new, fully disjoint path partway through training — a hidden-mode Markov decision process in which the agent knows a switch may occur but is never told when. Crucially, the researchers designed the second layout so that the two reward paths share no rewarded action sequences at all, making it the harshest possible test: excellence on the first path maximally damages performance on the second.

The results, averaged over hundreds of simulated runs, are striking. In the fixed-path scenario, the hybrid agent found its first reward in an average of 17.9 episodes, compared with 58.4 episodes for classical Projective Simulation — better even than the theoretical quadratic-speedup upper bound of 34.4 episodes — and after just 100 training episodes it had already reached an average success probability of 99.3 percent, while the classical agent lagged at 69.3 percent. Classical Q-Learning, with its one-step back-propagation of reward information, needed far longer for its values to percolate back to the starting state.

The changing-path scenario delivered the study’s most interesting twist. Immediately after the reward path switched, the hybrid agent’s success probability collapsed by roughly an order of magnitude below that of the classical agent — exactly the vulnerability the researchers had anticipated. But the forgetting mechanism did its work. The hybrid agent’s curve then climbed steeply, converging to nearly 100 percent success, while classical Projective Simulation rose more gradually to 98.4 percent and Q-Learning, hampered by its fixed exploration rate, plateaued between 60 and 80 percent. With a dissipation value of 0.05, the hybrid agent achieved the shortest total learning time in the benchmark — about 113 episodes against 149 for Q-Learning and 187 for the best classical Projective Simulation configuration. The researchers also showed that their adaptive estimate of the success probability tracks the true value closely, overshooting only marginally in the first few episodes after a switch thanks to the purging mechanism.

“Given that the problem scenario in this article is chosen such that the hybrid agent’s initial good performance puts it at a maximal disadvantage after a change in the reward function, we expect the results of this toy problem to generalize well to a wider range of problem cases,” the authors write. They go further, conjecturing that the quadratic speedup in learning may itself be what enables faster adaptation — potentially opening the door to learning in fast-changing environments where a classical agent would be too slow to learn at all.

The work sits within the broader and still-uncertain landscape of quantum reinforcement learning. Much current research in the noisy intermediate-scale quantum era relies on variational quantum circuits as function approximators, but whether those approaches can deliver genuine quantum advantage remains an open question. The hybrid agent, by contrast, carries a proven speedup and has already been demonstrated in a proof-of-principle experiment on a nanophotonic processor, suggesting it may be a serious contender for the post-NISQ era. The authors caution that their dissipation mechanism is deliberately passive — it does not detect changes but absorbs them — and that combining the hybrid agent with continual-learning methods that prevent catastrophic forgetting, or extending the study to time-dependent transition dynamics, are natural next steps. They also call for standardized continual-RL benchmarks, perhaps as extensions of the popular Gymnasium package, to put such comparisons on firmer footing.

For a field often accused of chasing speedups on toy problems that never change, the message of this study is pointed: quantum-accelerated learners can not only learn faster, but also unlearn faster. In a world where the only constant is change, that may be the advantage that matters.

At the heart of this new approach lies a modular, multi-layered structure that decomposes complex urban infrastructure systems into nested blocks. Each block represents a manageable unit—such as a neighborhood or a utility network segment—that can be analyzed and optimized independently while maintaining seamless integration with adjacent units. This paradigm allows for fine-grained control over infrastructure assets, promoting localized solutions without losing sight of the citywide system dynamics.

Traditional urban infrastructure management methods often struggle to balance efficiency with equitable service distribution across diverse communities. The nested-block framework addresses this by employing scalable algorithms that allocate resources dynamically based on real-time data streams and socio-economic indicators. This enables decision-makers to prioritize upgrades and maintenance in underserved areas, thereby reducing systemic inequalities commonly found in aging urban grids.

A notable technical innovation in this framework is its incorporation of adaptive feedback loops powered by machine learning. These loops continuously monitor infrastructure performance metrics—ranging from energy consumption patterns to water distribution anomalies—and self-tune resource allocation strategies accordingly. This adaptability ensures resilience against both predictable stresses like daily demand fluctuations and unpredictable shocks such as extreme weather events.

Furthermore, the research team emphasizes the framework’s interoperability with existing urban data platforms and governance structures. By leveraging open standards and modular design principles, cities can implement the nested-block system incrementally without disruptive overhauls. This pragmatic pathway is critical for real-world adoption, considering the high costs and complexities typically associated with infrastructure modernization.

The scalable nature of the framework also supports multi-sector coordination, facilitating integrated management across transportation, utilities, and public services. Such cross-domain synergy is essential to optimize cascading effects—for example, rerouting traffic in real time to prevent gridlock during power outages or optimizing water supply during heatwaves.

Experts believe this nested-block methodology could set a new global benchmark for sustainable urban infrastructure. As cities continue to expand and face pressures from climate change, novel solutions that combine technical sophistication with social fairness will certainly be pivotal. Lin and colleagues’ contribution represents a significant step towards smarter, more inclusive urban environments where technology empowers rather than alienates local communities.

While implementation challenges remain—especially with regard to data privacy and equitable governance—the framework provides a robust foundation for next-generation infrastructure systems. It invites urban planners, engineers, and policymakers alike to rethink how smart technologies can be harnessed for not just efficiency but also justice in urban living.

Subject of Research: Urban infrastructure management and optimization.

Article Title: A nested-block scalable framework for efficient and equitable urban infrastructure management.

Article References: Lin, P., Yu, X., Liu, T. et al. A nested-block scalable framework for efficient and equitable urban infrastructure management. Commun Eng (2026). https://doi.org/10.1038/s44172-026-00729-x

Image Credits: AI Generated