AI That Imagines Rewards: Diffusion Models Rewrite Traffic Signal Control

City traffic grids are among the most stubborn optimization problems in modern engineering. Every intersection acts as its own decision-maker, yet the performance that matters — average queue length, travel time, congestion across a whole district — emerges from the interaction of dozens of signals acting simultaneously. A new framework called BootDiffLights, described in Applied Intelligence by researchers at Chengdu University of Information Technology and Chengdu Jiaotou Information, proposes a strikingly different way to coordinate these lights: instead of painstakingly assigning credit for a region’s traffic outcomes to each individual signal, it lets a generative diffusion model imagine what good reward signals should look like, and then optimizes against those imagined rewards.

The work targets a long-standing pain point in multi-agent reinforcement learning (MARL), the dominant modern approach to decentralized traffic signal control. In a MARL setup, each intersection hosts an agent that learns its own policy — when to switch phases, how long to hold green, when to yield to cross traffic. The trouble is that the global reward, such as a network-wide reduction in travel time, is shared among all agents. Determining which intersection’s decisions actually produced the improvement is the credit assignment problem, and it is notoriously difficult when the effects of one signal ripple through neighboring blocks.

Conventional solutions tackle credit assignment with critic networks that attempt to decompose the global reward into per-agent contributions. These approaches carry two fundamental limitations, the authors argue. First, they are sample-hungry: agents must engage in extensive trial-and-error interactions with the traffic environment before the critics learn to allocate credit meaningfully, and every trial in a live or simulated network is expensive. Second, the critics must possess strong extrapolation and generalization capabilities — they must interpret policies they were never trained on and still distribute credit sensibly. When traffic patterns shift, as they constantly do, critic networks can produce misleading credit signals that push agents toward worse policies.

BootDiffLights sidesteps this entire machinery. The key insight is a reframing: rather than asking how to divide a global reward among agents, the framework adopts a semi-offline multi-agent reinforcement learning paradigm that transforms the problem from credit assignment into black-box optimization. In practical terms, the system no longer focuses on exploring optimal policies directly. Instead, it concentrates on generating optimal sample data — high-quality trajectories of traffic states and actions — and optimizes in that data space. The generative engine doing this work is a conditional diffusion model, the same class of model that has powered recent breakthroughs in image synthesis and, increasingly, decision-making.

Diffusion models work by learning to reverse a gradual noising process. Trained on data, they learn to transform pure noise into samples that resemble the training distribution, and conditioning signals can steer what they generate. In BootDiffLights, the conditional diffusion model serves as an implicit credit-assignment mechanism: it learns the relationship between traffic conditions, agent behaviors, and resulting rewards, and can then synthesize reward signals consistent with desired regional outcomes. Because the model captures the joint structure of the multi-intersection system, it implicitly encodes how each signal’s actions contribute to network-level performance — without an explicit critic network attempting the decomposition.

The semi-offline aspect matters as much as the diffusion engine. Fully offline reinforcement learning restricts agents to a fixed dataset, which is safe but often suboptimal; fully online learning requires costly and sometimes dangerous exploration. A semi-offline approach lets the system bootstrap from logged traffic data while selectively gathering new interaction data where it matters most. This is what enables BootDiffLights to improve sampling efficiency: the diffusion model can propose promising regions of the behavior space, and the learning system refines its data generation around those regions rather than wandering blindly through trial and error. The framework essentially shifts computational effort from interacting with the environment to generating better synthetic experience.

The authors evaluated the framework on both synthetic and real-world traffic datasets, running all experiments with five random seeds and reporting results as mean plus or minus standard deviation — a rigorous protocol that guards against lucky draws inflating performance claims. Across these benchmarks, BootDiffLights outperformed state-of-the-art baselines on regional traffic metrics, including average queue length and travel time. The comparison field includes well-established MARL techniques built on value decomposition and multi-agent actor-critic architectures, meaning the generative approach did not merely match conventional methods but surpassed them on the metrics that commuters actually feel.

The significance of this work extends beyond one benchmark suite. Traffic signal control has been dominated for decades by fixed-time plans and adaptive schemes such as SCOOT and SCATS, with reinforcement learning offering a data-driven alternative since the late 2010s. The field’s persistent obstacle has been scale: methods that work beautifully on a handful of intersections degrade as networks grow, because credit assignment becomes combinatorially harder. By replacing explicit credit decomposition with generative reward synthesis, BootDiffLights offers a path that may scale more gracefully. The diffusion model’s capacity to model complex joint distributions could prove decisive in dense urban cores where dozens of signals interact within a few blocks.

The approach also connects to a broader trend in machine learning: the migration of generative modeling into sequential decision-making. Diffusion-based policies and planners have recently shown impressive results in robotics and offline control, and BootDiffLights brings that momentum to a civic infrastructure problem with enormous practical stakes. Congestion costs cities billions annually in lost time and fuel, and emissions from idling vehicles degrade urban air quality. A control framework that learns efficiently from existing data, requires no explicit reward decomposition, and improves region-wide metrics could translate into real deployments — though the authors note that data availability constraints limit what can be shared publicly for now.

Challenges remain before such systems direct real intersections. Diffusion models carry their own computational burden at inference time, and traffic agencies will demand robustness guarantees that simulation studies alone cannot fully provide. Yet the conceptual shift is compelling: where previous frameworks asked a critic to guess who deserved credit, BootDiffLights asks a generative model to imagine the rewards worth pursuing — and then optimizes toward that imagination. For a problem as tangled as city-wide traffic coordination, that creative reframing may be exactly the kind of fresh thinking the field has been waiting for.

Subject of Research: Diffusion-based reward generation for decentralized multi-intersection traffic signal control in semi-offline multi-agent reinforcement learning

Article Title: BootDiffLights: Decentralized traffic signal control via diffusion-based reward generation

Article References: Hao, B., He, J., Zheng, J., Dong, W., Yang, Q., & Ma, X. (2026). BootDiffLights: Decentralized traffic signal control via diffusion-based reward generation. Applied Intelligence, 56(15), Article 437. https://doi.org/10.1007/s10489-026-07487-5

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07487-5

Keywords: diffusion model, traffic signal control, multi-agent reinforcement learning, credit assignment, offline reinforcement learning, semi-offline learning, black-box optimization, reward generation, queue length, travel time, smart infrastructure, Applied Intelligence