The study, authored by Li He of Ningbo University of Finance and Economics and published on 15 October 2026, addresses a well-known weakness in existing approaches. Rule-based driving models, such as the widely used Intelligent Driver Model, are transparent and easy to implement, but they tend to react late and leave thin safety margins when traffic becomes dense and nonlinear. Deep reinforcement learning systems adapt better, yet most of them optimize either discrete choices like whether to change lanes or continuous quantities like acceleration, rarely both at once. Optimizing the two levels independently can produce conflicting objectives, delayed responses, and unstable vehicle control, because a smooth acceleration policy is useless if it is executing a bad strategic decision, and a clever lane-change plan is dangerous if the steering and braking that follow are jerky or mistimed.

The proposed solution is a hierarchical reinforcement learning architecture that assigns each modality to the algorithm best suited for it. At the top of the hierarchy sits a Double Deep Q Network, or DDQN, which handles discrete behavioral intentions: following, lane changing, and avoidance. DDQN was chosen over the standard Deep Q Network because it separates action selection from value evaluation, correcting the systematic overestimation of Q-values that can push an agent toward overconfident, unsafe maneuvers. At the bottom of the hierarchy, a Deep Deterministic Policy Gradient, or DDPG, controller translates those high-level intentions into continuous acceleration commands. Because longitudinal control is naturally a continuous optimization problem, DDPG’s actor-critic structure is a natural fit: the actor network maps the current state directly to an acceleration, while the critic network evaluates state-action pairs and guides policy improvement without ever needing to discretize the action space.

The two layers communicate through an explicit constraint mechanism, formalized in the study as a policy function in which the continuous action at each time step is conditioned on the discrete decision handed down from above. In other words, the DDPG controller does not simply imitate the DDQN layer; it executes that layer’s intentions under the physical and safety constraints of the moment. A long short-term memory network encodes a window of five historical time steps, each containing the ego vehicle’s velocity and acceleration, relative distances, and surrounding vehicle information, giving both layers a short-term memory of traffic dynamics rather than forcing them to act on instantaneous snapshots. This temporal context is what allows the agent to recognize patterns such as a neighbor vehicle drifting toward a cut-in before the cut-in happens.

One of the more technically interesting contributions involves how the DDPG controller itself was tuned. Deep reinforcement learning is notorious for training instability, where results swing wildly depending on hyperparameters like the learning rate, discount factor, and exploration noise. Rather than hand-tuning these values, the study employs a Multi-Strategy Particle Swarm Optimization, or MSPSO, procedure in which candidate hyperparameter configurations are treated as particles moving through a search space. Each particle’s fitness is scored by the cumulative reward and control stability achieved in preliminary DDPG training runs. The swarm blends global and local search strategies, applies inertia weights and learning factors to shift from broad exploration early on to fine convergence later, and injects random perturbations when population diversity collapses, guarding against premature convergence into local optima. Crucially, the optimization runs offline: the winning hyperparameters are then fixed and handed to DDPG for its actual training, avoiding an unstable co-optimization loop between the two algorithms.

The reward design reflects a philosophy familiar to anyone who has ridden in an autonomous vehicle prototype: safety trumps everything. The reward function weights three objectives at 0.5 for safety, 0.3 for comfort, and 0.2 for efficiency. Safety rewards maintain a following distance above a dynamic threshold computed from a minimum standstill gap plus the current speed multiplied by an expected safe headway, and it imposes steep penalties as collision risk rises. Comfort rewards penalize excessive acceleration changes and control jitter, while efficiency rewards keeping pace with the desired speed so the agent does not become timid. This prioritized weighting prevents any single objective from dominating the learning process and ensures the vehicle never trades away collision avoidance for a smoother or faster ride.

Training and evaluation relied on the nuScenes public dataset, roughly 1,000 real driving scenarios of about 20 seconds each, collected by a full sensor suite of six surround-view cameras, one lidar, and five millimeter-wave radars across conditions including night, rain, congestion, and dense multi-vehicle interaction. Because nuScenes is an open-loop dataset, the author could not simply drop a learning agent into it. Instead, the recorded trajectories, relative distances, velocities, and interaction patterns were used to reconstruct realistic traffic states, and a closed-loop evaluation environment was built around them by combining vehicle motion constraints and traffic interaction rules. The agent perceives a state, issues a decision or acceleration command, the environment updates the vehicle state through a transition model, and the loop continues. All experiments were repeated five times with different random seeds, and reported results are means with standard deviations.

The car-following results are striking. The proposed model held an average inter-vehicle distance of 20.1 meters, about 8.1 percent greater than the comparison models, with spacing fluctuations contained within plus or minus 1.2 meters, compared with fluctuations of roughly 2.0 meters for a TD3 benchmark and even wider swings for the rule-based IDM, which at times let the gap fall to around 14.2 meters. The collision rate dropped to 1.3 percent, more than 55 percent lower than the TD3 benchmark, and the average headway settled at a stable 1.95 seconds, almost entirely above the 1.5-second safety threshold. Comfort improved in parallel: average jerk, the rate of change of acceleration that passengers feel as lurching, fell to 0.72 meters per second cubed, roughly 23 percent below TD3, and the comfort satisfaction rate reached 93.8 percent, with a safety constraint satisfaction rate of 96.5 percent. The model also converged in about 150 training rounds, roughly 28.6 percent fewer than TD3, achieving the highest average reward of 96.8.

The hierarchical decision layer showed comparable gains. Tested against standard DQN and Proximal Policy Optimization baselines, the DDQN decision module sustained accuracy between 0.88 and 0.96, peaking near 0.96 in a mid-session window and staying above roughly 0.90 during a late-session period when both baselines degraded, an improvement of about 5 to 8 percent overall. Its conflict rate stayed between 0.05 and 0.10, and at one point measured about 0.08 where DQN reached 0.22, a reduction of more than 60 percent. The agent also changed lanes about 23.7 percent less often than DQN on average, with a redundant decision share of only 10.4 percent, and it responded faster: decision latency averaged around 140 milliseconds, more than 25 percent quicker than DQN, with the narrowest latency distribution across free-flow, dense, cut-in, and ramp-merge scenarios. An ablation study confirmed that neither module alone explains the gains: the full DDQN-DDPG framework lifted decision accuracy to 93.5 percent, cut the collision rate to 7.6 percent, and raised the average reward to 97.2, outperforming both single-module variants on every metric.

The study is candid about its limits. The dual-module hierarchy adds computational overhead that could strain resource-constrained onboard platforms, although the author notes the extra cost falls mainly on offline training while online decisions remain fast. The evaluation environment, however realistic its initialization from nuScenes, still substitutes a transition model for the raw unpredictability of the open road, and real-vehicle closed-loop validation remains future work. The framework also does not yet account for heterogeneous road users, uncertain road conditions, or driver-specific preferences. Even so, the central lesson travels well beyond one paper: when a driving task spans two fundamentally different kinds of choice, deciding what to do and deciding how precisely to do it, forcing a single network to learn both may be the bottleneck. Splitting the problem across cooperative specialists, each trained with the other’s constraints in view, may be how autonomous vehicles finally learn to drive not just safely, but smoothly enough that their passengers stop noticing the robot at the wheel.

Subject of Research: Hierarchical reinforcement learning integrating DDQN and DDPG for intelligent driving decision-making in complex traffic scenarios

Article Title: Multi-modal intelligent driving decision-making in complex traffic scenarios integrating DDQN and DDPG

Article References: He, L. (2026). Multi-modal intelligent driving decision-making in complex traffic scenarios integrating DDQN and DDPG. Discover Artificial Intelligence, 6(1), Article 1503. https://doi.org/10.1007/s44163-026-02454-y

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02454-y

Keywords: autonomous driving, hierarchical reinforcement learning, DDQN, DDPG, car following, decision-making, particle swarm optimization, nuScenes, traffic safety, lane changing, deep learning, intelligent vehicles