Curiosity and artificial potential fields drive TD3 navigation in dynamic environments

Curious by Design, Safe by Force: New Robot AI Learns to Slip Through a World in Motion

Robots are leaving the ordered predictability of factories and stepping into spaces where nothing holds still—hospital corridors crossed by gurneys, warehouse aisles shared with forklifts and people, sidewalks crowded with pedestrians, drones threading through gusting air. For machines, these moving worlds are the hardest test there is: a route that is clear one second can be blocked the next, and every decision must be made on the fly. A research team at Nanjing University of Science and Technology in China now reports a new way to give robots exactly that skill, in a study published on 29 August 2026 in the International Journal of Intelligent Robotics and Applications. Led by first author Rui Zhang and supervised by corresponding author Xiang Wu, the group fused two seemingly opposite ideas—an algorithmic form of curiosity borrowed from psychology and a physics-inspired invisible “force field” for safety—inside a deep reinforcement learning framework known as TD3. In both simulation and real-world trials, the hybrid approach outperformed standard methods in how reliably robots reached their goals and how quickly they learned to do so.

The fundamental difficulty is that dynamic environments break the assumptions behind most classical navigation machinery. Landmark methods—A* graph search, formalized by Peter Hart, Nils Nilsson and Bertram Raphael in 1968; rapidly exploring random trees introduced by Steven LaValle in 1998; bug-style algorithms that trace obstacle boundaries, pioneered by Vladimir Lumelsky and Alexander Stepanov—were conceived for worlds frozen in place, or depend on hand-designed rules that engineers must anticipate for every possible scenario. When obstacles move, these systems must replan constantly, and no rulebook can enumerate the combinatorial explosion of possible encounters between one robot and many moving agents. Even the artificial potential field method, introduced by Oussama Khatib in 1986, which paints invisible hills and valleys so that a robot simply rolls downhill toward its goal while being pushed away from obstacles, is elegant yet notoriously brittle: robots can become trapped in local minima where attraction and repulsion cancel, and retuning the fields for each new environment is a laborious art. The Nanjing team’s starting point was blunt: manually designed rules cannot readily adapt to environmental change, so the robot must learn to adapt instead of merely being programmed.

Reinforcement learning offers a radically different route. An RL agent tries actions, collects rewards and penalties, and gradually discovers the behaviors that maximize long-term payoff; deep reinforcement learning couples this trial-and-error process with neural networks large enough to map raw sensor readings directly to motor commands, an approach known as end-to-end navigation that has already produced impressive demonstrations for mobile robots. Yet two chronic ailments have kept learning robots out of many real deployments. The first is slow convergence: when rewards are sparse—credit arrives only when the robot finally reaches its goal—the agent can wander for a very long time before stumbling onto anything worth repeating, and training becomes expensive and unreliable. The second is safety: a learning robot is, by definition, an imperfect robot, and in human spaces an imperfect robot that collides with people is unacceptable. Worse, standard value-learning procedures can systematically inflate the estimated goodness of risky actions, making danger look deceptively attractive to the learner. The Nanjing researchers designed their method to cure both diseases at once.

Their foundation is the twin delayed deep deterministic policy gradient algorithm, or TD3, introduced by Scott Fujimoto and colleagues in 2018 as a repair kit for continuous-control reinforcement learning. TD3 belongs to the actor-critic family: a neural “actor” learns which action to take—steering, velocity, heading—while a neural “critic” learns to judge how good those actions are. TD3’s three signature mechanisms attack a subtle failure called overestimation bias, in which the critic habitually over-values aggressive actions and lures the robot toward them. First, twin critics score every action independently, and only the more pessimistic estimate is used, keeping unwarranted optimism in check. Second, the actor’s policy is updated only after the critics have had time to settle, preventing self-reinforcing chains of estimation error. Third, small noise is added to target actions during value updates so that estimates do not overfit to single trajectories. For a navigating robot, TD3 outputs smooth, continuous commands—precisely what wheels and actuators demand—and is notably steadier during training than many alternatives. But vanilla TD3 still learns slowly in vast dynamic scenes and treats safety as an afterthought. That is where curiosity and the force field enter.

The first innovation is a curiosity-driven reward mechanism. In psychology and neuroscience, curiosity is understood as intrinsic motivation: humans and animals explore the unknown for its own sake, because novelty itself carries informational value. The Nanjing team engineered the same drive into the robot’s reward system. Beyond the extrinsic rewards for approaching the goal and penalties for collisions, the agent earns internal bonuses whenever it reaches states it has not encountered before. This transforms exploration from a blind random walk into a purposeful quest for the new: the robot is systematically pulled toward unfamiliar regions of its environment and, crucially, toward the kinds of encounters with moving obstacles that a purely goal-hungry learner would never bother to seek out. The payoff shows up in convergence. With curiosity supplying a dense, continuous stream of learning signal in place of rare and delayed goal rewards, the value networks and the policy receive far more informative training data per unit of training time. In the team’s simulations, this intrinsic push measurably accelerated training, letting the robot assemble a repertoire of avoidance maneuvers far earlier than baseline learners.

The second innovation resurrects a forty-year-old classical technique in a new role: the artificial potential field now serves not as the robot’s controller but as its guardian during learning. In the APF tradition, the goal exerts an attractive pull and every obstacle exerts a repulsive push, both diminishing with distance; the vector sum of these influences—the virtual resultant force—points along a sensible, safe direction of travel. Where classic APF used this force to steer robots directly, with all the brittleness that entailed, the Nanjing method feeds the resultant force into the reinforcement learning loop as safety guidance. It acts like an invisible hand on the learning policy, biasing exploration away from collision-prone courses and toward obstacle-avoiding behaviors, while the TD3 policy remains free to discover strategies more sophisticated than any hand-tuned field could express. The outcome is a division of labor: the potential field encodes the physics of “do not hit things” from the very first seconds of training, and the neural network gradually absorbs that lesson into a flexible, learned skill. Guidance and learned competence reinforce each other rather than compete.

The team evaluated the combined framework in simulated dynamic environments populated with moving obstacles, benchmarking it against baseline navigation methods that included conventional reinforcement learning approaches. Two metrics dominated the comparison: navigation success rate—how reliably the robot reached its destination—and training convergence—how quickly learning stabilized at high performance. On both counts, the curiosity- and APF-enhanced TD3 outperformed the baselines, according to the study. Faster convergence matters enormously in practice, because every hour of training carries computational and operational cost, and a robot that needs fewer training episodes is cheaper to deploy and easier to retrain when its mission changes. Higher success rates matter even more, since they quantify the bottom line of autonomy: does the machine actually arrive where it is needed, without colliding along the way. The simulation results were then confronted with reality itself—the researchers validated the method in real-world experiments with a physical robot navigating among dynamic obstacles, confirming that the learned behavior survives the transition from mathematics to motion.

The study lands amid a fast-growing international effort to wed reinforcement learning to dependable mobility. Research groups have reported TD3-based collision-avoidance decision-making for unmanned surface vessels threading through ship traffic, hybrid APF-TD3 schemes for autonomous underwater vehicles planning in three-dimensional unknown waters, and TD3-driven localized path planning for unmanned aerial vehicles. Other laboratories have attacked safety from different directions—shielding learned policies with human feedback, constraining risk with trust-region conditional value-at-risk methods, or steering learning with human demonstrations and sim-to-real transfer. Against this backdrop, the Nanjing contribution stands out for pairing two complementary catalysts around a single TD3 core: intrinsic motivation to solve the exploration problem, and classical field-theoretic guidance to solve the safety problem. The work echoes a broader lesson taking hold in modern robotics—decades of classical control mathematics need not be discarded when learning arrives; recast as priors, guides or guardians, the old theory can make the new learning dramatically better.

The research was supported by the National Natural Science Foundation of China, the China Postdoctoral Science Foundation, and the Fundamental Research Funds for Central Universities, and the authors declare no competing interests. In a transparency gesture that should ease independent replication, the data generated and analyzed in the study are available from the corresponding author upon reasonable request. The team’s division of labor reflects a mature collaboration: Zhang proposed the idea, designed the methodology, implemented the algorithm, ran the experiments and wrote the main manuscript; Zhaolei Li and Hengwei Xu assisted with simulation setup and experimental support; Peng Huang and Shibo Dai contributed to data analysis and experimental validation; Wu supervised the research and revised the manuscript. As with any learning-based system, open questions remain—how gracefully the policy generalizes to unfamiliar scenes, crowds of different densities and noisy sensors—and the authors’ real-world results offer a promising, though early, indication that the answer is favorable.

If the approach spreads, its implications stretch across the robotics economy: delivery robots negotiating crowded sidewalks, service robots gliding through hospitals, autonomous forklifts sharing aisles with human workers, inspection drones weaving through dynamic industrial sites, and eventually driverless vehicles reasoning about erratic traffic. The deeper appeal of the Nanjing method is conceptual. It demonstrates that a robot can be given, within a single training regime, both the restlessness of a curious child and the discipline of a field-shaped dance around obstacles—wonder supplying the drive to learn, potential fields supplying the instinct to survive. Neither alone is sufficient: curiosity without a safety field explores recklessly, and a safety field without curiosity learns slowly and shallowly. Together, they turned a slow, occasionally reckless learner into a faster, safer navigator, verified in simulation and then in the physical world. As machines continue their migration into human spaces, the algorithms that let them move among us may increasingly resemble this one—part explorer, part physicist, and entirely learned.