Model-Free Reinforcement Learning Controller Improves Trajectory Tracking Precision in Robotic Arms

A simulation-based control study has demonstrated that a model-free deep reinforcement learning architecture can successfully guide a multi-axis robotic arm along complex spatial trajectories, achieving significant error reductions relative to conventional baseline controllers. Researchers developed and evaluated a continuous-action framework utilizing the deep deterministic policy gradient algorithm to govern the movement of a five-degree-of-freedom Mitsubishi RV-2AJ manipulator arm modeled in a high-fidelity physics environment. Rather than relying on exact dynamic system equations that are prone to parameter drift and sensitive to calibration uncertainties, the learning agent acquired control behaviors directly through iterative interactions with the simulated physical environment. The simulation findings revealed that this autonomous learning strategy substantially lowered trajectory deviations across challenging three-dimensional paths, delivering improvements of up to 75 percent in root mean square error and reductions of up to 89 percent in time-weighted error metrics compared to classical proportional-integral-derivative and adaptive neuro-fuzzy controllers.

Articulated robotic manipulators serve as foundational equipment across modern manufacturing, surgical procedures, and hazardous handling operations, where precise end-effector placement is necessary. In traditional engineering settings, robotic control relies heavily on hardware-centered systems and classical control algorithms that necessitate exact kinematic and dynamic modeling. These established frameworks depend on solving inverse kinematics and deriving closed-form equations of motion that quantify link masses, rotational inertias, Coriolis effects, centrifugal forces, and gravitational vectors. However, traditional controllers operate optimally only within highly structured environments where operating conditions remain entirely predictable. In real-world operation, physical manipulators frequently face nonlinear joint dynamics, payload variations, friction fluctuations, and component wear, all of which introduce modeling mismatches that degrade control accuracy unless systems undergo laborious manual recalibration and routine parameter retuning.

Artificial intelligence and data-driven methods offer an alternative approach by decoupling control execution from rigid analytical formulations. While deep learning models can approximate complex nonlinear relationships between sensory inputs and motor outputs, their practical application in real-time robotic manipulation has been constrained by heavy dependencies on large labeled datasets and high computational training overheads. Reinforcement learning addresses these operational barriers by reformulating trajectory control as an active trial-and-error learning process structured around a Markov decision process. In this structure, the robotic controller acts as an autonomous agent that continuously samples physical observations, selects motor commands, and adapts its policy over time in response to numerical performance evaluations. By fusing the feature representation strengths of multi-layer neural networks with the adaptive feedback mechanisms of reinforcement learning, deep reinforcement learning provides a viable model-free paradigm capable of managing continuous action spaces without predefined dynamics models.

To explore the capability of deep reinforcement learning under realistic mechanical constraints, the investigation focused on the kinematic and dynamic profile of the five-degree-of-freedom Mitsubishi RV-2AJ robotic arm. Structurally, the manipulator consists of interconnected links actuated by dedicated servo motors across five rotational joints, comprising a rotating base followed by shoulder, elbow, and dual wrist axes terminating at an end-effector tool point. Kinematic relationships were analytically mapped using the Denavit-Hartenberg convention, linking joint rotations through sequential homogeneous transformation matrices to determine spatial positioning. The underlying physical dynamics were represented by coupled nonlinear equations that account for joint angle vectors, angular velocities, accelerations, an inertia matrix, a Coriolis and centrifugal matrix, and gravity vectors, alongside joint friction torques modeled via electrical motor voltage and torque constants.

Rather than implementing this derived dynamic formulation as a rigid online controller, the research team employed it to construct and validate a dynamic three-dimensional simulation platform within MATLAB and Simulink using the Simscape Multibody physics engine. The dynamic CAD model of the robot established within Simscape incorporated realistic physical mass distributions, link dimensions, and joint actuator constraints, matching the theoretical equations across comparative multi-joint angle trajectories. This verified physical engine served as the operational training ground for the deep deterministic policy gradient algorithm, allowing the agent to receive comprehensive real-time state feedback.

The deep deterministic policy gradient framework operates as an actor-critic architecture tailored for continuous control domains where discrete decision models are inadequate. In this system, an online actor network determines continuous motor torque commands for each of the five joint actuators based directly on incoming kinematic observations, while an online critic network evaluates the long-term expected reward associated with those actions. To stabilize network training and prevent divergence during backpropagation, the system utilizes target actor and critic networks whose weights are updated through soft target synchronization governed by a smoothing coefficient of 0.001. Training transitions comprising current states, selected joint torques, numerical rewards, and subsequent states were archived within an experience replay buffer, from which randomized mini-batches were iteratively sampled to update network weights via deterministic policy gradient ascent and mean squared Bellman error minimization.

A critical engineering element in training the learning agent was the development of a tailored reward mechanism inspired by artificial potential field theory. To promote both tracking accuracy and mechanical longevity, the reward formulation balanced path convergence against actuator exertion by penalizing both joint coordinate deviations and excessive motor torque magnitudes, using a weighting factor lambda set to 0.05. This potential-field formulation effectively established an attractive numerical gradient toward the desired trajectory coordinates while imposing a repulsive penalty on accumulated tracking errors. Furthermore, the algorithmic implementation incorporated critic-layer normalization and prioritized replay sampling, achieving a 28 percent reduction in action-value estimation variance and ensuring reliable convergence throughout training iterations.

The performance of the trained reinforcement learning controller was rigorously evaluated against two widely recognized control strategies: a standard proportional-integral-derivative controller and an adaptive neuro-fuzzy inference system. The comparative testing protocol encompassed three distinct continuous spatial reference trajectories designed to evaluate dynamic responsiveness: a piecewise linear N-shaped trajectory, a three-dimensional helical path, and an Archimedean planar helical spiral. The N-shaped path challenged the controllers with abrupt directional transitions and sharp diagonal velocity reversals. The helical trajectory evaluated simultaneous circular motion in the horizontal plane combined with constant vertical ascent along the vertical axis. The spiral trajectory tested real-time tracking across continuous radial expansion at constant elevation, requiring smooth multivariable coordinate management under varying rotational radii.

Quantitative evaluations demonstrated that the deep deterministic policy gradient controller consistently outperformed both comparison controllers across all three spatial benchmark paths. In evaluations on the N-shaped trajectory, the learning-based controller reduced tracking errors across multiple evaluation criteria, including integral absolute error, integral square error, integral time absolute error, integral time square error, and root mean square error, showing improvements ranging between 46 percent and 89 percent. The classical proportional-integral-derivative controller exhibited notable limitations, including phase lag, transient overshoot, and progressive error accumulation along sharp transitions, resulting from its static gain design. While the adaptive neuro-fuzzy controller adapted more effectively than the fixed-gain model, its reliance on a static fuzzy rule base constrained its ability to compensate for coupled nonlinear dynamics during rapid directional shifts, achieving roughly 40 to 50 percent error reductions compared to the baseline.

Consistent performance trends were recorded across the continuous curvature tests. Along the three-dimensional helical path, the policy gradient controller reduced tracking error metrics by 54 percent to 89 percent relative to the alternative methods. During the planar spiral path tests, the algorithm achieved an integral absolute error of 0.95, representing a 55.8 percent reduction over the neuro-fuzzy controller and a 75.3 percent reduction relative to the proportional-integral-derivative controller. Integral square error values were similarly suppressed by 55.3 percent and 77.5 percent against the respective baselines. Time-weighted indices showed pronounced gains, with integral time absolute error dropping to 0.41 and integral time square error reaching 0.24, reflecting improvements of 70.3 percent and 75.5 percent compared to the neuro-fuzzy system and 85.8 percent and 89.6 percent compared to the proportional-integral-derivative system. The overall root mean square error for the spiral trajectory reached 0.10, cutting error magnitudes by exactly 50 percent versus the neuro-fuzzy model and 71.4 percent against the standard controller.

Beyond classical control baselines, the study examined the relative merits of the deep deterministic policy gradient method against other advanced reinforcement learning architectures, specifically twin delayed deep deterministic policy gradient and soft actor-critic models. While the soft actor-critic algorithm achieved a marginally lower final tracking error margin of approximately two percent, it demanded 9.1 hours of training time and resulted in higher overall energy expenditure, rendering it less practical for real-time deployment. In contrast, the deep deterministic policy gradient framework completed training within 6.2 hours and maintained an actuator torque variance of 0.08, compared to 0.11 for the twin-delayed variant and 0.14 for the soft actor-critic model. This combination of lower torque fluctuation, reduced computational burden, and rapid inference latency supported the selected architecture as an effective balance between motion precision, training efficiency, and actuator protection.

Despite these simulated achievements, the researchers emphasized that several technical challenges and operational boundaries remain before physical deployment can occur. The stability and final tracking accuracy of the policy gradient controller remain highly sensitive to hyperparameter selection, requiring careful calibration of learning rates, reward scaling terms, discount factors, and exploration noise decay schedules to prevent unstable behavior. Furthermore, all evaluations were conducted within the simulated environment of MATLAB, Simulink, and Simscape Multibody. The authors noted that physical deployment onto an actual hardware prototype will require explicit validation of sim-to-real transfer techniques, domain randomization across physical joint friction, and mitigation of unmodeled sensor noise. Planned future investigations aim to explore hybrid architectures incorporating sliding-mode control, attention mechanisms, and meta-learning frameworks to enhance policy transfer onto physical manipulator hardware.

Subject of Research: Model-free reinforcement learning for robotic arm trajectory tracking

Article Title: Model-free trajectory tracking control of a 5-DOF mitsubishi robotic arm using deep deterministic policy gradient algorithm

Article References: Ben Hazem, Z., Guler, N., & Altaif, A. H. (2025). Model-free trajectory tracking control of a 5-DOF mitsubishi robotic arm using deep deterministic policy gradient algorithm. Discover Robotics, 1(1), Article 4. https://doi.org/10.1007/s44430-025-00004-2

Image Credits: AI Generated

DOI: 10.1007/s44430-025-00004-2

Keywords: deep deterministic policy gradient, robotic arm control, trajectory tracking, Mitsubishi RV-2AJ, reinforcement learning, actor-critic architecture, MATLAB Simulink, Simscape Multibody, continuous action space, adaptive control, proportional integral derivative, adaptive neuro-fuzzy inference system, root mean square error, torque optimization, nonlinear dynamics, potential field reward