The research, published in Frontiers in Control Engineering by Ahmad Al Ali and Zheng H. Zhu of York University’s Mechanical Engineering Department, tackles a challenge that has limited space robotics for decades. Space robotic systems come in two flavors. Free-flying robots use thrusters to actively hold the base spacecraft steady while the arm works, which makes control straightforward but burns precious fuel. Free-floating robots let the base drift naturally in response to the arm’s motions, conserving momentum and stretching the mission’s operational life. The trade-off is mathematical chaos: the position and orientation of the arm’s end effector depend not just on the current joint angles but on the entire history of joint and base velocities, a nonholonomic coupling that renders classical fixed-base planning techniques ineffective.

Earlier attempts to tame this coupling relied on inverse kinematics respecting angular momentum constraints, but dynamic singularities, unpredictable from kinematics alone, could derail the arm mid-task. Optimization approaches such as particle swarm optimization and sliding mode control improved matters but demanded heavy computation and required complete re-optimization whenever initial conditions changed. Reinforcement learning offered a compelling alternative. In this framework, an agent learns a control policy through trial and error, guided by a reward function, within a Markov Decision Process. Crucially, once training ends, the entire learned policy is compressed into the weights and biases of a neural network, lightweight enough to run on modest space-qualified hardware, a decisive advantage where computing power is scarce.

Prior reinforcement learning studies on space manipulators mostly targeted simpler systems, often three degrees of freedom, and frequently ignored practical constraints. The authors’ survey of the field found that most work neglected joint velocity limits, avoided quaternion-based attitude representation, and omitted obstacle avoidance entirely. Their new work addresses all three gaps simultaneously, applying the Deep Deterministic Policy Gradient algorithm, which combines an actor network that outputs continuous joint torques with a critic network that estimates the expected discounted return of state-action pairs. The architecture uses two hidden layers of 400 and 300 neurons, a replay buffer of 50,000 transitions, and a discount factor of 0.99, trained in a MATLAB and Simulink simulation of a six-link manipulator on a 1,700-kilogram base spacecraft.

The heart of the method is a carefully engineered reward function. A guidance term rewards the agent for shrinking both the positional distance and the orientation difference between the end effector and the target. Once the end effector enters a predefined capture zone, large positive rewards kick in to hold it there while fine alignment proceeds. Large negative penalties punish self-collision between links, contact with the target, and encroachment on an external obstacle placed directly in the approach path. A velocity penalty, proportional to the squared sum of joint angular velocities, discourages jerky motion and produces smoother trajectories.

One of the study’s most elegant technical choices is the use of quaternions instead of Euler angles to represent orientation. Euler angle differences discard information about rotation sequence, inflating the search space during learning and sometimes causing divergence. Quaternions, four-component representations of rotation, avoid the singularities that plague Euler angles and preserve full rotational information. The orientation error is computed from the relative quaternion between the end effector and the target, giving the learning agent a cleaner, more informative signal and accelerating convergence.

The simulation results demonstrate the approach across three progressively harder scenarios. In the first, without velocity constraints, the agent guided the end effector into the capture range within roughly 6 seconds of the 10-second episode, achieving a minimum distance error of 0.058 meters and an orientation error of 4.85 degrees, while threading the arm past the obstacle without a single collision. Training converged within about 5,000 episodes, with the guidance term pulling the end effector toward the target and the penalty terms actively preventing collisions throughout.

The second scenario added the velocity penalty, and the effect was striking. The squared sum of joint velocities along the path barely exceeded 80, compared with peaks of 160 in the unconstrained case, and the trajectory converged more smoothly during the first 3 seconds. The final pose errors were 0.088 meters and 6.61 degrees, a modest cost in precision for a dramatically smoother path, exactly the kind of motion a real gripper needs before capturing a tumbling satellite.

The third scenario confronted the elephant in every real mission: imperfect sensors. Vision-based estimation of a target’s pose is inherently noisy, so the researchers injected white noise with a 10-to-1 signal-to-noise ratio into the observed target pose and obstacle position during training. The agent still succeeded, reaching a distance error of 0.048 meters and an orientation error of 7.79 degrees, with somewhat more oscillatory joint torques reflecting the added uncertainty. The most revealing finding came from the cross-test: the noise-trained agent performed flawlessly in a noise-free environment, while the agent trained on clean data failed when noise was introduced. Training under realistic conditions, it turns out, is not optional but essential.

The implications reach well beyond one simulated arm. As orbital servicing, debris removal, and in-space assembly grow into commercial industries, robots that can plan their own collision-free paths without thruster support will be indispensable. Because the trained policy is just a set of neural network weights, deployment requires no onboard supercomputer, and the modular reward function can be retuned for different missions, adding torque or energy penalties as needed. The authors describe their work as a theoretical foundation for reinforcement learning in complex free-floating operations, and the results suggest that the next generation of orbital robots may learn to move the way astronauts do: gracefully, cautiously, and with an instinct for danger.

Subject of Research: Reinforcement learning-based path planning for a free-floating space robotic manipulator with collision avoidance and observation noise

Article Title: Reinforcement learning for path planning of free-floating space robotic manipulator with collision avoidance and observation noise

Article References: Al Ali, A., & Zhu, Z. H. (2024). Reinforcement learning for path planning of free-floating space robotic manipulator with collision avoidance and observation noise. Frontiers in Control Engineering, 5, Article 1394668. https://doi.org/10.3389/fcteg.2024.1394668

Image Credits: AI Generated

DOI: 10.3389/fcteg.2024.1394668

Keywords: space robotics, reinforcement learning, DDPG, free-floating manipulator, path planning, collision avoidance, quaternions, observation noise, on-orbit servicing, deep learning, spacecraft dynamics, Reinforcement