The technique, called residual-wrench-assisted reinforcement learning, or RWARL, addresses a subtle but fundamental problem in teaching robots to track human motion. Humanoid robots are underactuated at the floating base, meaning they cannot directly command the position and orientation of their own torso the way a crane can command the position of its hook. Everything the torso does must be produced indirectly through joint torques and intermittent contact with the ground. When a reference motion includes an aerial phase or a rapid contact transition, the set of robot states from which the maneuver can be successfully executed becomes sparse and hard to discover. If the training process never visits those states, the policy never learns to handle them, and the robot settles into conservative behavior that trades tracking accuracy for safety.

RWARL works in two stages. In the first stage, the training problem is deliberately relaxed: an auxiliary six-dimensional wrench, consisting of a three-dimensional force and a three-dimensional torque, is applied to the robot’s base alongside the normal joint-level controller. A separate residual policy, observing privileged information about the reference trajectory, outputs small desired offsets in base position and orientation, which a proportional-derivative controller converts into the assisting force and torque. A regularization penalty on wrench usage discourages the policy from leaning too heavily on this artificial support. The result is a policy that can explore difficult regions of the trajectory that would otherwise be unreachable, even though the policy itself is not yet physically feasible.

The second stage is where the scaffolding comes down. The residual scale, which multiplies the strength of the assisting wrench, is annealed monotonically toward zero. Each reduction is triggered only when the maximum segment-wise early-termination failure rate falls below a conservative threshold of 0.05, and the scale decreases by a small fixed step of 5 times 10 to the power of minus 5 per qualifying update. The curriculum is reduction-only: once assistance has been withdrawn, it is never restored. This one-way ratchet forces the policy to adapt to progressively more realistic dynamics while preventing it from redeveloping a dependence on support that will not exist at deployment time.

Withdrawing assistance is not as simple as turning a dial, however. As the residual scale decreases, the dynamics the policy experiences shift continuously, and the early-termination thresholds that worked under full assistance can become too strict, cutting off training episodes on difficult segments before the policy has a chance to learn from them. RWARL therefore adapts its termination bounds on a per-segment basis. Each reference motion is partitioned into temporal segments, and for each segment the team maintains separate threshold scales for three failure conditions: base-height error, torso-orientation error, and end-effector height error. These scales are updated according to observed failure rates within a hysteresis band, so thresholds can be loosened when a segment is failing frequently and tightened again as the policy improves. The team also uses the magnitude of the residual wrench as a smooth proxy for dynamic difficulty when sampling reset states, biasing training toward segments that are either unstable or require large assistance.

A further complication is what machine learning researchers call plasticity loss. Neural networks can lose their ability to adapt when trained under non-stationary conditions, and warm-started models can generalize worse than models trained from scratch. Because residual relaxation continuously shifts the training distribution from assisted to unassisted dynamics, the policy network risks becoming rigid precisely when it most needs to change. The authors mitigate this by adding layer normalization after the linear layers of the actor, critic, and residual policy networks, and by applying a shrink-and-perturb regularization scheme during training. Their ablation experiments show that these mechanisms matter most on longer reference motions: on a 131.48-second dance sequence, the plasticity module reduced base-position and body-position tracking errors by 43.9 percent and 42.4 percent respectively.

The experimental results are striking. Using the BeyondMimic framework as a baseline and training on five reference motions drawn from the AMASS and LAFAN1 datasets, the team found that the final unassisted RWARL policies reduced global body-position tracking error by 8.6 to 37.8 percent compared with the baseline. The most dramatic gain came on jump twist, a highly dynamic motion with an aerial phase, where the successful-frame rate jumped by 41.8 percentage points, from 57.84 percent to 99.61 percent. On breakdance, a contact-rich floor maneuver lasting 37.48 seconds, the method reduced body-position error by 19.7 percent while slightly improving the successful-frame rate. Notably, all evaluations were conducted with residual wrenches completely disabled, so the reported metrics reflect the purely physical capability of the trained controllers.

Perhaps the most intellectually interesting part of the paper is a diagnostic designed to explain why the method works. The authors hypothesize that residual-wrench-assisted training reshapes the closed-loop state distribution, allowing the policy to reach predecessor states from which critical phases of the motion are easier to track. To test this, they identified seven critical phases in the breakdance motion where the baseline policy showed substantially higher tracking error, then rolled out both policies from predecessor states at offsets of 0.25, 0.50, and 1.00 seconds before those phases. Predecessor states collected from the RWARL policy consistently led to higher hit rates than those from the baseline, across both controllers and all offsets. Intriguingly, reference-motion predecessor states were not always the easiest starting points, suggesting that some states in the raw reference are less dynamically compatible with the controller than the states the trained policy naturally reaches.

The ablation studies reveal how tightly coupled the method’s components are. When the team trained a residual-wrench-only variant, adding the wrench mechanism and its annealing schedule to the baseline but disabling adaptive termination and the plasticity module, relaxation failed to complete on jump twist and breakdance within the 40,000-iteration training budget, leaving substantial residual assistance in place. On the three motions where relaxation did complete, the resulting policies showed generally higher tracking errors and lower successful-frame rates than the baseline. Completing the annealing schedule alone, in other words, does not preserve tracking quality; the supporting mechanisms are what allow the policy to follow the relaxation path without collapsing.

The final validation came on hardware. The team deployed the unassisted controllers on a Unitree G1 humanoid robot, using a separate balance controller for transitions before and after each tracked clip. Because some observations available in simulation, such as base linear velocity, are not directly measurable on hardware, the deployed policies were retrained without them, with all other settings kept identical. The hardware experiments demonstrated that behaviors learned through residual-wrench relaxation remain physically executable once the virtual forces are gone. The authors are careful to note the method’s limits: for simple or already well-tracked motions, the scaffold provides little benefit and adds training overhead. But for the difficult, low-reachability transitions that dominate tracking error in agile motion, the approach offers a principled way to expand what a robot can explore during training and then hand back a controller that stands entirely on its own.

Subject of Research: Residual-wrench-assisted reinforcement learning for humanoid motion tracking of highly dynamic and contact-rich movements

Article Title: Assisted exploration of difficult transitions in humanoid motion tracking

Article References: Fu, C., Su, C., Jiang, L., Zhuo, C., Lan, K., Cheng, S., & Wang, H. (2026). Assisted exploration of difficult transitions in humanoid motion tracking. Frontiers in Neurorobotics, 20, Article 1920711. https://doi.org/10.3389/fnbot.2026.1920711

Image Credits: AI Generated

DOI: 10.3389/fnbot.2026.1920711

Keywords: humanoid robots, reinforcement learning, motion tracking, residual wrench, whole-body control, Unitree G1, plasticity loss, curriculum learning, underactuation, contact-rich motion, simulation-to-real, Frontiers in Neurorobotics