The study, conducted by Haotian Zhang, Chen Wang, and Jun Moon of the Department of Electrical Engineering at Hanyang University in Seoul, introduces TACCO, short for TD3+BC with Actor-Critic Contrastive Optimization. The method builds on TD3+BC, a minimalist and widely used offline reinforcement learning algorithm, and augments it with supervised contrastive learning branches that explicitly model out-of-distribution actions. The authors define out-of-distribution actions as those that lack local data support in a given state, do not match the current state, or exhibit low reliability. Rather than treating these problematic actions as an afterthought, TACCO makes them a first-class target of representation learning, reshaping the internal embeddings of the agent so that unreliable actions become visibly distinguishable from trustworthy ones.

To understand why this matters, it helps to see how offline reinforcement learning goes wrong. When an agent learns only from a fixed dataset, its value function, the critic that estimates the long-term reward of taking an action in a state, is only trustworthy for state-action pairs that actually appear in the data. During policy improvement, the actor proposes new actions that may drift outside the support of the dataset. The critic, having never seen these pairs, tends to overestimate their value, a phenomenon known as value overestimation. The actor then chases these inflated estimates, proposing ever more exotic actions, and the whole learning process spirals into degradation. This feedback loop between an overconfident critic and an adventurous actor is the central failure mode of offline reinforcement learning.

Existing approaches generally respond in one of two ways. Policy-constraint methods, such as TD3+BC itself, tether the learned policy to the behavior policy recorded in the data, penalizing any action that strays too far from what the dataset contains. Conservative value regularization methods, exemplified by CQL, pessimistically lower the estimated values of out-of-distribution actions so the critic never overhypes them. Both families work, but as the Hanyang team points out, few methods explicitly model the problematic actions from a representation learning perspective. That is the gap TACCO fills: instead of only constraining behavior or deflating values, it teaches the network’s internal representations to separate reliable actions from unreliable ones, providing a structural safeguard that complements the existing toolkit.

TACCO comes in two variants, each targeting a different side of the actor-critic architecture and a different reward regime. The first, TACCO-A, is designed for dense reward environments, where feedback signal is abundant. It attaches a supervised contrastive learning branch to the actor. The actor consists of a shared state-encoding backbone, a standard policy output head, and an additional embedding head that concatenates the hidden state representation with the proposed action and projects the result into a 64-dimensional embedding. Contrastive learning then operates on three kinds of actions: data-driven actions drawn from the offline dataset, random actions, and cross-state policy actions. The training objective pulls embeddings of dataset-supported actions together while pushing embeddings of random and mismatched actions apart, so the actor’s representation space develops a clear geometry in which out-of-distribution actions stand out and can be suppressed.

The second variant, TACCO-B, is tailored to sparse reward environments, where rewards arrive rarely and unreliable value estimates are especially corrosive. Here the contrastive branch moves to the critic. TACCO-B filters training samples on the critic side based on two criteria: high Q-values and low uncertainty. Samples that pass both filters are treated as reliable, while the remainder are flagged as suspect. The method then applies contrastive constraints to both reliable and remaining samples simultaneously, encouraging the critic’s shared state-action encoder, a small network ending in a LayerNorm-normalized 64-dimensional embedding, to learn discriminative representations that reflect this reliability structure. By shaping the critic’s representation space rather than merely clipping its outputs, TACCO-B aims to stabilize value learning even when the reward signal is thin and misleading bootstrapped targets are easy to come by.

The architectural details are deliberately modest, which is part of the method’s appeal. Both variants retain the dual Q-network structure of TD3, with two independent critics that help mitigate function approximation error, and both keep the behavioral cloning term of TD3+BC that anchors the policy to the dataset. The contrastive modules are small multilayer perceptrons, and the authors report that performance is relatively robust to the weighting of the contrastive loss. Sensitivity analyses reveal a more interesting story for the other knobs: in TACCO-A, relaxing the behavioral cloning weight makes the policy more adventurous but destabilizes training, and increasing the contrastive weight steadily compensates for that instability. In TACCO-B, the fraction of high-value samples used for filtering must be tuned carefully, since too few samples starve the model of coverage while too many reintroduce the very low-quality actions the filter was meant to exclude.

The empirical evaluation is broad. The team compared TACCO against six state-of-the-art offline reinforcement learning algorithms, including CQL, TD3+BC, the Decision Transformer, IQL, DTQL, and O-DICE, across 27 tasks spanning the MuJoCo locomotion suite and the Adroit dexterous manipulation suite within the D4RL benchmark. All methods were trained under an identical protocol: one million gradient steps, evaluation every five thousand steps, a batch size of 256, and results averaged over five random seeds with 95 percent confidence intervals, using the standard D4RL score normalization. According to the authors, TACCO achieved superior performance on most of these tasks, which they interpret as validation that mitigating out-of-distribution action interference through contrastive optimization enhances policy stability, in both dense and sparse reward settings.

The broader significance of the work lies in its reframing of a classic problem. Distribution shift has usually been handled as a constraint or a penalty, a blunt instrument applied to actions or values. TACCO instead treats it as a representation problem: if the network’s embeddings make the boundary between supported and unsupported actions explicit, then both the actor and the critic can exploit that structure during training. This connects offline reinforcement learning to the wider wave of contrastive learning that has transformed vision and speech processing, and to a growing line of research that uses contrastive objectives for goal-conditioned control and planning. It also suggests a practical path forward, since the contrastive branches add little architectural complexity on top of a minimalist baseline that many practitioners already deploy.

Challenges remain before such methods reach real-world robotics and industrial control, where offline data is often suboptimal, heterogeneous, and collected under shifting conditions. The authors note that code will be made available on request, and the work was supported by Korean government research programs through KETEP, MOTIE, and IITP. As offline reinforcement learning matures into a serious option for settings where online exploration is prohibitive, techniques like TACCO point toward a hybrid future: conservative enough to trust the data, expressive enough to improve on it, and, crucially, aware of exactly where its own knowledge ends.

Subject of Research: Contrastive actor-critic optimization for mitigating out-of-distribution actions in offline reinforcement learning

Article Title: TACCO: Actor–Critic Contrastive Optimization for OOD Action Mitigation in Offline Reinforcement Learning

Article References: Zhang, H., Wang, C., & Moon, J. (2026). TACCO: Actor–Critic Contrastive Optimization for OOD Action Mitigation in Offline Reinforcement Learning. Machine Learning, 115(10), Article 241. https://doi.org/10.1007/s10994-026-07170-3

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07170-3

Keywords: offline reinforcement learning, contrastive learning, out-of-distribution actions, TD3+BC, actor-critic, D4RL benchmark, value overestimation, distribution shift, representation learning, MuJoCo, Adroit, machine learning

The framework the team developed sits at the junction of two major threads in modern reinforcement learning. The first thread is offline-to-online learning, a pipeline in which a policy is first pretrained on static data and then fine-tuned with limited real-world or simulated interaction. This two-stage approach preserves the sample efficiency of offline learning while allowing some online adaptation, and it has become increasingly popular for robotics because it minimizes dangerous, expensive online exploration. The second thread is adversarial training, borrowed in spirit from the adversarial examples literature in computer vision, where imperceptible perturbations to images expose shocking weaknesses in neural networks. In reinforcement learning, adversarial methods have been used to perturb state observations or actions to stress-test policies. The Chiba team’s contribution is to fuse these ideas: after offline pretraining on clean data, they fine-tune the policy while injecting perturbations into the actions it executes, forcing the policy to learn compensatory behavior that restores performance even when its commanded actions are corrupted.

The technical details matter here, because naive adversarial training can destroy as much as it hardens. In their main experiments, the researchers pretrained policies using TD3+BC, an offline reinforcement learning algorithm that combines the twin delayed deep deterministic policy gradient architecture with a behavior cloning regularizer to keep the policy close to the dataset. Perturbations were generated using Differential Evolution, an evolutionary optimization method that searches for action corruptions which minimize the critic’s value estimate, effectively finding the worst-case damage that can be inflicted within a bounded perturbation budget. During fine-tuning, these adversarial perturbations were applied stochastically: with some probability, the action the policy intended to execute was replaced or corrupted by the adversarial version. The fine-tuning process then continued online in continuous-control locomotion environments, with the replay buffer mixing fresh adversarial experience with a controlled proportion of the original offline data. This combination allowed the policy to retain its offline competence while progressively building resistance to action-space disturbances.

One of the study’s central innovations is a performance-aware curriculum that governs how aggressively the perturbations are applied. In a linear curriculum, the perturbation probability increases at a fixed rate regardless of how well the policy is actually learning. The team showed both theoretically and empirically why this can be dangerous. They formalized the problem with a lemma on safe curriculum progression: if performance degrades smoothly as perturbation probability increases, at a rate captured by a Lipschitz constant, then increasing difficulty is only safe when the expected performance improvement from policy optimization exceeds the loss caused by the jump in perturbation level. A linear schedule ignores this condition entirely. When learning progress slows but the perturbation probability keeps climbing, the difficulty shift dominates the improvement, and nominal performance collapses. This analysis explains a persistent pathology observed in adversarial curricula: policies that become excellent at surviving perturbations but forget how to walk properly when nothing is wrong at all.

The adaptive curriculum the researchers designed solves this by listening to the learning signal. Rather than marching to a fixed schedule, the perturbation probability is updated using an exponential moving average of the policy’s normalized performance score. When the policy is improving steadily, the curriculum tightens the screws and increases perturbation frequency; when performance stalls or fluctuates, the curriculum eases off. The update uses a bounded step size, which directly enforces the safe progression condition from the theoretical analysis, and the exponential moving average smooths the noisy episodic returns that make naive adaptation unstable. Hyperparameter analysis revealed how the smoothing factor and the step-size parameter shape the trajectory of the perturbation probability: larger smoothing factors make the curriculum more sensitive to performance swings, while larger step sizes push the overall perturbation level higher. The combination that maximized average performance across normal, random, and adversarial evaluation conditions was selected for the headline experiments, and the adaptive strategy consistently avoided the nominal performance degradation that plagued the linear alternative.

The experimental results, conducted on standard continuous-control locomotion benchmarks from the D4RL dataset suite including Hopper, HalfCheetah, and Ant, demonstrate that adversarial fine-tuning delivers robustness gains that are both substantial and statistically significant. Fine-tuned policies consistently outperformed offline-only baselines under both random and adversarial perturbation conditions, and Welch’s t-tests confirmed that most of these improvements were significant at conventional thresholds. Equally important, the fine-tuning converged substantially faster than training a robust policy from scratch, validating the core premise of the offline-to-online pipeline: the expensive work of acquiring basic competence can be done safely offline, and the comparatively cheap work of hardening that competence can then be done online. The study also found that matching the fine-tuning conditions to the evaluation conditions produced the strongest robustness, a finding with practical implications for deployment. If a robot is expected to operate with degraded actuators, it should be fine-tuned under perturbations resembling that degradation, not under a generic disturbance model.

The researchers went to considerable lengths to show that their framework is not an artifact of one particular algorithm. Additional experiments replaced TD3+BC with Implicit Q-Learning for pretraining and Soft Actor-Critic for online fine-tuning, demonstrating that the adversarial fine-tuning recipe generalizes across both policy-constraint-based and value-based offline methods. The robustness improvements persisted, although the stability-robustness trade-off proved environment-dependent, appearing most sharply in Hopper, where precise balance control makes the task unusually sensitive to robustness-oriented updates. The team also compared static perturbations generated in advance by Differential Evolution against step-wise perturbations generated on the fly by Projected Gradient Descent, which attacks the critic’s value estimates at every interaction step. The comparison revealed a subtle but consequential truth: robustness depends not only on perturbation magnitude but on the temporal structure of the perturbations themselves. Policies fine-tuned with static perturbations excelled under static adversarial evaluation and preserved nominal performance, while policies fine-tuned with step-wise PGD attacks excelled under step-wise attacks at the cost of slight conservatism in normal conditions.

Beyond the specific algorithmic contributions, the study carries a broader message about how robust systems should be built. The traditional alternatives to adversarial fine-tuning, such as domain randomization, inject randomness into training dynamics or environments and hope that the resulting policy averages over the noise. Adversarial training instead targets the worst case, teaching the policy to compensate for the most damaging corruption within a bounded budget. The Chiba team’s work suggests these two philosophies can be reconciled within a curriculum that respects the policy’s actual learning progress, and their theoretical safety condition offers a principled template for curriculum design far beyond robotics. It also echoes a growing recognition across machine learning, from malware detection to remote sensing, that adversarial stress testing is not merely a security exercise but a training strategy: systems that have grappled with their worst case during development handle the messy middle of reality far better than systems trained only on the ideal case.

The practical implications extend to any deployed control system where failure is expensive. Industrial manipulators with wearing actuators, legged robots traversing uneven terrain, autonomous vehicles whose control signals pass through noisy channels, and even medical dosing systems all face action-space disturbances that offline datasets rarely capture. A framework that requires no additional real-world risk during the bulk of learning, adapts its own training difficulty to the pace of progress, and comes with a formal guarantee on when increasing that difficulty is safe offers a compelling template for the next generation of resilient robot controllers. The authors, whose work was supported by JSPS KAKENHI Grant Number JP23K24914, have released their methodology on publicly available D4RL datasets and the open-source d3rlpy library, lowering the barrier for other laboratories to adopt and extend the approach. As robots move out of laboratories and into factories, hospitals, and homes, the ability to fail gracefully rather than catastrophically may prove to be the difference between machines we trust and machines we merely tolerate. Adversarial fine-tuning, this research suggests, is a major step toward the former.

Subject of Research: Adversarial fine-tuning in offline-to-online reinforcement learning for robust robot control

Article Title: Adversarial fine-tuning in offline-to-online reinforcement learning for robust robot control

Article References: Ayabe, S., Kera, H., & Kawamoto, K. (2026). Adversarial fine-tuning in offline-to-online reinforcement learning for robust robot control. Applied Intelligence, 56(14), Article 420. https://doi.org/10.1007/s10489-026-07455-z

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07455-z

Keywords: reinforcement learning, offline reinforcement learning, offline-to-online learning, adversarial training, robot control, robustness, curriculum learning, continuous control, TD3+BC, D4RL, actuator faults, machine learning