Robots Learn to See the Future With Gaussian Splatting for Two-Armed Tasks

Teaching a robot to use two arms at once is one of the hardest problems in robotics. Unlike a single arm, bimanual manipulation demands precise temporal coordination between two manipulators, awareness of complex object interactions, and the ability to generalize across messy real-world conditions such as changing lighting, cluttered backgrounds, and unexpected tabletop distractors. Now, a research team from the Northwest Institute of Mechanical and Electrical Engineering in Xianyang, China, together with a collaborator from Jiangxi Guoke Defence Group, has unveiled a framework that could change how robots perceive and act in these scenarios. Their system, called GSP-3D, fuses three powerful ideas—generalizable 3D Gaussian Splatting, diffusion-based policy learning, and a predictive world model—into a single architecture designed for long-horizon bimanual manipulation. The work, published in Frontiers in Neurorobotics, shows consistent performance gains over existing methods while adding only minimal computational overhead.

The core insight behind GSP-3D is that the visual representations robots rely on are often too impoverished for the tasks they must perform. Most current diffusion-based visuomotor policies, which have become a dominant paradigm for robot learning, condition their action generation on sparse point clouds derived from depth cameras. These point clouds capture discrete samples of object surfaces but lack structural completeness, providing limited information about surface orientation, volumetric occupancy, and fine spatial relationships. Moreover, diffusion policies are typically optimized solely through behavior imitation objectives, with no explicit constraint on how the scene should evolve in the future. The result, the authors argue, is action generation that can be temporally inconsistent and physically implausible—exactly the kind of failure that cripples coordinated two-arm tasks where both arms and the manipulated objects must remain in sync over extended horizons.

GSP-3D addresses this by replacing sparse point clouds with explicit 3D Gaussian primitives. 3D Gaussian Splatting, or 3DGS, represents a scene as a collection of anisotropic 3D Gaussians, each parameterized by a mean position, a covariance matrix, an opacity value, and spherical harmonic coefficients for view-dependent color. Rendering is performed through differentiable rasterization, enabling real-time novel view synthesis with explicit geometry that is compatible with physics-based reasoning. The technology has transformed computer vision and graphics, but its potential for robotic policy learning remained largely unexplored—partly because conventional 3DGS requires costly per-scene optimization, making it impractical for online policy learning.

The team’s solution is a component they call the Generalizable Gaussian Regressor, or GGR. This lightweight encoder-decoder predicts per-pixel 3D Gaussian parameters directly from a single RGB-D frame, eliminating the need for scene-specific optimization and enabling real-time deployment. The architecture follows a U-Net-style design with three stages: a shared encoder processes the RGB-D input to extract multi-scale spatial features, a depth-to-point cloud back-projection module uses camera intrinsics to unproject each pixel into 3D space, and a lightweight point-based decoder regresses Gaussian parameters for each point. The covariance is parameterized as a scaled rotation with scale and quaternion components, opacity passes through a sigmoid activation, and color coefficients are normalized. GGR is trained end-to-end with a differentiable rasterizer that renders the predicted Gaussians back into image space, supervising both RGB reconstruction and depth rendering. Training data comes from the RoboTwin-OD dataset under aggressive domain randomization, including cluttered scenes, varied lighting, and diverse background textures.

Robustness under visual perturbation is engineered into GGR through several complementary mechanisms. The U-Net encoder is trained with random color jitter, Gaussian blur, and simulated depth sensor noise, forcing the feature extractor to learn illumination-invariant representations. The depth back-projection uses median-filtered depth maps to mitigate outlier readings caused by reflective surfaces or sensor artifacts. The differentiable rendering loss implicitly regularizes the predicted Gaussians by enforcing multi-view photometric and geometric consistency, encouraging the regressor to reconstruct the scene faithfully even under challenging lighting or texture variations. Downstream, sparse voxelization and max-pooling aggregation naturally discard outlier Gaussians with low opacity or inconsistent covariance, and the diffusion policy itself is trained on domain-randomized data that teaches the denoising network to treat moderate Gaussian prediction errors as part of the observation noise distribution.

Once the Gaussians are predicted, they must be converted into a form the diffusion policy can consume. Directly feeding all Gaussians into the diffusion network would be computationally prohibitive, so GSP-3D aggregates them through a sparse 3D convolution backbone operating on voxelized Gaussian features at multiple resolutions. For each voxel, the Gaussian parameters inside are aggregated via probabilistic max-pooling, producing a multi-scale feature volume. This volume is then flattened and projected into a compact latent vector through a transformer encoder with cross-attention, using learnable query vectors. The resulting 3DGS latent code serves as visual conditioning for the diffusion denoising network, injected via cross-attention layers within the denoising UNet, alongside robot proprioception and the diffusion timestep. The policy generates action chunks of six future steps conditioned on three observation steps, trained with an AdamW optimizer and a cosine learning rate scheduler.

The most distinctive element of GSP-3D is its 3DGS-based world model, which the authors call GS-WM. Unlike prior world models that operate in low-dimensional latent spaces or point cloud representations, GS-WM operates directly in the 3D Gaussian primitive state space, capturing fine-grained geometric and temporal scene dynamics that would be lost in lower-dimensional representations. The world model is a conditional dynamics model that learns a transition function over the action space of joint position targets for both arms. Given a history of past Gaussian states and an action chunk, a transformer-based dynamics predictor forecasts future Gaussian sets. Each Gaussian is projected into a token embedding by vectorizing its covariance matrix, and the transformer models temporal dependencies through causal attention, outputting residual updates to the Gaussian parameters while keeping opacities bounded. The world model is trained separately on offline expert demonstration data with a supervised prediction objective, then frozen during policy learning.

During policy training, the frozen world model computes the prediction error between the forecast future Gaussians and ground-truth Gaussians extracted from subsequent observations. Critically, this auxiliary loss operates exclusively during training, not during real-world deployment. At test time, the diffusion policy executes actions based solely on current observations and proprioception, without requiring any future ground truth or online prediction from the world model. The auxiliary loss simply regularizes the policy’s training objective, encouraging it to favor actions that lead to temporally coherent scene transitions as modeled by GS-WM. The authors are careful to clarify that GS-WM is not a physics simulator and does not guarantee physically accurate predictions under complex interactions such as deformation or collision. Instead, it learns empirical transition patterns from expert demonstrations that implicitly encode physical constraints—objects move continuously, occlusions evolve smoothly—and penalizes large or abrupt changes in Gaussian parameters that deviate from these learned patterns.

The team evaluated GSP-3D on the RoboTwin 2.0 benchmark, a scalable simulation platform for bimanual robotic manipulation, using Aloha-AgileX dual-arm platforms. All methods, including the baselines RDT, Pi0, ACT, DP, and DP3, were trained and tested on exactly the same set of 50 expert demonstrations per task, with the same 14-dimensional joint position action space and the same domain-randomized evaluation settings of 100 trials per task. The tasks included clicking an alarm clock, handing over a block, moving a can pot, placing an empty cup, rotating a QR code, scanning an object, placing a fan, and pressing a stapler. GSP-3D consistently outperformed all baselines, achieving success rates of 84 percent on clicking the alarm clock, 78 percent on handing over a block, 76 percent on moving the can pot, 74 percent on placing the empty cup, 82 percent on rotating the QR code, 40 percent on scanning the object, 42 percent on placing the fan, and 76 percent on pressing the stapler. The strongest baseline, DP3, reached 75, 70, 70, 65, 74, 31, 36, and 69 percent respectively on the same tasks. An ablation study confirmed that removing either the 3DGS representation or the world model auxiliary loss degraded performance, with the full system achieving the highest success rates across every task.

Computational efficiency is a notable strength. Compared to DP3, GSP-3D introduces only minimal memory overhead—average parameter counts of roughly 265.7 million versus 262.4 million, peak memory of 2.384 gigabytes versus 2.356 gigabytes, and average inference time of 1.52 seconds versus 1.40 seconds—while achieving higher success rates. The authors acknowledge limitations, including occasional failure cases under occlusion or dynamic tasks, sensitivity to the world model loss weight, degraded robustness under extreme lighting or reflective surfaces, unvalidated sim-to-real transfer, quadratic scaling of the world model with Gaussian count, and unsuitability for deformable objects, extremely long horizons, or high-frequency control. Future work includes evaluating GSP-3D on real-world hardware, extending the framework to dynamic objects and deformable manipulation, incorporating online adaptation of the world model during deployment, and integrating more principled physical inductive biases such as collision detection or rigid-body dynamics priors. For now, the findings demonstrate that integrating explicit 3D Gaussian representations with diffusion policies offers an efficient and robust solution for long-horizon, temporally coherent bimanual manipulation—effectively addressing the generalization and consistency challenges that limit current approaches.

Subject of Research: GSP-3D: generalizable 3D Gaussian Splatting with diffusion policy and world model for long-horizon bimanual manipulation

Article Title: GSP-3D: generalizable 3D Gaussian Splatting with diffusion policy and world model for long-horizon bimanual manipulation

Article References: Liu, X., Huang, J., Wei, S., Hou, L., Qiao, J., Liu, A., Huang, H., & Xie, F. (2026). GSP-3D: generalizable 3D Gaussian Splatting with diffusion policy and world model for long-horizon bimanual manipulation. Frontiers in Neurorobotics, 20, Article 1867375. https://doi.org/10.3389/fnbot.2026.1867375

Image Credits: AI Generated

DOI: 10.3389/fnbot.2026.1867375

Keywords: GSP-3D, generalizable, Gaussian, Splatting, diffusion, policy, world, model, long-horizon, bimanual, manipulation, scientific research