Learning vision-based manipulation from one human video with open-world object graphs

In a development that could fundamentally change how robots acquire new skills, researchers at the University of Texas at Austin have unveiled ORION, an algorithm that enables a robotic arm to watch a single video of a human performing a task—recorded on an ordinary iPhone—and then reproduce that task in a completely different environment. The system, described in the journal Autonomous Robots, achieved an average success rate of 74.4 percent across real-world evaluations spanning short-horizon and long-horizon manipulation tasks, even when the videos contained no depth information at all.

The central challenge the researchers set out to solve is one that has long frustrated the field of robot learning: how to teach a robot from a single, actionless human video, without any robot demonstrations, task-specific object models, or prior training in the environment where the robot will operate. Most previous approaches to learning from human videos have relied on pre-training visual representations or value functions across large datasets, but they fail to explicitly capture how objects interact and move in three-dimensional space—the very information a robot needs to execute a manipulation. As a result, these systems typically require separate teleoperation data for every combination of objects, locations, and lighting conditions, an approach that scales poorly to the messy variability of the real world.

ORION, short for Open-woRld video ImitatiON, takes a fundamentally different, object-centric route. The team’s key insight is that most manipulation tasks can be adequately described by the interactions between objects, and that task completion hinges on reaching specific intermediate states, or subgoals. Rather than reconstructing the human demonstrator’s body and hand motions—as contemporary methods like OKAMI do by retargeting human kinematics onto humanoid robots—ORION focuses entirely on the objects themselves. It extracts a manipulation plan as a sequence of what the researchers call Open-World Object Graphs, hierarchical graph structures that model the states of task-relevant objects and their relationships at key moments in the video.

Each Open-World Object Graph has a two-level architecture. At the high level, object nodes identify and localize relevant objects using outputs from vision foundation models, while a specialized grasp node encodes interaction cues such as contact points and grip status—whether a hand is open or closed. At the low level, point nodes correspond to object keypoints, and their features detail how those keypoints move through 3D space over the course of the video. Edges between nodes carry binary attributes indicating whether two objects, or an object and a hand, are in contact. This contact-relational structure turns out to be crucial: at execution time, the robot can compare the contact relations it currently observes against the plan and retrieve the matching keyframe, allowing it to determine which object to manipulate next.

The pipeline begins with video annotation. ORION samples frames from the demonstration and passes them to a vision-language model, GPT-4o, which returns textual descriptions of the task-relevant objects. These descriptions feed into an open-world segmentation model, Grounded-SAM, which annotates the objects with masks. Because running such models on every frame is computationally expensive—roughly 0.4 seconds per frame on an NVIDIA A5000 GPU—the system exploits object permanence: it segments only the first frame and then propagates those masks through the rest of the video using a video object segmentation model called Cutie, reducing processing to about 0.024 seconds per frame. For RGB-only videos, ORION tracks objects’ full six-degree-of-freedom trajectories with FoundationPose, modified to search across pose hypotheses within a plausible range of depths estimated by monocular depth estimation, and generates approximate object meshes with InstantMesh.

Keyframe discovery is where the method shows particular elegance. Changes in contact relations during manipulation—transitioning from free-space motion to grasping an object, for instance—are typically accompanied by sudden changes in object velocity. ORION samples keypoints within the object segmentation of the first frame and tracks them across the video using CoTracker, a track-any-point model with explicit occlusion modeling that handles the frequent occlusions inherent in manipulation scenes. It then applies an unsupervised changepoint detection algorithm to the velocity statistics of these trajectories to identify the keyframes at which contact relations change. Notably, the team found that for RGB-only videos, even a minimal plan with just two keyframes—the first and last video frames—was sufficient to describe the object motions.

Once keyframes are identified, ORION assembles a graph for each one and, at execution time, constructs a policy in three repeating steps: match the current observation to a keyframe in the plan, predict how the target object should move, and optimize robot actions accordingly. The prediction step involves a clever “warping” operation. Because the robot’s workspace will almost never match the video’s spatial layout, ORION normalizes the object-centric trajectory from the video—stripping away its absolute start and end locations—leaving only the directional and curvature patterns. It then rescales this normalized trajectory to fit the new start and goal positions in the robot’s environment, determined by registering the video’s object point clouds to the observed ones. For RGB-only pipelines, the system additionally warps rotations: it aligns the demonstration’s reference-to-target object direction with the test-time direction, normalizes the rotation sequence, and interpolates between boundary rotations using spherical linear interpolation to guarantee the trajectory begins and ends exactly where it should while preserving the original rotational “shape.” The resulting trajectories are converted into SE(3) action sequences for the robot end-effector through an optimization procedure, followed by inverse kinematics and joint impedance control running at 500 hertz on a 7-DoF Franka Emika Panda arm.

The experimental design was deliberately demanding. Demonstration videos were recorded with an iPad mounted on a stand in ordinary settings—an office, a kitchen, even outdoors—while the robot operated in its own workstation observed through an Intel RealSense D435 camera, with visual backgrounds, camera angles, and object arrangements all mismatched from the demonstrations. Across seven short-horizon RGB-D tasks, including placing a mug on a coaster, assembling a toy boat, and inserting succulents into a llama-shaped vase, ORION achieved a 66.7 percent average success rate over 15 real-world trials each. Comparisons against baselines underscored the importance of the object-centric design: a hand-motion imitation baseline performed poorly because copying human hand trajectories fails to generalize when the initial object layout differs, while a dense-correspondence baseline using optical flow misidentified keyframes during smooth transitions, producing fundamentally wrong action plans. An ablation study using three videos of the same mug-on-coaster task filmed in radically different settings—a kitchen, an office, and outdoors—found no statistically significant difference in the resulting policies’ performance.

Perhaps most strikingly, ORION handled demonstration videos containing no depth information whatsoever, achieving an 85.3 percent average success rate on five RGB-only tasks. These included a pouring task learned from an in-the-wild YouTube video and tasks learned from videos generated entirely by Google DeepMind’s Veo 2 text-to-video model—suggesting that robots might one day learn directly from synthetic or internet-sourced footage. The RGB-only pipeline even performed slightly better than the RGB-D version on one shared task, which the researchers attribute to the strong geometric priors embedded in the pose estimation and mesh generation models compensating for the lack of depth. The system also scaled gracefully to long-horizon, multi-step tasks such as preparing breakfast, which requires placing a mug on a coaster and arranging a food box and can on a plate, with no measurable performance drop compared to their single-step counterparts.

The work does carry limitations. ORION defines task goals through contact relations, sidestepping ambiguities in spatial language but leaving it unable to handle intentions like “place the item next to the cup” or collision avoidance. It assumes a stationary camera, whereas most everyday videos involve camera motion, and it does not model contact dynamics or force regulation. Still, the researchers argue that as vision foundation models continue to improve, stronger perceptual back-ends can be swapped into the framework with minimal modification. The broader implication is considerable: the readily available ocean of internet video—encompassing a vast distribution of human activities—may finally be within reach as a training ground for robot manipulation skills, with a single clip and a consumer phone camera standing in for hours of teleoperation and painstaking engineering.