Robots Learn to Track Moving Objects by Watching Human Contact

Robots navigating a busy living room face a deceptively hard problem: the world refuses to stay still. A companion robot’s camera sees people walking past, chairs pulled across the floor, and books or cups carried from one table to another. Every one of those moving things can corrupt the map the robot is quietly building of its surroundings. A new study published in Autonomous Robots proposes a way for a robot to lean on a distinctly human clue—the fact that people tend to hold certain objects—to keep its localization steady even when the furniture is on the move.

The research, led by Jilin Zhang of the University of Jinan with colleagues from Shandong Normal University, the University of Jinan and Lunan Technician College, targets a weak spot in modern visual Simultaneous Localization and Mapping, or SLAM. Classical SLAM systems assume the world they observe is static. When people walk through the frame, feature points attached to them move for reasons that have nothing to do with camera motion, and the system’s estimate of its own trajectory drifts. Recent dynamic SLAM methods, such as Dyna-SLAM and SaD-SLAM, attack this by detecting humans and other obviously dynamic objects and discarding their pixels. But the authors point out a stubborn category of uncertainty: movable objects like books and cups. Most of the time a book sits still on a desk, so treating it as static background is reasonable—until someone picks it up and carries it across the room. A SLAM system that blindly trusts those features inherits the object’s motion as phantom camera motion.

The team’s answer is a dynamic SLAM system built around what they call human contact experience. Rather than hard-coding which objects are dynamic and which are static, the system learns from observation how often humans come into contact with particular categories of objects. Objects that are frequently held, such as cups and books, receive a prior state that makes the system suspicious of their apparent motion; objects that people rarely touch keep their static status. When a human and a movable object are in contact, the object’s features are treated as unreliable and excluded from pose estimation. In effect, the robot accumulates a form of common-sense knowledge about indoor life and uses it to decide which of the things it sees can be trusted as reference points.

Technically, the system weaves together three modules. The first is an adaptive frame selection strategy driven by semantic segmentation results. Instead of feeding every RGB-D frame into the computationally expensive segmentation pipeline, the system adaptively chooses which frames to process, reducing computational resource consumption while improving the quality of the prior information available to later stages. This matters for real robots, which must localize in real time on hardware far less powerful than a laboratory workstation. The second module refines the geometric analysis: by combining optical flow with epipolar constraints, the system determines the motion states of both humans and movable objects. Optical flow captures how pixels shift between consecutive frames, while the epipolar constraint describes how a static point in a rigid scene should move given the camera’s own motion. A point that violates the epipolar geometry is almost certainly moving independently of the camera—an elegant, geometry-based way to flag dynamic content without relying on semantics alone.

The third and conceptually novel piece is the contact experience module itself. Drawing on information from multiple consecutive frames, the module records the contact frequency between humans and movable objects and uses that history to update the prior state of objects in the indoor environment. An object seen repeatedly in human hands shifts its prior toward dynamic; an object that has never been touched retains a static prior. Because this knowledge is updated continuously, the system adapts to a particular environment over time rather than relying on a fixed, hand-tuned list of dynamic classes. The authors describe this as using human contact experience with movable objects to predict their true states—a statistical prior grounded in the everyday physics of how people interact with their belongings.

Everything rests on accurate camera trajectories, so the researchers evaluated their system on the TUM RGB-D benchmark, the standard dataset for testing RGB-D SLAM under dynamic conditions. The benchmark includes sequences in which people walk, sit and interact with objects while the camera moves through the scene—precisely the conditions that break static-world assumptions. The proposed method was compared against ORB-SLAM2, the widely used open-source baseline for monocular, stereo and RGB-D cameras, and against two representative dynamic-environment systems, Dyna-SLAM and SaD-SLAM.

The reported results show the new system operating stably in dynamic environments and, crucially, handling state changes of indoor movable objects more effectively than its predecessors. Where Dyna-SLAM and SaD-SLAM can mask out walking people, they have no principled mechanism for the cup that was static in frame one and moving in frame ten. By combining adaptive frame selection, flow-and-epipolar geometry and contact-frequency priors, the new method covers both ends of the problem: it ignores pixels belonging to independently moving entities and reclassifies movable objects the moment their behavior changes. The authors note that the adaptive frame selection also keeps the computational cost in check, which matters for indoor companion robots that must run continuously.

The implications reach beyond a cleaner trajectory estimate. Indoor companion robots are expected to interact naturally with humans, and that requires knowing not just where the robot is, but what in the room is trustworthy as a landmark. A robot that understands that a person carrying a mug makes the mug’s features unreliable, but that the mug becomes a valid landmark again once set down, gains a more realistic model of its environment. The contact experience framework is also a small but suggestive step toward robots that learn everyday physics from observation—the kind of implicit knowledge humans use constantly without noticing.

The work was supported in part by the National Natural Science Foundation of China, the Taishan Scholar Foundation of Shandong Province and the Outstanding Youth Foundation of Shandong Province. As robots move from factory floors into homes, offices and hospitals, the ability to localize reliably amid human activity will stop being a research curiosity and become a baseline requirement. This study suggests that some of the best clues for separating a stable world from a shifting one may come from simply paying attention to what people are holding.

Subject of Research: A dynamic RGB-D SLAM method that uses human contact experience to determine the motion states of movable objects for robot localization in dynamic indoor environments.

Article Title: A RGB-D SLAM method based on contact experience in dynamic environment

Article References: Zhang, J., Huang, K., Geng, H., Song, C., & Zhang, M. (2026). A RGB-D SLAM method based on contact experience in dynamic environment. Autonomous Robots, 50(4), Article 40. https://doi.org/10.1007/s10514-026-10268-1

Image Credits: AI Generated

DOI: 10.1007/s10514-026-10268-1

Keywords: SLAM, RGB-D camera, dynamic environments, robotics, semantic segmentation, optical flow, epipolar constraints, contact experience, movable objects, ORB-SLAM2, Dyna-SLAM, TUM RGB-D dataset