A robot dog has learned to imagine parts of the world it cannot see—and that ability could help autonomous machines navigate the cluttered, confusing environments where conventional mapping systems often fail. In a new study, researchers at the University of Colorado Boulder describe SceneSense, a generative artificial-intelligence system that predicts unseen three-dimensional terrain from incomplete sensor data. The model was integrated with a quadruped robot and tested in real indoor environments, where it helped the machine plan routes through narrow hallways, around corners and across regions hidden from its lidar. The results suggest that robots may become more reliable explorers when they combine direct observations with carefully weighted predictions about what lies beyond an obstruction.
Robotic mapping traditionally depends on a simple rule: if a sensor has not measured a location, the robot treats it as unknown. That caution is essential for safety, but it can also paralyze an autonomous system. A lidar mounted high on a robot may fail to see the ground immediately beneath the machine, behind a wall, around a tight corner or beyond a glass barrier. In those cases, a planning algorithm may interpret missing information as impassable terrain, even when a human would recognize that a safe path probably exists. The problem becomes particularly serious in search-and-rescue missions, underground exploration and disaster response, where no detailed map exists in advance and stopping repeatedly to gather new measurements wastes precious time.
SceneSense addresses this problem by representing an environment as a three-dimensional occupancy map. The map divides space into small volume elements, or voxels, and assigns each one a probability of being occupied. Sensor measurements can mark voxels as containing an obstacle or as empty, while unmeasured regions remain uncertain. Instead of simply filling those gaps with a fixed geometric rule, SceneSense uses a diffusion model trained to generate plausible occupancy patterns from partial observations. Diffusion models are the same broad family of neural networks that have become famous for generating images, but here the output is not a picture. It is a probabilistic 3D description of surfaces, open space and obstacles that could reasonably exist beyond the robot’s field of view.
The model works through a process of controlled denoising. During training, the researchers begin with complete local occupancy maps collected in indoor buildings and progressively corrupt them with Gaussian noise. A U-net neural network learns to reverse that corruption, reconstructing the underlying geometry from increasingly noisy versions of the map. During deployment, SceneSense starts with random noise and iteratively removes it, producing a candidate occupancy map consistent with the observed surroundings. Crucially, an inpainting procedure repeatedly reintroduces the robot’s actual lidar measurements at every denoising step. This prevents the generative model from overwriting known occupied or free space. The system is therefore asked to imagine only what has not been observed, while measured geometry remains authoritative.
The researchers also changed how predictions are incorporated into the robot’s running map. Earlier versions used a “fire and forget” approach: each new prediction replaced the previous estimate, potentially causing the planner to swing abruptly toward a false path if one generated map was implausible. The new system uses probabilistic map merging based on occupancy probabilities and log-odds updates, a method also used by the established OctoMap mapping framework. Sensor observations are given greater confidence than model predictions, while multiple SceneSense outputs can accumulate over time. Because diffusion is inherently stochastic, the model can produce several different but plausible maps from the same partial input. Combining those alternatives allows unlikely voxel predictions to be filtered out while preserving structures that appear consistently.
The robot used in the experiments was a Boston Dynamics Spot equipped with an Ouster OS1-64 lidar and an inertial measurement unit. Lidar supplied point clouds for mapping and localization, while the LIO-SAM system fused lidar and inertial data to estimate the robot’s position. SceneSense was connected to an existing graph-based exploration planner rather than replacing the entire navigation stack. The planner generated candidate exploration points, or frontiers, and ranked them according to factors including distance, reachability and expected information gain. SceneSense could then predict occupancy either around the robot or at a selected frontier as far as 7 meters away. The model was trained on approximately 11,296 poses derived from about one hour of real-world occupancy data collected in indoor buildings, with test areas excluded from training.
In comparisons with sensor-only maps, the generative system produced occupancy maps that more closely resembled fully observed ground truth. The researchers report a 24.44 percent improvement in Fréchet Inception Distance around the robot and a 75.59 percent improvement at range, where the robot had less direct information. In a narrow L-shaped course requiring a 90-degree turn, frontier-enhanced SceneSense produced the best autonomous performance, with the lowest average exploration time and path length and lower variation between runs. Individual predictions took approximately half a second to generate, with a similar time needed to merge them into the map. By predicting terrain ahead while the robot was still moving, the system could have the next plan ready before the machine reached the area where new decisions were needed.
Several demonstrations show why this matters. At startup, the lidar could not see the ground directly under the robot, leaving a large artificial hole in the map. SceneSense inferred a traversable surface and allowed the robot to begin moving without teleoperation, extra downward-facing sensors or a special startup routine. In a narrow hallway, the lidar again missed ground measurements to one side, preventing the planner from finding a route forward. The predicted occupancy filled the gap and enabled continued exploration. The system also handled a more dangerous case near a glass railing. A simplistic hole-filling algorithm or unconstrained planner might have treated the space beyond the railing as open ground and directed the robot over an edge. SceneSense instead predicted untraversable geometry, preserving the boundary and blocking the unsafe route.
The results were not uniformly successful, however, and the failure reveals an important challenge for predictive robotics. When SceneSense filled distant unknown regions with predicted occupancy values, a conventional frontier planner could no longer distinguish between space observed by sensors and space merely imagined by the model. It sometimes concluded that an environment had been explored before the robot had actually reached all of it. Frontier-centric SceneSense therefore failed to complete some full-scene exploration trials, even though it improved local mapping and short-range navigation. The authors argue that future planners must reason jointly about observed and predicted voxels instead of treating generative predictions as ordinary sensor data. The distinction is essential: uncertainty can help a robot move, but erasing the boundary between knowledge and inference can make it stop searching too soon.
The study’s practical promise is also limited by the conditions under which SceneSense was trained and tested. The data came from structured indoor environments, so it remains unclear how well the system would generalize to forests, rubble, caves, outdoor terrain or rapidly changing scenes. Transparent and reflective surfaces may still generate incorrect predictions, and a highly confident error could distort the map despite the system’s preference for sensor measurements. In addition, the current implementation relies on an off-board GPU, an awkward requirement for robots constrained by battery capacity, weight and communications links. Even so, the work demonstrates a striking shift in robotic perception: rather than waiting for every surface to appear in a sensor scan, a machine can form a probabilistic geometric hypothesis, preserve uncertainty and use that hypothesis to act. The next generation of autonomous explorers may not need human intuition—but they may begin to approximate its most useful feature: knowing how to make a cautious guess.
