MIT System Identifies and Corrects Mislabeled Object Descriptions in Robot Memory

Researchers at the Massachusetts Institute of Technology have developed a robotic memory system called UQ-DAAAM that identifies and corrects inaccurate object descriptions stored in a robot’s memory, improving the machine’s ability to interact with its physical environment. The method, which stands for Uncertainty Quantification for Describe Anything, Anywhere, at Any Moment, compares multiple descriptions of the same object observed from different viewpoints to quantify the robot’s uncertainty about what it saw. When evaluated against baseline approaches, the system boosted the robot’s confidence and improved the accuracy of its answers to questions regarding observed objects, their physical locations, and the timestamps of when they were encountered.

The system addresses a critical reliability challenge in autonomous assistance, where inaccurate stored memories could lead to severe errors during complex, multistep tasks. In an emergency scenario, a human might direct a robotic assistant to retrieve a wall-mounted defibrillator located down a hallway, but if the robot has incorrectly labeled the defibrillator as a fire extinguisher in memory, it may not know where to find the life-saving device. Luca Carlone, an associate professor in MIT’s Department of Aeronautics and Astronautics, principal investigator in the Laboratory for Information and Decision Systems, director of the MIT SPARK Laboratory, and senior author of the paper, noted that unlike ChatGPT, physical artificial intelligence cannot rely on a simple disclaimer warning that the system might be wrong, emphasizing that allowing a robot to know what it does not know is essential for safe societal deployment.

The research, co-authored by MIT graduate students Harry Zhang and Nicolas Gorlo, is scheduled for presentation at the Conference on Neural Information Processing Systems. The method builds directly on the researchers’ prior system, DAAAM, which uses an onboard vision-language model to generate rich descriptions of objects captured by camera sensors. In that previous architecture, these descriptions are stored as detailed captions within a three-dimensional map-based representation of the environment, enabling the robot to recall spatial and visual information later to answer questions.

However, mobile robots frequently encounter imperfect conditions that degrade visual observations, such as seeing objects only partially, viewing them from long distances, or capturing blurry or poorly lit images. Under these conditions, the baseline DAAAM framework could generate incorrect descriptions, such as mistaking a gray chimney for a squirrel perched on a rooftop. As co-lead author Harry Zhang explained, without a mechanism to correct itself, that incorrect memory would persist as the robot navigates through its environment, negatively affecting downstream applications.

UQ-DAAAM integrates uncertainty quantification into the DAAAM architecture to allow a mobile robot to evaluate its confidence and correct faulty captions in real time. While traveling through an area, the robot continuously collects and stores imagery showing the same objects from different angles. The system first evaluates the vision-language model’s confidence level in the words chosen to describe a potentially mislabeled item, and then measures consistency by comparing that description against captions generated from other viewpoints. Carlone noted that the framework fuses both sources of uncertainty into a single number representing the vision-language model’s overall comprehension of the object.

When this fused uncertainty metric exceeds a defined threshold, UQ-DAAAM initiates a refinement process using additional imagery of the item. Rather than reprocessing all available visual data, the system relies on a mathematically proven heuristic rule that selects the specific vantage points most capable of reducing uncertainty, such as selecting a frame taken moments later showing smoke emanating from a chimney. The researchers mathematically proved that this heuristic identifies the most informative frames without rerunning the vision-language model, dramatically lowering computational requirements. Carlone emphasized that a central accomplishment was combining vision-language models with uncertainty quantification into a process that executes in a fraction of a second on a moving platform.

After selecting additional captions that reduce ambiguity, UQ-DAAAM uses a large language model to fuse them into a single, cohesive description, giving greater weight to newer captions and those showing less ambiguity. In experimental evaluations, this refinement procedure resolved ambiguities for approximately 92 percent of the objects in the robot’s environment while requiring only two additional captions per object. The approach increased object description accuracy by 5 percent compared to the state of the art and provided superior precision over baseline methods when responding to questions about what objects were seen, where they were located, and when they were last observed.

Co-lead author Nicolas Gorlo highlighted that executing this uncertainty quantification procedure across the robot’s entire map improves its comprehensive understanding of observed surroundings. Moving forward, the research team aims to apply UQ-DAAAM to other platforms, including mobile robotic manipulators, and extend the uncertainty quantification techniques to other robotic domains. The researchers are also exploring methods for robots to identify and revisit high-uncertainty areas to improve environmental maps, as well as applying uncertainty quantification to systems predicting future environmental states. The work was supported in part by the U.S. Army Research Laboratory and the U.S. Office of Naval Research.

Subject of Research: Uncertainty quantification system for correcting mislabeled object descriptions in robot memory

Article Title: Clearing up a robot’s fuzzy memory

Article References: Clearing up a robot’s fuzzy memory. Original publication (Bibliographic details unavailable.)

Image Credits: AI Generated

DOI: Not provided

Keywords: Robotics, Artificial Intelligence, Robot Memory, Uncertainty Quantification, Vision-Language Models, MIT, Autonomous Systems, Neural Information Processing Systems, Clearing, robot, fuzzy, memory