Tiny AI brain lets a $10 microcontroller remember hidden objects and grab them

Robots that can reason about objects they cannot currently see have long been the preserve of data-center-scale artificial intelligence, with vision-language-action models built on tens of billions of parameters and racks of GPUs. A new open-access study in Discover Informatics now demonstrates a strikingly different approach: a robotic manipulation controller small enough to run entirely on an ESP32 microcontroller, yet capable of tracking objects through visual occlusion and completing physical pick-and-place tasks with an observed success rate of 85 percent. The system, called Tiny-STM, packs a recurrent neural network, an external coordinate memory, and analytical inverse kinematics into just 68.4 kilobytes of quantized model weights.

The core insight behind Tiny-STM is that object permanence, the cognitive ability to understand that things continue to exist when hidden from view, does not require a learned memory mechanism of the kind used in large transformer-based systems. Instead, the researchers, Swarnajit Bhattacharya of National Yang Ming Chiao Tung University in Taiwan and Ian Chao of the University of Washington, paired a standard four-gate Long Short-Term Memory network with a deterministic four-slot coordinate bank. Each slot stores the most recently observed three-dimensional position of one task object. When a cube disappears behind a stack of other blocks, its stored coordinate simply holds steady until the camera sees it again, a scheme the authors describe as a zero-order-hold update rule.

The architecture itself is deliberately conventional. The LSTM takes a seven-dimensional input vector combining the vector from the gripper tip to the target object’s stored position, the current tip position, and a one-dimensional infrared range reading from a Sharp GP2Y0A41SK0F sensor mounted on the gripper. A hidden state of 128 dimensions feeds a linear output layer that predicts an incremental three-dimensional displacement of the end effector. With 70,019 parameters in total, the network is tiny by modern standards, and after INT8 quantization through the TensorFlow Lite Micro pipeline it occupies just 68.4 KB, a small fraction of the ESP32’s 520 KB of static RAM.

Training relied on remarkably little data. The authors manually guided a Waveshare 5-DOF RoArm-M3 robotic arm through exactly 30 pick-and-place demonstrations, recording joint angles, cube centroids from a side-mounted camera, and infrared range measurements at each control step. Backpropagation through time with mean-squared-error loss and the Adam optimizer at a learning rate of 0.001 produced the deployed checkpoint. This data efficiency stands in sharp contrast to imitation-learning platforms such as ALOHA, which achieve 80 to 90 percent success on fine-grained bimanual tasks but require roughly ten minutes of teleoperated demonstrations and substantially more powerful compute.

The division of labor between hardware components is a key part of the design. A host PC or single-board computer captures camera frames and performs deterministic color-thresholding segmentation, converting detected pixel centroids into three-dimensional workspace coordinates through a calibrated pinhole camera model. The camera was calibrated with 25 checkerboard images using Zhang’s method, yielding a mean reprojection error of 0.3 pixels, while camera-to-base extrinsic calibration over 12 known points achieved a root-mean-square alignment error of 1.18 millimeters. The ESP32 then handles everything on the control side: the recurrent forward pass, analytical inverse kinematics, and PWM servo command generation, all in a measured 2.80 milliseconds.

The inverse kinematics routine reduces the five-degree-of-freedom arm problem to a planar three-degree-of-freedom calculation, exploiting the fact that the task uses a top-down grasp with a vertically oriented gripper. Base rotation comes from an arctangent of the target coordinates, while the shoulder and elbow angles follow from two-link geometry via the law of cosines. The authors handle three degenerate cases explicitly: base singularity when the target lies on the rotation axis, arm singularity near full extension where the Jacobian becomes ill-conditioned, and unreachable targets outside the geometric annulus defined by the link lengths, which are simply rejected so the arm retains its previous pose.

Physical evaluation consisted of 20 trials in which the arm sorted four labeled cubes from randomized positions into an ordered arrangement, including a vertical stacking sequence in which the fourth block becomes completely hidden behind the first three before its final grasp. The single deployed INT8 checkpoint completed 17 of 20 trials, an observed success rate of 85.0 percent with a two-sided 95 percent exact binomial confidence interval of 62.1 to 96.8 percent. The authors are careful to note that this interval describes uncertainty in the 20 observed outcomes of one checkpoint, not variability across independently trained models, since the original training seed and stopping rule were not retained.

The three failures are instructive. Mechanical backlash in the servo gearboxes introduced joint-level positional uncertainty of roughly 1.5 to 2.5 millimeters, which can accumulate across joints. Fixed color thresholds proved vulnerable to strong lighting changes and shadows that shifted pixel values out of range, degrading the segmentation masks. And when an occluded object was physically moved while hidden, the stored coordinate became stale, causing the controller to reach toward a position where the object no longer was; the gripper tolerance is approximately 18 millimeters. The authors formalize this limitation in a corollary showing that if an occluded object is displaced, the memory error grows at least in proportion to the displacement.

On latency, the measured ESP32 computation of 2.80 milliseconds combines with the nominal 33.3-millisecond camera period at 30 frames per second, profiled segmentation and Wi-Fi communication times, and a datasheet-based actuator response allowance to yield an aggregated end-to-end latency estimate of 92.6 milliseconds, equivalent to roughly 10.8 hertz. The authors emphasize that this figure is an estimate assembled from components rather than a single timed end-to-end measurement. Peak SRAM usage on the microcontroller was 94.8 KB, or 18.23 percent of capacity, leaving substantial headroom. Offline comparisons suggest the FP32 model would occupy 273.5 KB with an estimated 11.20 milliseconds of inference, underscoring the value of INT8 quantization for embedded deployment.

The study is notable as much for its honesty about boundaries as for its results. The authors explicitly state that no ablation was performed to separate the contribution of the coordinate memory from that of the recurrent state, that no numerical comparison with RT-2, OpenVLA, TinyVLA, or Diffusion Policy is possible given different tasks and hardware, and that the color-thresholding front end was never validated against ground-truth masks. Future work, they write, should include matched ablation studies with multiple training seeds, documented reproducibility protocols, perception validation, tests of behavior when occluded objects are displaced, and scaling experiments beyond four tracked objects. Even so, Tiny-STM offers a compelling demonstration that a form of object permanence can live in 68 kilobytes, bringing occlusion-aware manipulation within reach of the cheapest embedded hardware.

Subject of Research: Lightweight recurrent neural network controllers with external coordinate memory for occlusion-aware robotic manipulation on microcontrollers

Article Title: Lightweight LSTM-based memory architecture for occlusion-aware robotic manipulation on microcontrollers

Article References: Bhattacharya, S., & Chao, I. (2026). Lightweight LSTM-based memory architecture for occlusion-aware robotic manipulation on microcontrollers. Discover Informatics, 1(1), Article 23. https://doi.org/10.1007/s44564-026-00023-0

Image Credits: AI Generated

DOI: 10.1007/s44564-026-00023-0

Keywords: TinyML, LSTM, robotic manipulation, object permanence, occlusion, ESP32, microcontroller, imitation learning, inverse kinematics, INT8 quantization, embedded robotics, recurrent neural networks