Deep learning enables markerless tracking of cyborg insects outdoors

In a development that could reshape how scientists study and steer living robots, researchers in Indonesia and Japan have unveiled a camera-based artificial intelligence system that tracks cyborg insects in real time without attaching a single physical marker to their bodies. The work, led by Habib Ja’far Nuur and Mochammad Ariyanto of Diponegoro University in collaboration with Keisuke Morishima of The University of Osaka, addresses one of the most persistent engineering headaches in the emerging field of biohybrid robotics: how to precisely measure where a small insect is, and which way it is heading, without weighing it down or interfering with its natural movement.

Cyborg insects, typically cockroaches or beetles fitted with tiny electronic backpacks that deliver electrical stimulation to their antennae or sensory organs, have attracted intense attention in recent years as potential search-and-rescue scouts, environmental monitors, and infrastructure inspection agents. The appeal is straightforward. Millions of years of evolution have produced locomotion systems that no human engineer can match at small scales: insects can climb rubble, squeeze through crevices, recover from falls, and sustain operation on meager energetic resources. By interfacing electrodes with the insect’s nervous system, researchers can, in principle, command the animal to turn left, turn right, accelerate, or pause. But commanding an insect is only half the problem. To close the control loop, the system must know continuously where the insect actually is and where it is pointed, a measurement task that has historically required either bulky laboratory motion-capture systems or reflective markers glued to the animal’s body.

The traditional marker-based approach carries hidden costs. A small frame of reflective markers mounted on the insect’s back adds payload to an animal that may itself weigh only a few grams, and it changes the aerodynamics, center of mass, and natural gait of the creature. Worse, the researchers found, physical marker frames can snag on walls and obstacles, entangling the insect as it navigates cluttered terrain. In experiments comparing the new markerless method with earlier marker-based setups, no obstacle entanglement was observed with the markerless approach, while the marker-based configurations risked exactly the kind of snagging that disrupts both the experiment and the insect’s welfare. For a field whose long-term vision is deploying these animals in disaster zones full of debris, eliminating protruding hardware is not a cosmetic improvement but a functional necessity.

The new framework, described in the International Journal of Intelligent Robotics and Applications, relies on a single inexpensive overhead web camera pointed at the experimental arena. The core challenge is one of scale: a cockroach viewed from directly above occupies only about 0.13 percent of the image frame, making it one of the smallest conceivable detection targets. To handle this, the team built their system around a lightweight YOLO detector, a family of neural network architectures famous for performing object detection in a single forward pass, which makes it fast enough for real-time video processing even on modest hardware. The detector locates each cyborg insect within every frame of the camera feed, even in arenas cluttered with obstacles that would confound simpler background-subtraction methods.

Detection alone, however, does not reveal the insect’s heading, the orientation of its body axis, which is essential for closed-loop steering. The researchers therefore augmented the YOLO detector with three different pose-estimation strategies and compared them head to head. The first strategy extracted keypoints directly from the YOLO network, effectively asking the same architecture that finds the insect to also identify anatomically meaningful points on its body. The second employed a dedicated convolutional neural network that predicts heatmaps, probability distributions over the image in which each anatomical landmark lights up as a bright blob whose weighted centroid yields the keypoint position; this approach descends from human pose-estimation techniques that regress part heatmaps rather than coordinates directly. The third benchmark was DeepLabCut, the widely adopted open-source pose-estimation toolkit originally developed for neuroscience, which has become a de facto standard for markerless tracking of animals in behavioral laboratories.

The comparative evaluation produced a clear winner. When the team ran single and pairs of cyborg cockroaches through both obstacle-free and obstacle-filled arenas, the YOLO-based keypoint estimator achieved the most favorable balance of accuracy, robustness, and speed, delivering inference in just 32 milliseconds per frame. The CNN heatmap model and DeepLabCut, while capable, were slower and in some conditions less reliable at maintaining continuous measurement of a target so small and so erratically moving. Critically, in the two-insect experiments, the system maintained stable identity preservation, correctly distinguishing which tracked animal was which over time, a nontrivial feat when two nearly identical cockroaches cross paths or pass close to one another in a confined arena.

Real-time performance mattered for a second reason: the tracking system had to be integrated with the wireless stimulation backpack carried by each insect. The complete pipeline, camera capture, neural inference, trajectory computation, and command transmission, ran at 23 frames per second, fast enough to synchronize the insect’s observed kinematics with the electrical commands being issued. The researchers report a clear correlation between the stimulation commands sent to the antennae and the measured changes in the insect’s position and heading, meaning that for the first time in their experimental setup, feedback control of cyborg cockroach locomotion could proceed continuously without any external commercial motion-capture equipment and without any physical attachment beyond the functional backpack itself.

The technical significance of the work lies partly in its demonstration that low-cost, commodity hardware can substitute for laboratory-grade tracking infrastructure. Traditional approaches to measuring insect locomotion include tethered treadmills and spherical trackballs, servo-driven spheres on which a fixed insect walks while sensors measure ball rotation, as well as high-end optical motion capture requiring calibrated multi-camera arrays. Each of these constrains the experiment: trackballs prevent free navigation through clutter, and motion-capture systems demand reflective markers and precisely controlled lighting. A single web camera paired with an efficient neural network removes those constraints, opening the door to locomotion experiments in environments that genuinely resemble the unstructured terrain, rubble piles, dense vegetation, collapsed structures, where cyborg insects would one day be deployed.

There are also implications for animal welfare and experimental validity. Payload constraints on small insects are severe; every additional gram of marker hardware alters the animal’s energy expenditure and gait, potentially confounding the very locomotion data researchers are trying to collect. By measuring the insect purely through vision, the system supports natural locomotion of backpack-equipped animals, and the elimination of snag-prone marker frames reduces both experimental failures and physical harm to the insects. The team notes that the proposed system is well suited to real-time locomotion tracking experiments in controlled laboratory arenas, a modest but important scope claim: this is a laboratory validation, with overhead cameras and defined arenas, rather than a field-ready deployment.

The broader context is a rapidly maturing cyborg-insect ecosystem. Recent years have seen swarm navigation of cyborg insects through unknown obstructed terrain, automated robotic assembly lines for mounting electrodes on live insects, onboard solar cells that keep backpack batteries charged without restricting mobility, and 3D-printed ergonomic harnesses designed to minimize invasiveness. What has lagged behind is the sensing layer, the ability to know, cheaply and continuously, what the insects are doing. Machine-learning-based behavioral tracking has already reached human-level accuracy in other domains, outperforming some commercial solutions in neuroscience laboratories, and this study extends that trajectory to one of the most demanding tracking regimes imaginable: millimeter-scale targets, from above, in real time, among obstacles.

The research team, which also included Hafiz Akbar Simanjorang, M. Munadi, and Rifky Ismail of Diponegoro University and Ade Kurniawan of Institut Teknologi Sains Bandung, was funded by Indonesia’s Ministry of Higher Education, Science, and Technology. Their results suggest that the pieces of a fully autonomous cyborg-insect control system, wireless stimulation, markerless perception, and closed-loop feedback, can now be assembled from components that cost a small fraction of traditional motion-capture installations. As the field moves toward multi-insect swarms searching for earthquake survivors or mapping hazardous interiors, the ability to watch many small animals at once, with ordinary cameras and fast neural networks, may prove as decisive as the electrodes that steer them.

For now, the system remains confined to laboratory arenas with a single overhead viewpoint, and the authors are careful to frame it as a foundation rather than a finished product. Raw data are available from the corresponding author upon request, and the team emphasizes that the framework’s low latency, markerless operation, and demonstrated identity preservation across multiple animals collectively remove several long-standing bottlenecks in cyborg-insect experimentation. If biohybrid robots are to leave the laboratory, they will need systems like this one: unobtrusive, fast, and indifferent to clutter, watching from above while the insects do what insects do best.