RGB-D weed detection enables precise automated weeding in peony fields

In the sprawling medicinal herb plantations of Bozhou, in China’s Anhui Province, a quiet technological revolution is taking root. Researchers have unveiled a new artificial intelligence system that allows weeding robots to distinguish crops from weeds with unprecedented accuracy in one of agriculture’s most visually chaotic environments—dense, mature fields of Paeonia lactiflora, a prized traditional Chinese medicinal herb. The innovation, described in a study published in the journal Artificial Intelligence in Agriculture, combines ordinary color cameras with depth-sensing hardware in a way that could reshape how autonomous machines perceive and manage vegetation in unstructured farmland.

The problem the researchers set out to solve is deceptively simple to state but notoriously difficult to solve in practice. Paeonia lactiflora, a perennial clump-forming herb used widely in traditional Chinese medicine, grows into a dense, tangled canopy during the growing season. Surrounding it, weeds cluster thickly, occluding one another and the crop, while shifting sunlight throughout the day plays havoc with the color and texture cues that most computer vision systems depend on. Manual weeding remains the dominant practice in most Chinese medicinal herb fields, but it is labor-intensive, inefficient, and increasingly threatened by labor shortages. Chemical herbicides, the standard weapon against weeds in crops like wheat and maize, are largely off the table for medicinal herbs, which face stringent limits on pesticide residues because of their pharmaceutical end use. Physical methods such as laser and flame weeding consume enormous energy, and mechanical weeding demands the kind of uniform, precise row spacing found only in structured plantings like lettuce—not in the unruly reality of a perennial herb field.

That leaves weeding robots—integrated platforms combining mobility, navigation, visual perception, and actuation—as the most promising path forward. But the heart of any such robot, its visual perception module, has long been the weakest link. Most existing weed-detection systems rely on single-modality RGB cameras and deep learning models trained primarily on early-stage crops in sparse, simple backgrounds. When confronted with the dense, occluded, variable-lighting conditions of a mature medicinal herb field, their performance degrades sharply. As the researchers note, relying on visible texture and color alone simply does not provide enough discriminative information in such scenes.

The team’s key insight was that the answer was already hanging in front of the robot’s face. Modern weeding robots routinely carry depth cameras—low-cost 3D imaging devices that have become standard equipment for navigation and obstacle avoidance. Yet, as the researchers observed in their analysis of the literature, depth data in these systems has been used almost exclusively for computing three-dimensional coordinates after detection, not for the detection task itself. Meanwhile, in the field, measurable height differences and geometric relationships consistently exist between the herb plants and the weeds, and among the weeds themselves. That geometric signal, encoded in depth images, was being discarded exactly where it could be most useful.

Enter FW-DETR, the researchers’ new single-stage object detection model built on a hybrid convolutional neural network and Transformer architecture. The system’s core philosophy is what the team calls a modality-graded feature processing paradigm: rather than treating RGB and depth images as equal partners, it treats depth as an auxiliary modality whose geometric priors guide and refine the rich visual information coming from the color camera. This matters because the two modalities are fundamentally different in character. RGB images carry complex, high-frequency color and texture detail; depth images convey low-frequency, simple geometric structure and distance constraints. Most existing RGB-D fusion architectures, the researchers argue, fail to account for this disparity, conflating modality-specific components during fusion and blurring high-frequency features in the process.

To fix this, the team engineered two novel modules. The first, a cross-modal hierarchical feature interaction module, exploits the functional complementarity of the two sensors while suppressing redundant information. Within it, a cross-modal global context modeling component uses Transformer-based attention to enable global information exchange and alignment between the heterogeneous modalities, while a spatially adaptive local feature modeling component leverages convolutional operations to aggregate local details and boundary cues, calibrating features adaptively and suppressing noise. The second innovation, a cross-modal cascade attention module, performs grouped cross-modal interactions between high-level semantic features and depth features, explicitly embedding spatial geometric structure into the semantic representations. The result, according to the researchers, is reduced feature ambiguity and improved localization performance in complex scenes—a critical capability when a robot’s next action depends on knowing precisely where a weed begins and the crop ends.

The architecture was put to the test against a new benchmark dataset the team constructed, named PLPW2024. Assembled from images captured at a medicinal herb plantation in Bozhou using the first-generation field weeding robot developed by the group, the dataset comprises 1,512 paired RGB and depth images containing 47,633 annotated instances of crops and weeds. Images were acquired with an Intel RealSense D435i active-stereo camera, a device offering a valid depth range of 0.3 to 3 meters, mounted on a robot whose four main components—a vision system, mobile chassis, control system, and actuation module—were purpose-built for image collection and accuracy validation in real fields.

The choice of location was deliberate. Bozhou is one of China’s major medicinal herb production regions, where Paeonia lactiflora is widely cultivated, providing exactly the kind of real-world, unstructured scenarios that laboratory datasets so often lack. By building a standardized, paired RGB-D dataset from these conditions, the researchers have given the broader research community a unified benchmark for evaluating weed detection methods under genuinely challenging field conditions—dense canopies, mutual occlusions, and pronounced illumination variation among them.

The workflow that connects perception to action is as important as the detection model itself. After the vision system produces bounding box coordinates and class predictions in the two-dimensional image plane, those predictions are converted into executable three-dimensional spatial operation points that the robot can act upon, enabling targeted weed control without harming the crop. This closing of the loop—from multimodal perception to precise physical intervention—is what transforms an impressive detection algorithm into a functioning agricultural machine.

In extensive experiments, the proposed FW-DETR model improved detection accuracy across diverse scenarios and exhibited strong robustness and generalization, the researchers report. Crucially, the system was not merely validated in the abstract: field trials with the weeding robot demonstrated its effectiveness under real-world conditions, with the depth-guided architecture maintaining its performance where single-modality approaches would falter. The findings suggest that the humble depth camera, long relegated to a supporting role in agricultural robotics, may be one of the most underutilized assets in the fight for sustainable weed management.

The implications extend well beyond peony fields. Demand for Chinese medicinal herbs continues to climb, wild populations of Paeonia lactiflora can no longer meet market needs, and cultivated acreage keeps expanding—along with the weed pressures that threaten yield and quality. Weeds are not merely competitors for light and nutrients; they also serve as intermediate hosts for numerous pests and diseases, exacerbating infestations across entire fields. A perception system that can reliably separate crop from weed amid occlusion and visual clutter offers a template for selective weeding in other dense-canopy, unstructured crops where chemical, physical, and mechanical control have all fallen short.

More broadly, the study contributes to a growing recognition in the computer vision community that multimodal learning succeeds or fails on the quality of its fusion strategy. Data-layer and decision-layer fusion approaches, prior studies have shown, fail to capture the cross-modal correlations inherent in heterogeneous data. Feature-layer fusion—with independent branches learning modality-specific representations that are then interactively combined—has emerged as the dominant paradigm, and the Bozhou team’s graded, geometry-guided approach pushes that paradigm further by explicitly respecting the informational asymmetry between rich RGB detail and sparse depth structure. As attention mechanisms and Transformers continue to prove their worth at aligning representations across heterogeneous modalities, architectures like FW-DETR hint at how agricultural robots might eventually perceive fields the way experienced farmers do: not just by what plants look like, but by how they occupy space.

For now, the researchers’ weeding robot continues its work in the herb fields of Anhui, its stereo eyes gathering both light and distance, its neural network weighing color against geometry. What was once a hardware afterthought has become the guiding intelligence of the machine—a shift that could bring autonomous, chemical-free weed control closer to practical reality for the world’s medicinal herb growers, and perhaps for unstructured agriculture far beyond them.