The problem the researchers set out to solve is deceptively simple to state but notoriously difficult to solve in practice. Paeonia lactiflora, a perennial clump-forming herb used widely in traditional Chinese medicine, grows into a dense, tangled canopy during the growing season. Surrounding it, weeds cluster thickly, occluding one another and the crop, while shifting sunlight throughout the day plays havoc with the color and texture cues that most computer vision systems depend on. Manual weeding remains the dominant practice in most Chinese medicinal herb fields, but it is labor-intensive, inefficient, and increasingly threatened by labor shortages. Chemical herbicides, the standard weapon against weeds in crops like wheat and maize, are largely off the table for medicinal herbs, which face stringent limits on pesticide residues because of their pharmaceutical end use. Physical methods such as laser and flame weeding consume enormous energy, and mechanical weeding demands the kind of uniform, precise row spacing found only in structured plantings like lettuce—not in the unruly reality of a perennial herb field.

That leaves weeding robots—integrated platforms combining mobility, navigation, visual perception, and actuation—as the most promising path forward. But the heart of any such robot, its visual perception module, has long been the weakest link. Most existing weed-detection systems rely on single-modality RGB cameras and deep learning models trained primarily on early-stage crops in sparse, simple backgrounds. When confronted with the dense, occluded, variable-lighting conditions of a mature medicinal herb field, their performance degrades sharply. As the researchers note, relying on visible texture and color alone simply does not provide enough discriminative information in such scenes.

The team’s key insight was that the answer was already hanging in front of the robot’s face. Modern weeding robots routinely carry depth cameras—low-cost 3D imaging devices that have become standard equipment for navigation and obstacle avoidance. Yet, as the researchers observed in their analysis of the literature, depth data in these systems has been used almost exclusively for computing three-dimensional coordinates after detection, not for the detection task itself. Meanwhile, in the field, measurable height differences and geometric relationships consistently exist between the herb plants and the weeds, and among the weeds themselves. That geometric signal, encoded in depth images, was being discarded exactly where it could be most useful.

Enter FW-DETR, the researchers’ new single-stage object detection model built on a hybrid convolutional neural network and Transformer architecture. The system’s core philosophy is what the team calls a modality-graded feature processing paradigm: rather than treating RGB and depth images as equal partners, it treats depth as an auxiliary modality whose geometric priors guide and refine the rich visual information coming from the color camera. This matters because the two modalities are fundamentally different in character. RGB images carry complex, high-frequency color and texture detail; depth images convey low-frequency, simple geometric structure and distance constraints. Most existing RGB-D fusion architectures, the researchers argue, fail to account for this disparity, conflating modality-specific components during fusion and blurring high-frequency features in the process.

To fix this, the team engineered two novel modules. The first, a cross-modal hierarchical feature interaction module, exploits the functional complementarity of the two sensors while suppressing redundant information. Within it, a cross-modal global context modeling component uses Transformer-based attention to enable global information exchange and alignment between the heterogeneous modalities, while a spatially adaptive local feature modeling component leverages convolutional operations to aggregate local details and boundary cues, calibrating features adaptively and suppressing noise. The second innovation, a cross-modal cascade attention module, performs grouped cross-modal interactions between high-level semantic features and depth features, explicitly embedding spatial geometric structure into the semantic representations. The result, according to the researchers, is reduced feature ambiguity and improved localization performance in complex scenes—a critical capability when a robot’s next action depends on knowing precisely where a weed begins and the crop ends.

The architecture was put to the test against a new benchmark dataset the team constructed, named PLPW2024. Assembled from images captured at a medicinal herb plantation in Bozhou using the first-generation field weeding robot developed by the group, the dataset comprises 1,512 paired RGB and depth images containing 47,633 annotated instances of crops and weeds. Images were acquired with an Intel RealSense D435i active-stereo camera, a device offering a valid depth range of 0.3 to 3 meters, mounted on a robot whose four main components—a vision system, mobile chassis, control system, and actuation module—were purpose-built for image collection and accuracy validation in real fields.

The choice of location was deliberate. Bozhou is one of China’s major medicinal herb production regions, where Paeonia lactiflora is widely cultivated, providing exactly the kind of real-world, unstructured scenarios that laboratory datasets so often lack. By building a standardized, paired RGB-D dataset from these conditions, the researchers have given the broader research community a unified benchmark for evaluating weed detection methods under genuinely challenging field conditions—dense canopies, mutual occlusions, and pronounced illumination variation among them.

The workflow that connects perception to action is as important as the detection model itself. After the vision system produces bounding box coordinates and class predictions in the two-dimensional image plane, those predictions are converted into executable three-dimensional spatial operation points that the robot can act upon, enabling targeted weed control without harming the crop. This closing of the loop—from multimodal perception to precise physical intervention—is what transforms an impressive detection algorithm into a functioning agricultural machine.

In extensive experiments, the proposed FW-DETR model improved detection accuracy across diverse scenarios and exhibited strong robustness and generalization, the researchers report. Crucially, the system was not merely validated in the abstract: field trials with the weeding robot demonstrated its effectiveness under real-world conditions, with the depth-guided architecture maintaining its performance where single-modality approaches would falter. The findings suggest that the humble depth camera, long relegated to a supporting role in agricultural robotics, may be one of the most underutilized assets in the fight for sustainable weed management.

The implications extend well beyond peony fields. Demand for Chinese medicinal herbs continues to climb, wild populations of Paeonia lactiflora can no longer meet market needs, and cultivated acreage keeps expanding—along with the weed pressures that threaten yield and quality. Weeds are not merely competitors for light and nutrients; they also serve as intermediate hosts for numerous pests and diseases, exacerbating infestations across entire fields. A perception system that can reliably separate crop from weed amid occlusion and visual clutter offers a template for selective weeding in other dense-canopy, unstructured crops where chemical, physical, and mechanical control have all fallen short.

More broadly, the study contributes to a growing recognition in the computer vision community that multimodal learning succeeds or fails on the quality of its fusion strategy. Data-layer and decision-layer fusion approaches, prior studies have shown, fail to capture the cross-modal correlations inherent in heterogeneous data. Feature-layer fusion—with independent branches learning modality-specific representations that are then interactively combined—has emerged as the dominant paradigm, and the Bozhou team’s graded, geometry-guided approach pushes that paradigm further by explicitly respecting the informational asymmetry between rich RGB detail and sparse depth structure. As attention mechanisms and Transformers continue to prove their worth at aligning representations across heterogeneous modalities, architectures like FW-DETR hint at how agricultural robots might eventually perceive fields the way experienced farmers do: not just by what plants look like, but by how they occupy space.

For now, the researchers’ weeding robot continues its work in the herb fields of Anhui, its stereo eyes gathering both light and distance, its neural network weighing color against geometry. What was once a hardware afterthought has become the guiding intelligence of the machine—a shift that could bring autonomous, chemical-free weed control closer to practical reality for the world’s medicinal herb growers, and perhaps for unstructured agriculture far beyond them.

The research, led by Wentao Song, He Huang, Fang Qu, Caiguo Tang, Youqiang Sun, and Junqing Zhang of the Hefei Institutes of Physical Science at the Chinese Academy of Sciences, together with collaborators at the University of Science and Technology of China and the Institute of Hefei Artificial Intelligence Breeding Accelerator, addresses a problem that has limited the accuracy of high-throughput phenotyping pipelines for years. When only partial regions of a wheat spike are visible in an image, downstream measurements such as spike length, width, and area are systematically biased. Traditional multi-view approaches can mitigate this by capturing the plant from many angles, but they demand expensive hardware, complex calibration, and time-consuming acquisition procedures that are impractical for large-scale breeding programs.

The new framework takes a fundamentally different approach. Rather than requiring multiple cameras or elaborate scanning rigs, it starts with a single image and applies a visible-prompt-driven amodal completion model to recover the complete appearance and structural cues of occluded targets. In computer vision terms, amodal completion refers to the estimation of an object’s full extent, including regions that are not directly observable because they are covered by other objects or fall outside the visible silhouette. The completion model in this study, Pix2Gestalt, uses visible portions of the wheat plant as prompts to synthesize plausible completions of the hidden parts, producing an image in which the target appears whole. This completed image is then fed into a single-view 3D reconstruction model that generates a full 3D structure of the plant organ.

A critical prerequisite for developing and validating such a system is data, and here the team made a substantial contribution of its own. They constructed the MMWO dataset—Multi-view Multi-instance Wheat Occlusion—derived from the MMW dataset captured under controlled indoor scenarios. Because real-world occlusion is difficult to annotate precisely, the researchers synthesized occlusion samples artificially through a carefully designed pipeline. Organ-level cutouts were extracted from the source imagery, subjected to random geometric transformations, and pasted back onto images within region constraints that mimic natural occlusion patterns. Each synthesized sample comes with rich annotations, including visible masks that delineate what can actually be seen, occlusion masks that mark the hidden regions, and complete target images that serve as ground truth for training and evaluation. This synthetic strategy gives the community a benchmark with exact knowledge of what is hidden and what is not, something nearly impossible to obtain from field photography alone.

With this dataset in hand, the researchers systematically evaluated six representative single-view 3D reconstruction methods: Direct3D, Real3D, SF3D, Spar3D, TRELLIS.2, and Hunyuan3D. These models represent the current frontier of 3D generative artificial intelligence, and applying them to wheat phenotyping is among the first systematic tests of their agricultural potential. The evaluation used complementary metrics: Chamfer Distance in its L1 and L2 variants (CD-L1 and CD-L2) to measure geometric fidelity between the reconstructed and reference 3D shapes, and the F-score as a balanced measure of precision and recall in the reconstructed geometry. Rendering quality was assessed with peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and the learned perceptual image patch similarity metric (LPIPS), which captures how visually similar rendered images appear to human observers and neural networks alike.

The results were revealing. Hunyuan3D achieved the best geometric performance among the six methods, recording the lowest mean CD-L1 and CD-L2 values of 0.1286 and 0.0536 respectively, along with the highest mean F-score of 0.5668. In other words, its generated 3D wheat structures were geometrically closest to the ground-truth targets. SF3D, by contrast, excelled in rendering quality, achieving the best scores on PSNR, SSIM, and LPIPS. This divergence between geometric accuracy and rendering fidelity is an important practical finding: a model that produces visually convincing images does not necessarily produce the most dimensionally accurate 3D geometry, and phenotyping applications that depend on precise measurements must weigh these trade-offs carefully when selecting a reconstruction backbone.

Perhaps the most striking evidence for the value of amodal completion came from the phenotypic measurements themselves. Without completion, the estimation errors for spike length, width, and area stood at 9.31 percent, 10.89 percent, and 32.23 percent respectively. The area error in particular—over thirty percent—illustrates just how severely occlusion distorts two-dimensional measurements derived from partially visible spikes. After applying Pix2Gestalt completion to the images before reconstruction and measurement, those errors dropped to 4.64 percent, 9.70 percent, and 9.45 percent. The improvement in spike area estimation is dramatic, cutting the error by more than two-thirds, while length estimation errors were roughly halved. These numbers demonstrate that filling in the invisible portions of a plant image is not merely an aesthetic enhancement but a substantive correction that propagates through the entire measurement pipeline.

The technical logic behind this improvement is worth unpacking. Single-view 3D reconstruction models learn strong priors about object shape from large training corpora, but when the input image contains a truncated or occluded target, the model receives misleading evidence about the object’s true extent. The reconstruction network may either hallucinate arbitrary geometry in the missing regions or, worse, faithfully reproduce the truncation as if the object genuinely ended where the occlusion begins. By intervening upstream with amodal completion, the framework provides the reconstruction model with a structurally complete prior. The visible-prompt mechanism ensures that the completion remains anchored to what is actually observed, so the synthesized hidden regions are constrained by the genuine appearance of the target rather than being generated in a vacuum.

The implications extend well beyond wheat. The authors position the work as a systematic reference for applying 3D generative models to agricultural phenotyping in general, and the framework’s architecture—completion followed by reconstruction—is agnostic to the specific crop, provided suitable completion and reconstruction models are available. For breeding programs that must screen thousands of genotypes for traits such as spike size and architecture, the ability to extract reliable 3D measurements from single photographs could dramatically reduce equipment costs and throughput barriers. Spike morphology is closely tied to grain number and yield, making accurate, non-destructive spike phenotyping a high-value target in crop improvement.

The study also offers a candid view of the current state of 3D generative modeling in plant science. The variation in performance across the six tested methods, and the divergence between geometric and rendering quality, signal that no single off-the-shelf model is yet ideal for phenotyping workloads. Benchmark resources like MMWO, with their exhaustive annotations of visible and occluded regions, give researchers the tools to measure progress systematically as generative 3D technology matures. The synthetic occlusion pipeline itself—cutouts, transformations, and constrained pasting—could be adapted to other crops and organ types, multiplying the utility of existing multi-view datasets by enriching them with occlusion scenarios.

This research was supported by the Anhui Provincial-Level Scientific and Technological Innovation Tackling Key Problems Plan Project, the Anhui Provincial Key Research and Development Project, and the Institute of Hefei Artificial Intelligence Breeding Accelerator. Published open access, the work invites plant scientists and computer vision researchers alike to build upon its framework. As the authors note, the study offers a feasible solution for 3D wheat phenotyping under occlusion, and with generative 3D models advancing at a remarkable pace, the marriage of amodal completion and single-view reconstruction may soon become a standard component of the digital agriculture toolbox—turning what a camera cannot see into data that breeders can use.