Robots Learn to Grab Any Fruit With 99 Percent Accuracy in Cluttered Scenes

Robots that pick fruit have long struggled with a deceptively simple problem: real harvests are messy. Fruit does not sit alone on clean tables; it hides behind leaves, overlaps with neighbors, and arrives in every size from a cherry to a papaya. A new study published in Artificial Intelligence in Agriculture tackles this gap with a trio of lightweight neural networks that can find a safe place to grip a piece of fruit in real time, even when the scene around it is crowded with clutter.

The study, led by Boyuan Cao and colleagues, arrives at a moment when agricultural robotics is shifting from laboratory demonstrations to field trials, and its central insight is that the bottleneck is often not the robot arm itself but the perception layer that decides where a gripper should close. A grasp, in the formal sense used by the authors, is not simply a point on an object. It is an oriented rectangle defined by five numbers: the horizontal and vertical coordinates of its center, the width of the opening between the gripper fingers, the height of the fingers themselves, and the angle at which the gripper should approach. Getting all five right, and getting them right quickly enough for a robot to act without pausing mid-harvest, is the core technical challenge the paper sets out to solve.

One of the most consequential contributions is a new dataset rather than a new robot. The authors assembled the Noisy Background Multi-Object grasp Detection dataset, or NBMOD, which contains 31,500 paired RGB and depth images. Depth cameras record not just the color of a scene but the distance of every pixel from the sensor, which matters enormously for grasping because a gripper must know how far to reach before it closes. The dataset is deliberately split into three subsets that mirror escalating real-world difficulty. The first, containing 13,500 samples, shows single fruits against simple backgrounds, the kind of tidy conditions that dominate older benchmarks. The second adds 13,000 samples in which everyday objects clutter the frame and partially occlude the target, replicating the visual chaos of a packing bench or a dense orchard canopy. The third contributes 5,000 multi-object scenes in which two or more graspable fruits appear together, forcing the network to decide not only where to grasp but which object to grasp first. Labels are stored in XML format, making the dataset straightforward for other groups to adopt and benchmark against.

The architectural choices in the paper reflect a careful reading of how deep learning for detection has evolved. The authors trace a lineage from AlexNet, which demonstrated in 2012 that deep convolutional networks could outperform hand-engineered features, through the family of detectors that followed, including R-CNN, SSD, feature pyramid networks, RetinaNet, and the YOLO series. Grasp detection inherited much of this machinery, and the paper organizes prior approaches along two axes. Two-stage methods first generate candidate grasp boxes and then rank them, a coarse-to-fine strategy that can be accurate but slow, because the staged pipeline adds latency that a harvesting robot cannot afford. Single-stage methods regress grasp coordinates directly from backbone features, trading some refinement for speed. Along the other axis, anchor-based methods impose prior structure on the prediction space, which helps models converge when training data is scarce, while anchor-free methods leave the model freer to explore but typically need far more examples to perform well.

That trade-off is particularly sharp in agriculture. As the authors note, grasp detection differs from generic object detection in three ways that shape design decisions. Targets in an orchard or greenhouse do not vary as wildly in scale as objects in general-purpose imagery. Labeled agricultural data is expensive and therefore scarce, since every image must be annotated by hand with oriented grasp rectangles. And precise angle prediction is unusually demanding, because a gripper that approaches a round fruit at the wrong orientation may collide with a stem or squeeze through a gap between neighbors. Given limited data and the need for angular precision, the authors chose an anchor-based design, and their key novelty is the Rotation Anchor Mechanism, which builds angle priors directly into the anchor set so the network does not have to learn orientation from scratch.

The backbone they selected is equally deliberate. Convolutional neural networks carry image-specific inductive biases, including locality, translation invariance, and a hierarchical structure, which make them sample-efficient learners. Their weakness is that shallow layers see only small receptive fields, limiting how much context they can gather early in the network. Transformers invert this profile: they excel at long-range connections and capture global information even in shallow layers, but they lack built-in image priors and are notoriously data-hungry. The hybrid solution, combining convolutional blocks with transformer blocks, has become a productive middle ground, and the authors adopted MobileViT in its small configuration as their backbone, a choice that keeps the model light enough for real-time inference on hardware that might plausibly be mounted on a field robot.

The related-work review in the paper doubles as a history of the field’s hard-won lessons. Lenz and colleagues, among the first to apply deep learning to grasping in 2015, used a two-network design that reached 73.9 percent accuracy on the Cornell Grasp Dataset with RGB-D input, establishing the coarse-to-fine template at the cost of real-time performance. Later, Song and coauthors refined Faster R-CNN with an anchor mechanism and pushed accuracy to 96.2 percent on the same benchmark using RGB images alone. On the single-stage side, Redmon and Angelova tested three prediction modes built on AlexNet and found that direct regression achieved 84.4 percent, regression with an auxiliary classification signal reached 85.5 percent, and a grid-based scheme reached 88 percent, an instructive result showing that extra supervision and spatially structured outputs both help. Their analysis also identified a persistent failure mode: flattening feature maps before a final multilayer perceptron discards spatial information that grasp prediction depends on.

Subsequent work wrestled with that limitation. Kumra and Kanan moved to ResNet-50, exploiting the residual connections that He and colleagues introduced in 2015 to solve the degradation problem that had long prevented very deep networks from training effectively. Their best configuration reached 89.21 percent on Cornell, but global average pooling still collapsed spatial detail. Ribeiro and Grassi proposed an end-to-end convolutional architecture with sliding-window augmentation and reached 94.8 percent, while Caldera and colleagues, also using ResNet-50, reported 93.91 percent. For multi-object scenes, Chu and colleagues decoupled angle prediction from the oriented box center, an early acknowledgment that rotation deserves its own treatment, a direction the Rotation Anchor Mechanism in the new paper extends systematically.

What makes the new study useful beyond its own benchmarks is the way it converts these accumulated lessons into design principles for a domain with unforgiving constraints. A fruit-harvesting robot cannot wait seconds for a two-stage detector to propose and then re-rank candidates while fruit waits on the branch. It cannot be trained on millions of labeled examples, because no one has annotated tens of thousands of orchard images with oriented grasp rectangles. And it cannot tolerate angle errors, because the difference between a clean pick and a bruised fruit is often a few degrees of gripper orientation. By pairing an anchor mechanism tuned for rotation with a hybrid backbone that balances sample efficiency against global context, and by releasing a dataset whose three subsets explicitly encode background clutter and multi-object crowding, the authors give the research community both a method and a measuring stick. The progression from simple to noisy to multi-object scenes in NBMOD also provides a diagnostic lens: a model that performs well on the simple subset but degrades sharply on the noisy one is revealing exactly where its perception fails, information that a single averaged accuracy figure would hide. For a field moving toward deployment in variable outdoor environments, that kind of granular evaluation may prove as valuable as any single accuracy number.

Subject of Research: Deep learning-based robotic grasp detection of fruits in noisy and multi-object scenes

Article Title: Real-time, robust, and highly accurate robotic grasp detection of fruits with variable sizes and shapes

Article References: Cao, B., Li, Y., Guo, C., Zhang, B., Liu, Y., & Tan, Q. (2026). Real-time, robust, and highly accurate robotic grasp detection of fruits with variable sizes and shapes. Artificial Intelligence in Agriculture. https://doi.org/10.1016/j.aiia.2026.05.013

Image Credits: AI Generated

DOI: 10.1016/j.aiia.2026.05.013

Keywords: robotics, grasp detection, deep learning, computer vision, agricultural robots, RGB-D imaging, Transformer, neural networks, fruit harvesting, dataset, automation, machine learning