Tomatoes present a particularly difficult harvesting problem for robots because they grow in tightly packed clusters on vines that are easily damaged. A robotic arm must navigate between fruits, avoid contact with surrounding produce, and apply just enough force to detach the tomato without crushing it. Earlier systems struggled with this task because they relied on generic fruit-detection models that could not account for the complex spatial relationships within a cluster.

The research team, led by Zheming Cui and Jufeng Wang, developed a two-stage approach. First, a convolutional neural network identifies individual tomatoes and classifies them by ripeness stage. Second, an optimisation algorithm determines the optimal approach vector for the end-effector, taking into account the positions of surrounding fruits, the orientation of the pedicel, and the geometry of the gripper itself.

In field trials conducted in a commercial greenhouse, the system achieved a harvesting success rate of 84.6 percent while reducing fruit damage to 6.2 percent, a substantial improvement over earlier vision-based harvesting systems that reported damage rates above 15 percent. The average time to complete a single harvest cycle was 9.4 seconds, which the researchers note is competitive with human pickers working at moderate speed.

The key innovation lies in how the system models the picking action. Rather than treating each tomato as an isolated target, the algorithm constructs a local geometric model of the entire cluster, identifying which fruits are obstacles and which are targets. This allows the robot to plan approach trajectories that thread between fruits rather than approaching them head-on, dramatically reducing unintended contact.

The researchers also addressed the problem of occlusion, where leaves and neighbouring fruits partially obscure the target. Their segmentation network was trained on annotated images of occluded tomatoes, allowing it to infer the full three-dimensional shape of a fruit even when only a portion is visible. This capability proved essential for reliable pose estimation in dense canopy conditions.

The system was tested on three tomato varieties with different fruit sizes and cluster densities, and the researchers report that performance remained consistent across all three. They suggest that the approach could be adapted to other crops that grow in clusters, including grapes, peppers, and certain citrus varieties, although further validation would be required.

The work represents part of a broader effort to develop autonomous harvesting systems capable of operating in real agricultural environments rather than controlled laboratory conditions. As labour shortages continue to affect the agricultural sector worldwide, the demand for reliable automated harvesting technology has intensified considerably in recent years.

The research team acknowledges several limitations. The current system requires relatively uniform lighting conditions and cannot yet operate effectively in direct sunlight or at night. The gripper design, while effective for the tested varieties, may require modification for fruits with different skin properties or attachment geometries. The researchers plan to address these challenges in follow-up work, including testing the system across multiple growing seasons and under varying weather conditions.

The study, authored by Zheming Cui, Jufei Wang, Wuxiong Weng, Jun Fu, Xingwang Wang, Shuhe Zheng, and Jinbo Ren, addresses a problem that has long frustrated agricultural roboticists: the gap between knowing where a fruit is and knowing how to grasp it safely. Large-fruited varieties such as Provence and Beefsteak tomatoes ripen asynchronously within a single cluster, which means that even when most fruits on a truss are still green, one or two may be ready for picking. A harvesting robot therefore cannot simply strip an entire cluster; it must single out individual fruits while their unripe neighbours remain firmly attached and in the way.

A central insight of the work is that geometric fitting should not be performed blindly on whatever point cloud the sensor happens to return. Many earlier systems reconstructed fruit geometry directly from partial scans, fitting spheres or other primitives to whatever surface points were visible. When occlusion removes a large fraction of a fruit’s surface, such fitting can produce a centre estimate that is systematically biased, because the visible points no longer represent the fruit symmetrically. The Fujian team introduced a reliability assessment step that evaluates whether the observable surface contains enough information to support stable sphere estimation before any fitting is attempted. Point clouds judged insufficient are prevented from propagating errors downstream into pose generation.

Where the observable surface does pass this screening, the researchers apply a reliability-oriented sphere-fitting strategy that combines robust fitting with an inlier-consistency evaluation and a bootstrap-based uncertainty assessment. The bootstrap component is notable: by repeatedly resampling the inlier points and refitting the sphere, the algorithm can quantify how confident it should be in the estimated centre. This kind of uncertainty awareness is uncommon in agricultural perception pipelines, where a single point estimate is typically passed forward regardless of how well the underlying data constrain it.

The hardware platform underlying the experiments reflects the practical constraints of greenhouse deployment. The system runs on an NVIDIA Jetson Orin Nano, an embedded computing module chosen for its balance of processing power and low power consumption, which is essential for a mobile harvesting robot that cannot rely on a tethered workstation. An end-effector-mounted LiDAR camera captures both colour and depth information, and hand-eye calibration aligns the camera coordinate frame with that of the robotic arm so that perception outputs can be translated directly into arm motions. Mounting the sensor on the end-effector itself, rather than on a fixed overhead frame, allows the robot to re-examine a cluster from a favourable vantage point as it approaches.

The dual-mode pose-generation strategy represents another departure from conventional approaches. When the pedicel, the stalk connecting the fruit to the vine, can be detected in three dimensions, the system uses its spatial orientation to guide the gripper along a natural detachment direction. Pedicels, however, are thin, frequently occluded structures, and keypoint detection on them has historically been unstable. When pedicel information is unavailable, the system falls back on the sphere-centre coordinates of neighbouring fruits, using their positions to select an approach vector that minimises the risk of collision. This graceful degradation between modes is what allows the system to maintain feasibility across the wide variety of cluster configurations found in a real greenhouse.

The distinction between positioning accuracy and harvesting success deserves emphasis. Prior studies cited in the paper reported positioning accuracies that could exceed seventy percent while actual harvesting success rates lagged well behind. The discrepancy arises because a correctly localised fruit can still be harvested unsuccessfully if the gripper, whether claw-type, sleeve-type, or suction-type, clips a neighbouring fruit during approach or detachment. By explicitly incorporating neighbouring-fruit geometry into pose generation, the new method targets the interaction phase of harvesting rather than the perception phase alone, which is where many earlier systems quietly failed.

The broader literature context situates this work within a decade-long progression. Earlier efforts combined binocular vision with interpolation to reconstruct citrus fruits, achieving diameter errors below 2.75 millimetres, while other teams integrated Mask R-CNN with RANSAC for apple posture estimation, reaching positional accuracy of fifteen millimetres and a seventy percent successful positioning rate. Deep completion networks have been applied to recover full fruit geometry from partial observations, but these typically depend on pre-trained shape models that may not generalise across fruit morphologies. Model-free approaches that estimate pose directly from image features offer faster inference but can struggle with the ambiguity inherent in heavily occluded scenes. The reliability-aware strategy occupies a middle path, using classical geometric fitting strengthened by statistical uncertainty quantification rather than depending on learned shape priors.

The choice of LiDAR over stereo vision or structured light also carries practical significance. LiDAR cameras provide direct depth measurement without the correspondence errors that plague stereo matching on textureless surfaces, a real concern for smooth-skinned tomatoes. However, LiDAR suffers from missing depth returns at grazing angles and on small structures such as pedicels, which is precisely why the paper notes that depth information at keypoint locations is often absent. The dual-mode strategy can be read as a direct engineering response to this sensor limitation: rather than trying to fix unreliable pedicel depth estimates, the system routes around them.

For commercial deployment, the implications extend beyond tomatoes. Dense-planting systems are increasingly favoured by growers because they raise per-hectare yields, but they simultaneously worsen the occlusion and collision problems that plague robotic harvesting. Any crop grown in tight clusters, from table grapes to capsicum, faces a version of the same challenge. The general principle demonstrated here, that pose generation should account for the neighbourhood geometry of the target rather than treating it in isolation, transfers readily to these settings, though the specific gripper geometry and ripeness classification would require adaptation.

The work also highlights a methodological point relevant to the field at large. Evaluating reconstruction quality only through final fitting accuracy, as many prior studies did, conceals whether the input data were adequate in the first place. By assessing the suitability of observable surface data before reconstruction, the researchers introduce a quality gate that makes the downstream pipeline more predictable and its failures more diagnosable. As agricultural robots move from laboratory demonstrations into commercial greenhouses, such reliability-aware design patterns are likely to become standard practice rather than novelty.

Subject of Research: agricultural robotics, tomato harvesting, deep learning, precision agriculture, grasp planning, agricultural automation

Article Title: End-effector picking-pose estimation for tomato harvesting: a fusion approach combining instance segmentation and optimised point-cloud fitting

Article References: Cui, Z., Wang, J., Weng, W., Fu, J., Wang, X., Zheng, S., & Ren, J. (2026). End-effector picking-pose estimation for tomato harvesting: a fusion approach combining instance segmentation and optimised point-cloud fitting. Artificial Intelligence in Agriculture. https://doi.org/10.1016/j.aiia.2026.09.011

Image Credits: AI Generated

DOI: 10.1016/j.aiia.2026.09.011

Keywords: agricultural robotics, tomato harvesting, deep learning, precision agriculture, grasp planning, computer vision, agricultural automation, greenhouse technology, fruit damage reduction, robotic perception, End-effector, picking-pose