The new system, described in the journal Artificial Intelligence in Agriculture, is called Depth4PH, short for “Depth for Plant Height.” It was developed by Zhi Wang, Zhi Yao, Demin Xu, Huayang Wang, Shuaipeng Fei, Xinyu Gu, Muxin Lu, Dongyu Wang, Jiayuan Li, Chunli Lv, Yuntao Ma, and Jinyu Zhu, and it tackles a problem that has frustrated agricultural scientists for years: how to get absolute physical measurements—not vague relative estimates—from consumer-grade cameras in messy, real-world field conditions.

Why a Single Photo Is So Hard

Modern approaches to measuring crops without touching them have generally fallen into two camps. LiDAR systems scan fields with lasers and build precise three-dimensional point clouds, but the hardware is expensive and the data processing is computationally punishing, which has kept LiDAR out of most large-scale applications. Structure-from-motion and multi-view stereo techniques reconstruct 3D scenes from multiple overlapping photographs, and while they demand cheaper equipment, they are notoriously fragile in the field. Strong sunlight, wind-blurred leaves, and dense overlapping canopies all cause the feature-matching algorithms to fail, and pulling clean ground points out of the resulting data is often a losing battle.

Monocular depth estimation—inferring depth from one single image—has emerged as an appealing alternative because it needs no stereo calibration, no extra sensors, and no multiple viewpoints. But general-purpose depth models have a fundamental limitation when pointed at farm fields: they output relative depth. They can tell you that one plant looks closer than another, but not that a maize stalk is 2.3 meters tall. Converting those fuzzy relative predictions into true metric measurements has required supervised training on data from the specific environment, and such data is scarce precisely where it matters most—unstructured, sun-drenched, wind-buffeted farmland.

Depth4PH attacks this gap with four linked components: a synthetic training data engine, a fine-tuned metric depth model, an automated segmentation system, and a robust physics-based height calculation algorithm.

Building a Fake Farm to Train Real Models

The first obstacle was training data. Accurate depth ground truth in real fields is nearly impossible to collect: structured-light and time-of-flight depth cameras are blinded by outdoor infrared interference, producing hole-riddled depth maps, while LiDAR point clouds are too sparse to capture slender stems and fine leaf edges when projected into images.

The team’s workaround was to grow their crops inside a computer. Using the Blender 3D physics engine, they constructed a high-fidelity Virtual Agricultural Scene, or Blender VAS. Procedural models built on L-system theory and branching fractal algorithms generated thousands of soybean and maize plants with botanically plausible topology, each varied by pseudo-random seeds. Multifractal noise displacement maps perturbed the virtual terrain to reproduce the ridges and micro-undulations of real field soil.

Lighting was handled with equal care. The rendering engine implemented Nishita sky models grounded in Rayleigh and Mie scattering physics, allowing the researchers to simulate harsh morning sun, high-contrast noon shadows, and diffuse evening light by tuning solar elevation, atmospheric turbidity, and cloud cover. Crucially, because this is a virtual world, absolute depth comes for free: the renderer reads the Z-buffer depth channel directly from the virtual camera and computes the true Euclidean distance from every surface to the optical center, encoded losslessly in 16-bit floating point.

To keep synthetic images from being too alien to real-world photos, the team fused their virtual dataset with open-source agricultural benchmarks like AgriBench and WE3DS plus real field imagery, using median scaling to align relative depth spaces with absolute physical scale. All images were resampled to 518 × 518 pixels to match the vision transformer architecture underneath the model.

A Depth Model That Refuses to Forget

The core of the system is TAM-Depth V2, a fine-tuned version of the open-source Depth Anything V2 foundation model adapted for agricultural scenes. Rather than retraining the whole network—a recipe for catastrophic forgetting when training data is limited—the team used parameter-efficient fine-tuning. The first 18 blocks of the ViT-L encoder, containing roughly 226.77 million parameters, were frozen solid, preserving the general visual and geometric knowledge learned from millions of natural images. Only the deeper, task-specific blocks and a reconstructed Dense Prediction Transformer decoder were left trainable, keeping total trainable parameters to about 31.8 percent.

A new absolute depth prediction head converts the network’s fused features into true metric depth. A single convolutional layer and a sigmoid activation produce values between 0 and 1, which are then mapped to meters using a configurable maximum scene depth of 4.5 meters—a figure chosen because imaging platforms typically hover 3 to 4 meters above the ground to capture tall crops like maize, which can reach 2.8 meters.

Training used a masked hybrid loss combining three objectives: an L1 loss robust to optical outliers, a structural similarity (SSIM) loss that preserves the topological coherence of crop surfaces, and a multi-scale edge gradient loss that sharpens depth discontinuities where overlapping leaves meet. Training ran for 50 epochs with an AdamW optimizer, differential learning rates, and gradient accumulation on an Apple M3 Max workstation—hardware that reflects the framework’s emphasis on efficiency over brute force.

Auto-Piloting the Segment Anything Model

Accurate depth alone isn’t enough; the system must isolate individual plants from cluttered canopies and background clutter like irrigation pipes and support stakes. Depth4PH uses SAM 2, the Segment Anything Model, but replaces its usual human-provided prompts with an automated engine called MSP-SAM2.

The engine works from two streams. In the RGB branch, the Excess Green vegetation index with Otsu thresholding carves out an initial vegetation mask. In the depth branch, an inverse watershed algorithm flips the depth topography so that protruding canopy apexes become local convergence centers, which the H-minima transform identifies as positive prompt points marking individual plants. Meanwhile, a discrete Laplacian operator on the depth map, combined with Line Segment Detector geometry checks, identifies linear man-made structures—pipes, poles, stakes—that would otherwise seduce SAM 2’s masks into semantic overflow. Dense negative prompts along these structures suppress the expansion. The result is fully zero-shot instance segmentation with no human in the loop.

From Pixels to Centimeters

The final component, RANSAC-Per, converts depth maps and masks into physical heights. For each segmented plant, it collects the depth values within the mask and takes the 3rd-percentile depth—the nearest points—as the canopy apex, a truncated percentile trick that ignores floating optical noise spikes that would corrupt a naive minimum. On the ground side, a RANSAC regression fits a local plane to soil pixels beneath the plant, resisting ridge undulations and terrain distortion. After compensating for camera tilt using either strict nadir calibration or onboard IMU measurements, the vertical height falls out of simple trigonometry.

Tested against 350 physically measured plants across five crops—cucumber, tomato, soybean, cotton, and maize—collected at the Chinese Academy of Agricultural Sciences’ Xinxiang base and Beijing’s Xiaotangshan National Precision Agriculture Research Base, with zero overlap between training and test sites, the framework consistently outperformed baseline methods that relied on global or dilated min-max statistics, which were far more vulnerable to terrain distortion and optical noise.

The implications reach beyond plant height. The same synthetic-data-plus-foundation-model pipeline could extend to canopy volume, biomass estimation, lodging risk assessment, and precision irrigation planning. And because the framework runs on consumer hardware rather than exotic sensors, it lowers the entry barrier dramatically for breeding stations and research farms worldwide. As climate pressures intensify the need for rapid crop improvement, tools that turn an ordinary photo into a breeding decision may soon be as essential as the ruler they replace.

At the core of this innovation lies the development of an intricate digital environment designed to simulate and optimize the operation of self-driving tractors within vineyard settings. This virtual ecosystem, often referred to as a “digital twin,” meticulously mirrors the vineyard’s physical characteristics—including slope gradients, soil texture variations, and the precise geometry of row spacing. By creating this highly realistic simulation, researchers can rigorously test autonomous navigation algorithms, refine control strategies, and predict operational outcomes without the inherent risks and costs of real-world trials.

The research team, comprising experts from the Departments of Mechanical Engineering and Electronics, Information and Bioengineering, has engineered a comprehensive methodological framework for scenario generation. This framework encompasses the synthesis of detailed terrain models and the integration of sensor data streams typical of actual agricultural machinery. By employing low-cost Global Navigation Satellite Systems (GNSS) alongside Inertial Measurement Units (IMU), the virtual tractors execute precise movements that replicate true field conditions. These autonomous vehicles demonstrate an ability to navigate between tightly spaced vine rows and perform complex off-field maneuvers with unprecedented accuracy.

Key to this advancement is the synergistic combination of terrain modeling, sensor emulation, and advanced control algorithms, all embedded within a singular simulation environment. This integration ensures that every aspect influencing vineyard operations—from micro-topography to sensor noise—is faithfully represented. Such fidelity accelerates the iterative process of developing autonomous control systems, bridging the gap between theoretical algorithmic strategies and practical field applications. Furthermore, it dramatically curtails the time and resources traditionally allocated to on-site validation, presenting a transformative leap forward for agricultural engineering.

The implications of this digital twin extend beyond mere operational efficiencies. Researchers envision these simulations as vital tools for operator training, facilitating a controlled setting where human supervisors can familiarize themselves with autonomous machinery behaviors and refine intervention protocols without risking crop damage or machinery faults. This educational facet is seen as a crucial catalyst for broader adoption of automation technologies, breaking down barriers related to unfamiliarity and mistrust that commonly hinder innovation in traditional farming communities.

The initiative has been bolstered through a collaborative partnership with Soluzioni Ingegneria s.r.l., an industry leader specializing in dynamic vehicle simulation software. This alliance embeds academic innovation within the practical frameworks demanded by commercial agricultural equipment manufacturers, ensuring that theoretical models translate effectively into market-ready technologies. It is this blend of scholarly research and industrial application that positions the project at the nexus of agricultural automation and sustainable farming.

From a sustainability perspective, the precise control afforded by autonomous tractors promises substantial environmental benefits. More exact navigation reduces soil compaction and disturbance, while the ability to execute carefully modulated maneuvers minimizes resource wastage. Additionally, optimizing field operations through predictive algorithms aligns with broader ecological goals to reduce carbon footprints and conserve soil health, thereby contributing to resilient agricultural ecosystems in the face of climate change.

This research arrives at a pivotal moment for global agriculture, where the dual pressures of increasing food demand and environmental stewardship mandate innovative technological responses. By embedding advanced robotics and sensor technologies within the unique context of vineyards, the Politecnico di Milano project addresses a sector ripe for disruption. Traditional viticulture, often labor-intensive and tailored to delicate crop requirements, is now poised to embrace automation without compromising the intricate balance of terroir and grape quality.

The published findings, detailed in the latest issue of AgriEngineering, offer a comprehensive analysis of the methodologies employed, simulation parameters, and algorithmic control strategies. The paper elucidates the challenges inherent in replicating complex vineyard environments digitally and discusses the iterative optimization process through which autonomous tractor performance was enhanced. This transparency not only advances academic discourse but also provides a valuable reference point for developers and practitioners aiming to implement similar systems across diverse agricultural contexts.

Beyond the immediate scope of viticulture, the technological principles underpinning this research hold broader applicability. The combination of digital twins with sensor-driven autonomous machinery paves the way for smarter, more resilient farming operations across various crops and terrains. By validating control strategies in virtual environments, stakeholders can anticipate system behaviors under myriad conditions, thereby reducing uncertainty and enhancing decision-making efficacy.

The vision articulated by Professor Federico Cheli, who leads the Mechanical Engineering department and coordinates the project, encapsulates a future where digital simulation is integral to agricultural innovation. His emphasis on integrating realistic sensors, terrain data, and advanced controls reflects a holistic approach that acknowledges the multifaceted challenges of field automation. This mindset exemplifies the evolving role of engineering disciplines in shaping next-generation food systems.

As autonomous technologies gain foothold in agriculture, the nexus of simulation science, robotics, and environmental sustainability will undoubtedly expand. The achievements of the Politecnico di Milano team demonstrate the feasibility and value of a simulation-first approach to developing high-precision vineyard operations. Their work embodies a critical step toward smart agriculture, where data-driven insights and autonomous control converge to cultivate not only grapes but a more sustainable and productive future for farming worldwide.

Subject of Research: Not applicable
Article Title: Scenario Generation and Autonomous Control for High-Precision Vineyard Operations
News Publication Date: 11th November 2025
Web References: http://dx.doi.org/10.3390/agriengineering7020046
References: Ruiz Mayo, C.; Cheli, F.; Arrigoni, S.; Paparazzo, F.; Mentasti, S.; Pezzola, M.E. Scenario Generation and Autonomous Control for High-Precision Vineyard Operations. AgriEngineering 2025, 7(2), 46.
Image Credits: Politecnico di Milano
Keywords: Agricultural engineering, Sustainable agriculture, Robots