AI Helps Excavators Track Their Buckets Even When Hidden From View

Excavators are among the most recognizable machines on Earth, their articulated arms swinging over construction sites, open-pit mines, and disaster zones. Yet behind every smooth bucket swing lies a stubborn measurement problem: to automate these machines, engineers must know exactly where the bucket is and how it is oriented, in real time, even when the bucket is buried in soil, hidden behind a dump truck, or obscured by material piles. A research team led by Zhi Chen, Tongyao Wang, Zain Anwar Ali, Quang Phuc Ha, and Qing Sun, publishing in the open-access journal Results in Engineering, has now unveiled a learning-based position tracking method that keeps a virtual eye on the excavator bucket under exactly those punishing conditions, fusing inertial sensors with depth-camera vision and compressing the deep learning model until it fits comfortably on the excavator’s own embedded computer.

The stakes are considerable. Excavator operation has long depended on skilled human operators, but the hazardous nature of the work has stalled the influx of new workers, pushing researchers and manufacturers toward automation as a route to operator safety and productivity. Accurate measurement of the machine’s motion state is the essential prerequisite for that transition: the position and orientation of the boom, stick, and bucket must be captured continuously to support motion planning, trajectory control, and interaction analysis with soil, humans, and other machines. The bucket, acting as the end-effector, ultimately determines excavation accuracy, which makes its pose the most critical and the most difficult quantity to estimate.

Existing approaches have struggled on two fronts. Contact-based schemes embed accelerometers, encoders, LiDAR scanners, or inertial measurement units directly on the machine, but these sensors grapple with calibration discrepancies, noise, placement intricacies, synchronization issues, and maintenance concerns. Strong vibrations accompanying intelligent excavators can cause linear deviation, fatigue, and wear of LiDAR equipment, degrading detection accuracy, while a bucket-mounted IMU faces vibration, installation loosening, contact with excavation material, and contamination over long periods. Vision-based measurement, meanwhile, is vulnerable to illumination changes and viewpoint variation, and the frequent interactions of the bucket with soil and structures produce visual occlusions that can block as much as 30 percent of the field of view when a dump truck obscures the excavator body or a material pile hides the arm.

The team’s answer is a multi-sensor fusion framework that refuses to choose between contact and non-contact measurement. Instead of relying on a single modality, the method fuses IMU-generated keypoints with RGB-D features from a depth camera, boosting pose observability precisely where occlusion interference would otherwise cripple a purely visual system. The architecture is deliberately two-layered: a lightweight vision algorithm extracts structural keypoints on the excavator’s linkage surface, and a time-series neural network then processes the fused information to capture continuous bucket angle estimation across four distinct excavator activities: static, excavation with rotation, excavation without rotation, and leveling.

The vision module begins with YOLOv8-Pose as its baseline for keypoint detection but is thoroughly redesigned for embedded deployment. An improved C2f backbone module integrates channel reorganization, re-parameterizable convolution, and multi-scale semantic aggregation within a unified structure, splitting the feature map into a residual shortcut branch for gradient stability and a re-parameterizable convolutional branch that progressively enriches semantic information before all intermediate outputs are concatenated and projected to the target channel dimension. The detection head is likewise rebuilt with three efficiency-oriented modifications: convolutional weight sharing across pyramid levels to avoid redundant parameters, Group Normalization to ensure stable training under the small batch sizes typical of embedded scenarios, and decoupled prediction branches for bounding box regression and keypoint localization to reduce task interference.

Model compression is then taken a decisive step further through layer-wise adaptive magnitude pruning, or LAMP, a global pruning strategy that dynamically adjusts sparsity levels across layers. Each parameter’s importance is scored by the ratio of its squared magnitude to the sum of squared magnitudes of all remaining connections, weights are sorted globally, and the lowest-scoring parameters are removed iteratively until the desired reduction in floating-point operations is achieved, after which the model is fine-tuned to recover any compromised accuracy. The numbers are striking: the pruned model is compressed to 0.25 million parameters and 2.5 GFLOPs from the baseline’s 3.08 million parameters and 8.3 GFLOPs, marking 91.88 percent and 69.88 percent reductions in trainable parameters and computational overhead, while model size shrinks from 6.17 MB to 0.75 MB, an 87.85 percent reduction. Crucially, detection accuracy barely moves, with mAP@0.5 holding at 99.4 percent against the baseline’s 99.5 percent and precision actually rising to 99.5 percent.

The temporal estimation layer is a lightweight time-series network the authors call RF-RNN, a residual and feed-forward module integrated recurrent neural network comprising a fully connected layer with ReLU activation, a standard RNN, and residual connections that smooth data flow and preserve training. The team deliberately chose a standard RNN over gated architectures such as GRU or LSTM to keep computational overhead low, a defensible trade because the system models short-term temporal dependencies with a sliding window of just 12 steps, so vanishing gradients and long-term dependency modeling are not hindrances in real-time deployment. The network follows an N-to-One architecture, receiving integrated IMU and visual features as sequential inputs and producing a single bucket angle estimate for the current time step, with only 3,431 trainable parameters. During training, the bucket-embedded IMU serves purely as a supervision cue, meaning that after training the model can estimate the bucket angle without requiring any bucket-mounted sensor at all.

Geometry plays a supporting role that is as elegant as it is practical. The excavator’s working device is modeled with the improved Denavit-Hartenberg convention, establishing coordinate systems at each joint with four parameters per link, and the bucket’s rigid four-bar linkage with the stick allows the bucket joint angle to be inferred from visual keypoint observations through the law of cosines and auxiliary angles derived from mechanical design specifications. Because camera distortions, mounting errors, and deviations from planar suppositions create nonlinear projection errors in real-world scenes, the geometrically inferred angle is served not as the final output but as an auxiliary observation, which the learning-based temporal model then refines, curbing projection bias and improving robustness under dynamic scenarios. Kinematic modeling is integrated into geometric modeling so that the estimated joint angles can be transformed into full 3D bucket poses, enabling consistent pose feedback for motion control.

Experiments on a compact hydraulic prototype equipped with an RGB-D camera, three high-precision IMUs, an encoder, and an NVIDIA Jetson TX2 processing unit validated the framework across all four operating conditions. Under static conditions the predicted curve completely overlaps the IMU reference over a 100-second interval, with errors between -0.5 and 0.4 degrees and a root mean square error of just 0.3358 degrees. During fixed-point excavation without rotation the RMSE rises to 6.7741 degrees with a skewed error distribution whose median stays near zero, while rotational excavation, with its quicker and more complex motion patterns, yields 6.4009 degrees RMSE with a negative bias. Leveling, involving smoother and more regular movements, produces the most concentrated error distribution at 3.8612 degrees RMSE with a moderate positive bias. The authors attribute the larger local deviations to fixed camera installation effects, limited representation of certain motion patterns in the training data, and processing delays between the visual and IMU streams, but they note that the angle errors correspond to only about 1 to 2 centimeters of end-effector pose deviation, within the acceptable criterion for engineering applications.

Compared with prior learning-based and vision-based methods, the framework holds its own: earlier deep learning approaches reported average RMSE values of 8.18 degrees and worst-case errors approaching 49.6 degrees, while monocular vision marker systems achieved orientation errors below 8.5 degrees but lacked temporal prediction and multi-sensor information. The new method delivers a lower overall average bucket angle error with no spikes, moderate accuracy with better adaptability under different occlusion scenarios, and a modular design that can be deployed on different sized excavators. The team acknowledges that the laboratory dataset, captured at Shanghai University under natural lighting with 161 retained samples augmented to 322, is prototype-level validation, and future work will expand the dataset to include various camera mounting positions, diverse operating conditions, and multiple viewpoints. For now, the research stands as a robust measurement solution that harmonizes estimation accuracy with real-time deployment, exactly where sensor durability and occlusion have long been the critical bottlenecks of autonomous excavation.

Subject of Research: A learning-based position tracking method for excavator bucket

Article Title: A learning-based position tracking method for excavator bucket

Article References: Chen, Z., Wang, T., Ali, Z. A., Ha, Q. P., & Sun, Q. (2026). A learning-based position tracking method for excavator bucket. Results in Engineering, 32, Article 113464. https://doi.org/10.1016/j.rineng.2026.113464

Image Credits: AI Generated

DOI: 10.1016/j.rineng.2026.113464

Keywords: learning-based, position, tracking, method, excavator, bucket, scientific research