The problem is deceptively simple to describe. Imagine a diver hovering over a coral reef who wants a robotic companion to inspect a piece of debris, photograph a specific coral head, or retrieve a dropped tool. If two objects sit in the same direction but at different distances, a two-dimensional interpretation of the pointing gesture is fundamentally ambiguous. The robot cannot tell whether the diver means the sponge three meters away or the wreck ten meters beyond it. Resolving that ambiguity requires depth information, and obtaining reliable depth information underwater is notoriously difficult. Traditional dense stereo vision, the standard technique for computing distance from paired camera images on land, breaks down beneath the surface because seawater absorbs most frequencies of incident visual light, leaving scenes washed in blue-green, starved of texture, and plagued by optical attenuation. Algorithms that estimate distance by matching every pixel between a left and right camera image produce wildly inconsistent results in these conditions.

The research team, led by Chelsey Edge of Carleton College, Demetrious Kutzke and Junaed Sattar of the University of Minnesota, and Megdalia Bromhal of the University of North Carolina Wilmington, sidestepped the problem with an elegant trick. Rather than attempting to reconstruct the entire scene in three dimensions, their algorithm reconstructs only a handful of carefully chosen points. A human pose estimator identifies three anatomical landmarks on the diver’s body: the wrist, elbow, and shoulder of the pointing arm. A feature detector, in this implementation the classic SIFT algorithm, picks out distinctive keypoints on candidate objects of interest in the scene. Because these points appear in both the left and right images of the robot’s stereo camera, they can be matched and triangulated using standard stereo geometry, yielding sparse but reliable three-dimensional coordinates even when dense reconstruction fails.

The mathematics at the heart of the method is a triangulation pipeline familiar from classical computer vision. For each matched keypoint pair, the disparity between its horizontal positions in the two rectified images is computed, and depth follows from the relationship that disparity equals the camera’s focal length multiplied by the stereo baseline, divided by the distance. A reprojection matrix then converts these disparities into full camera-frame coordinates. The team filtered out implausible results, discarding keypoints whose wrist-to-elbow or elbow-to-shoulder separation exceeded empirically determined limits, and rejecting computations that produced invalid, infinite depths. What remains is a sparse constellation of three-dimensional points describing the diver’s arm and each potential target.

From there, the robot reasons geometrically. The algorithm extends a vector from the diver’s elbow through the wrist, scaling it outward by a factor of three so that it sweeps through the volume in front of the diver. Each candidate object’s perpendicular distance to this extended pointing line is calculated using the cross-product formula for point-to-line distance in three dimensions. The object closest to the line wins. The robot then projects that winning object back into the two-dimensional image plane, where conventional visual servo controllers can drive the vehicle toward it. Crucially, the framework is modular: any state-of-the-art pose estimator or object detector can be swapped in as those technologies improve, without redesigning the pipeline.

Validating the system demanded real-world evidence, and the team delivered it in two dramatically different arenas: the controlled environment of a swimming pool and the open waters of the Caribbean Sea off the coast of Barbados. Underwater ground-truth measurements are genuinely difficult because the robot, diver, and objects are all in constant relative motion. To manage this, the researchers laid an instrumented trackline on the pool floor and the seafloor, marked with numbered placards at meter intervals and rigged with fishing floats for visibility, allowing approximate distances from the camera to be known during each experimental run. Both the Aqua AUV’s onboard stereo cameras, with a baseline of roughly 91 millimeters, and an external ZED-Mini stereo camera were used to guard against hardware-specific artifacts, and the cameras were calibrated in situ using waterproof AprilTag calibration boards.

The results were encouraging. In the pool, when both the diver and the target object were within about three meters of the vehicle, DIP-3D selected the correct object in the majority of trials, and the located object landed on average 117 pixels from the ground-truth annotation, roughly ten percent of the image height. In the open ocean, where turbidity and natural swells degraded visibility and station-keeping, performance held up within about two meters, with the average error shrinking to about 76 pixels, or six percent of the image height. The system even handled cases where the target was barely within the camera’s frame. Beyond those distances, the pose estimator simply could not detect the diver reliably, a limitation the team attributes to current pose-detection technology rather than to the pointing framework itself.

Perhaps the most thought-provoking evaluation involved comparing the algorithm against human perception. Because no prior method exists for interpreting three-dimensional diver pointing, the researchers recruited 305 participants through Amazon Mechanical Turk to examine ten images in which DIP-3D had succeeded, drawing bounding boxes around where they believed the diver was pointing. Each image received sixty independent labels, totaling 600. The comparison revealed something intuitive: when humans had an anchor feature such as the seafloor or pool bottom, their judgments clustered close to the algorithm’s prediction, often within a few pixels. But when the scene was ambiguous, for instance with a diver pointing at an object suspended in open water with no reference bottom, human error ballooned to as much as half the image dimensions, with some participants convinced the diver was indicating a small object behind them. The algorithm, anchored in genuine three-dimensional geometry, was immune to precisely the visual illusions that fooled people.

Runtime performance matters for a robot that must respond to a human holding a pose in open water. On a standard laptop-class CPU, the full DIP-3D pipeline completed in 0.74 seconds per stereo pair. On the Nvidia Jetson TX2 edge computer that flies aboard the Aqua AUV, the cycle took 2.56 seconds, about 0.39 cycles per second. That sounds slow compared with terrestrial real-time vision, but the authors argue it is entirely practical: recreational, commercial, and military diving standards already instruct divers to make deliberate, clear gestures and to pause between signals, so holding a pointing pose for three seconds is well within normal diving practice. Future improvements in pose estimation and object detection networks will only accelerate the pipeline.

The broader significance reaches beyond a single gesture. Underwater work, from pipeline inspection and gas-leak detection to coral surveys, archaeological mapping, and deep-ocean monitoring, increasingly relies on robots, yet deployed AUVs typically receive hard-coded mission plans before launch, and the only channel for in-situ reconfiguration is low-bandwidth acoustic communication that cannot carry rich instructions. ROVs offer flexibility but tether the operator’s awareness to the vehicle’s own sensors, creating blind spots and entanglement risks. DIP-3D points toward a third paradigm in which experienced divers and autonomous robots collaborate directly, with the diver simply pointing at what needs attention and the robot doing the rest. Because pointing is already part of every diver’s natural gesture repertoire, the interaction adds essentially no cognitive or physical burden to a human who is already managing a demanding task.

The authors are candid about remaining challenges. Divers assume poses underwater that no terrestrial pose estimator was trained to anticipate: cross-body pointing, hovering horizontally above the seafloor, and the visual clutter of buoyancy vests, regulators, and fins all complicate keypoint detection. A wider stereo baseline could extend the effective range but would demand a larger vehicle; the team notes that newer platforms such as the CUREE and MeCO AUVs could accommodate one. Newly emerging stereo-based three-dimensional pose estimators that require no 3D supervision may also boost accuracy and range. Still, the foundation is in place. The team envisions a future in which gesture classification is layered on top of pointing, letting a diver tell a robotic dive companion not only where to look but what to do there, whether that means picking up trash, photographing an artifact, or investigating a suspicious structure. In the silent, dim world beneath the waves, a simple outstretched arm may soon be all the instruction a robot needs.