A team of researchers at the University of Leeds, led by Lucy Fothergill and Duygu Sarikaya of the School of Computer Science, together with Pietro Valdastri and Dominic Jones from the School of Electronic and Electrical Engineering, has now unveiled a vision-based alternative designed to close that gap. Their system, called PICO for Projection-Informed Consistency Optimisation, estimates the full 6DoF pose of a surgical tool directly from a single monocular RGB image, with no markers, no trackers and no external hardware attached to the instrument. Published in the International Journal of Computer Assisted Radiology and Surgery, the work demonstrates that a single, end-to-end trainable neural network can rival far more cumbersome multi-stage pipelines while running at true real-time speed, and it holds up even when blood, smoke or other instruments obscure the camera’s view.

The case for markerless vision is straightforward once the constraints of the operating room are understood. External markers and trackers demand sterilisation procedures that interrupt surgical workflow, and they require an unobstructed line of sight between camera and marker, something a surgical field full of tissue, fluid and smoke simply cannot guarantee. Yet most existing markerless approaches dodge the hardest part of the problem. Rather than predicting the 6DoF pose outright, they first extract intermediate representations, such as 2D keypoints, segmentation masks or dense 2D-3D correspondences, and only then compute the pose using a Perspective-n-Point solver, template matching, template-based rendering or iterative refinement. Each extra stage is another place where noise accumulates, another source of computational delay, and another dependency that can fail. If the initial estimate is poor, the refinement step may never fully correct it, and iterative render-and-compare loops are notoriously slow at inference time, degrading performance in the fast-changing environment of a live operation.

PICO’s architecture attacks the problem from a different direction. Given a cropped monocular RGB image, a shared encoder based on ResNet-50, a widely used deep residual network pre-trained on ImageNet, extracts visual features. Those features feed two parallel paths. One path runs through a U-Net style decoder, the workhorse of biomedical image segmentation, into two task-specific heads: a segmentation head that produces a binary mask of the tool, and a depth head that outputs a pseudo-depth map of the scene. The other path bypasses the decoder, passing the encoder features through fully connected layers into regression heads that directly output the tool’s rotation and translation. Rotation is regressed as a full 3×3 matrix, which is projected onto the nearest valid rotation in the mathematical group SO(3) using Singular Value Decomposition, a parameterisation the authors chose to avoid the singularities and ambiguities that plague Euler angles and quaternions. Translation is split into the (x, y) pixel coordinates of the tool joint in the image frame and a separate depth value z relative to the camera, each with its own activation function tuned to the range of plausible values.

The genuinely novel ingredient is the pair of geometric consistency losses, or proxy tasks, that bind these outputs together. The projection loss takes the network’s predicted pose, applies it to a randomly sampled set of 3D points on the tool’s mesh model, and projects those points into the 2D image plane using the camera’s intrinsics. The contour of the resulting concave hull yields a projected binary mask, which acts as a pseudo-ground-truth against which the predicted segmentation mask is scored with a Dice loss. If the network’s pose and its segmentation disagree, the loss rises, forcing the model to keep its 2D appearance and its 3D geometry in register. The point-to-point loss works entirely in 3D: the predicted and ground-truth pose transformations are applied to the same sampled model points, and the root mean squared error between corresponding points is minimised. Together with a geodesic loss that measures the true angular distance between rotation matrices, and separate root-mean-squared-error terms for the (x, y) and z translation components, the full multi-task objective supervises the network simultaneously in 2D image space and 3D model space.

The auxiliary tasks proved to have complementary but delicate roles. In ablation studies across four test datasets, adding depth supervision alone yielded the lowest translation errors on three of the four sets, for example cutting error from 13.28 to 8.36 millimetres on the occluded large needle driver set, because it primarily constrains the depth component of the pose. Segmentation supervision, by contrast, sharpened rotation estimates by encoding the tool’s projected shape, improving rotation error from 18.89 to 11.11 degrees on the large needle driver and from 12.95 to 9.67 degrees on the Maryland bipolar forceps. Curiously, combining the two naively did not stack the gains and sometimes hurt performance, but the projection and point-to-point losses, which couple both signals through a single predicted transformation, resolved the trade-off and produced the best overall results.

Benchmarked on the SurgRIPE dataset, introduced at the MICCAI 2022 SurgRIPE challenge and still the only public benchmark with ground-truth 6DoF pose annotations for surgical instruments, PICO ranked second in rotational accuracy across all four test sets, covering two tools, the large needle driver and the Maryland bipolar forceps, each in occluded and unoccluded conditions. It recorded rotation errors of 5.78 degrees on the large needle driver and 21.02 degrees on the occluded forceps, trailing only the top-performing multi-stage entry from ImFusion, which combined SurfEmb surface embeddings with an iterative render-and-compare refinement. PICO’s translational performance remained competitive, particularly under occlusion, where its 8.87-millimetre error on the occluded needle driver far outperformed the 28.09 millimetres of PVNet, a widely cited two-stage method. Against the only other single-stage method in the benchmark, PICO cut rotation errors dramatically, from 27.21 to 5.78 degrees on the large needle driver and from 34.13 to 21.02 degrees on the occluded forceps.

Perhaps the most impressive figure is the runtime. On a consumer-grade NVIDIA Tesla T4 GPU, PICO completes inference at 33.3 frames per second, roughly 30 milliseconds per image, clearing the customary 30 FPS threshold for real-time performance. PVNet, by comparison, manages about 25 FPS even on a GTX 1080ti. At inference time the U-Net decoder is simply discarded, stripping away computational overhead, while a fine-tuned YOLOv5 detector locates the tool in the image, achieving intersection-over-union scores as high as 0.89. The combination of single-step prediction and geometric supervision means no iterative refinement, no correspondence solving and no render-and-compare loop stand between the camera image and the pose estimate.

The authors were unusually candid about the method’s remaining weakness. PICO scored lowest of all benchmarked methods on the ADD metric, which measures the fraction of samples whose average distance between ground-truth and predicted point clouds falls below 10 percent of the instrument’s diameter. An error decomposition revealed why: depth error along the camera axis, with mean values as high as 11.82 millimetres on the occluded forceps set, tracks ADD distance almost perfectly, with Spearman correlations of 0.955 to 0.982, dwarfing the correlations of image-plane error. In other words, the pseudo-depth maps generated by sampling mesh points and assigning them to the nearest pixel, then filling holes with neighbourhood averages, remain too coarse to supervise depth precisely. The team also acknowledges that resizing non-square crops to a fixed 224-pixel square can introduce mild aspect-ratio distortion, and that the scarcity of public surgical pose datasets restricts comparisons to benchmark-reported figures rather than fully reproducible implementations.

Even so, the trajectory of the work is hard to ignore. Accurate, markerless, real-time 6DoF tool pose estimation is a prerequisite for surgical autonomy, robotic proprioception and safe tissue interaction, and PICO demonstrates that an end-to-end network guided by geometric consistency can match multi-stage pipelines without their latency or fragility. The Leeds group’s next steps, refining depth modelling and extending generalisation to unseen instruments and environments through domain adaptation and data augmentation, will determine how quickly this kind of software settles into the operating theatre. But the central message already stands: a neural network that forces its own 2D and 3D views of the world to agree can track a robot’s instruments with the speed and reliability that surgical autonomy demands.

Subject of Research: Real-time markerless 6DoF pose estimation of surgical instruments from monocular images using multi-task learning and geometric consistency losses

Article Title: PICO: Projection-Informed Consistency Optimisation for 6DoF surgical tool pose estimation

Article References: Fothergill, L., Valdastri, P., Jones, D., & Sarikaya, D. (2026). PICO: Projection-Informed Consistency Optimisation for 6DoF surgical tool pose estimation. International Journal of Computer Assisted Radiology and Surgery. https://doi.org/10.1007/s11548-026-03802-0

Image Credits: AI Generated

DOI: 10.1007/s11548-026-03802-0

Keywords: surgical robotics, 6DoF pose estimation, PICO, multi-task learning, geometric consistency, SurgRIPE benchmark, markerless tracking, depth estimation, computer vision, da Vinci, real-time inference, occlusion robustness