The research, led by Yawen Liu and Wei Tang with colleagues Zhen Zhang and Xinrong Chen, addresses a persistent bottleneck in robotic manipulation. Conventional approaches to force sensing rely on physical instruments such as force and torque sensors or tactile arrays mounted directly on the gripper. Rigid-body models calculate grasping force through explicit mechanical equations, while soft-body models attempt to capture the nonlinear deformations of compliant structures. Although advances in materials science, including conductive hydrogels that endow soft robots with human-like sensory capabilities, have pushed the field forward, physical sensors bring inherent drawbacks: they add system complexity and cost, and their measurement range is limited, making it difficult to adapt to the diverse and dynamic demands of different grasping tasks.

Vision-based pressure estimation has emerged as an elegant alternative. By pointing cameras at the interaction between a gripper and an object, computer vision techniques can infer the magnitude and distribution of applied forces without any physical contact with the measured surface. This eliminates the need for structural modifications to the gripper and offers flexible, scalable perception. Yet most existing methods, including the current state-of-the-art system known as HiVPE, rely on single RGB images. That restriction means the network sees only surface appearance, missing explicit three-dimensional geometric deformation and high-level semantic context, both of which become crucial when dealing with severe occlusions or the complex hyperelastic behavior of soft materials.

The new framework breaks free of that single-modality constraint by weaving together four distinct streams of information. The first is the ordinary RGB image capturing the interaction between the gripper and a high-resolution pressure sensing array. The second is a binary segmentation mask that precisely delineates the gripper’s contact regions, generated by a pre-trained deep segmentation model that isolates the hand from the background. The third is a depth map produced from the RGB image by a monocular depth estimation network, encoding the geometric and spatial relationships of the contact. The fourth is perhaps the most surprising: a short textual prompt, either the word Straight or Roll, describing the gripper’s action, which is converted into a semantic embedding by the CLIP text encoder, the same contrastive language-image model that underpins many modern multimodal AI systems.

Architecturally, the system proceeds in three stages: feature extraction, feature fusion, and decoding. During extraction, a dual-branch hierarchical encoding scheme employs two parallel SE-ResNeXt50 networks. The first processes the raw RGB image to produce five feature maps at progressively abstract scales, spanning downsampling factors from 2 to 32 and channel dimensions from 64 to 2048, capturing everything from low-level textures to high-level semantics. The second encoder jointly processes the concatenated segmentation and depth inputs, yielding a parallel set of features rich in structural and geometric cues. In this way, one branch learns what the scene looks like while the other learns how it is shaped in space.

The fusion stage is where the framework’s distinctive machinery comes into play. Multi-head self-attention, with four attention heads of 64 dimensions each operating on 256-dimensional features, first refines the highest-resolution feature maps within each modality, strengthening intra-modal contextual dependencies through residual connections and feed-forward networks. A bidirectional cross-attention operator then lets the two streams talk to each other: the appearance features absorb structural context from the segmentation and depth branch, while the structural features incorporate appearance cues from the RGB branch. Finally, the 512-dimensional textual embedding is projected through a multilayer perceptron into a 2880-dimensional feature space, reshaped to align spatially with the visual features and concatenated to form a unified multimodal representation. A Feature Pyramid Network decoder then performs top-down aggregation, fusing high-level semantic cues with low-level spatial details before a segmentation head produces the final dense pressure prediction.

Training and evaluation were carried out on a publicly available benchmark containing roughly 650,000 synchronized RGB-pressure frames from two very different soft grippers, one tendon-actuated and one pneumatic. Ground-truth pressure annotations came from a high-resolution Sensel Morph pressure sensing array, spatially aligned with the camera images through calibration and homography transformation. The dataset covers four interaction types: contact, slide, close, and no-contact, the last serving as adversarial samples. The authors discretized continuous pressure values into nine logarithmically spaced categories, from zero background to a top bin of 64 kilopascals and above, and optimized the network with a pixel-wise weighted cross-entropy loss that penalizes underrepresented contact classes to counteract the imbalance between vast non-contact regions and sparse pressure-bearing areas. All results were averaged over six independent runs to verify statistical stability.

The results are striking for the tendon-actuated gripper, where the multimodal method surpassed the previous state of the art on every metric. It achieved a temporal accuracy of 96.57 percent, a contact area intersection-over-union of 77.20 percent, a volumetric IoU of 62.87 percent, and a mean absolute error of just 4.33 pascals, compared with 5.0 pascals for HiVPE and 5.3 for the earlier VPEC-Net. For the pneumatic gripper, the method attained the highest temporal accuracy and volumetric IoU, though it fell marginally short of HiVPE on contact IoU and mean absolute error, a shortfall the authors attribute to multimodal fusion occasionally introducing noise when soft surfaces deform so severely that depth and segmentation maps become blurred. Ablation experiments confirmed that each component earns its place: segmentation masks alone improved performance, adding attention strengthened contact-region predictions, depth maps reduced error further, and textual prompts delivered the final gains in spatial-pressure consistency.

The implications extend well beyond a leaderboard. Because the approach requires no embedded sensors, it could make force-aware grasping accessible to cheap, commodity robotic hardware, from laboratory manipulators to warehouse pickers handling fragile goods. The demonstration that a single descriptive word about the gripper’s action can measurably sharpen pressure predictions hints at a future where robots are instructed in natural language and perceive the physical consequences of their movements through vision alone. The authors are candid about the current limitations: the framework has so far been validated on static, offline benchmark data rather than deployed on physical robots in dynamic manipulation, where real-time sensor noise, unpredictable physical interactions, and inference latency constraints all come into play. They also note that soft grippers, with their complex nonlinear hyperelastic deformation, remain the hardest case, and they point toward more advanced depth estimation, adaptive fusion architectures that dynamically re-weight unreliable modalities, and explicit deformation modeling as the next frontiers. If those challenges are met, the line between seeing and touching may grow ever thinner, bringing robots one step closer to hands that truly understand what they hold.

Subject of Research: Vision-based multimodal pressure estimation for robotic gripper manipulators

Article Title: Vision-Based multimodal pressure estimation for robotic gripper manipulators

Article References: Liu, Y., Tang, W., Zhang, Z., & Chen, X. (2026). Vision-Based multimodal pressure estimation for robotic gripper manipulators. Results in Engineering, 32, Article 113295. https://doi.org/10.1016/j.rineng.2026.113295

Image Credits: AI Generated

DOI: 10.1016/j.rineng.2026.113295

Keywords: robotics, pressure estimation, multimodal AI, soft grippers, computer vision, deep learning, attention mechanism, CLIP, tactile sensing, depth estimation, feature pyramid network, robotic manipulation

A team of researchers at the University of Leeds, led by Lucy Fothergill and Duygu Sarikaya of the School of Computer Science, together with Pietro Valdastri and Dominic Jones from the School of Electronic and Electrical Engineering, has now unveiled a vision-based alternative designed to close that gap. Their system, called PICO for Projection-Informed Consistency Optimisation, estimates the full 6DoF pose of a surgical tool directly from a single monocular RGB image, with no markers, no trackers and no external hardware attached to the instrument. Published in the International Journal of Computer Assisted Radiology and Surgery, the work demonstrates that a single, end-to-end trainable neural network can rival far more cumbersome multi-stage pipelines while running at true real-time speed, and it holds up even when blood, smoke or other instruments obscure the camera’s view.

The case for markerless vision is straightforward once the constraints of the operating room are understood. External markers and trackers demand sterilisation procedures that interrupt surgical workflow, and they require an unobstructed line of sight between camera and marker, something a surgical field full of tissue, fluid and smoke simply cannot guarantee. Yet most existing markerless approaches dodge the hardest part of the problem. Rather than predicting the 6DoF pose outright, they first extract intermediate representations, such as 2D keypoints, segmentation masks or dense 2D-3D correspondences, and only then compute the pose using a Perspective-n-Point solver, template matching, template-based rendering or iterative refinement. Each extra stage is another place where noise accumulates, another source of computational delay, and another dependency that can fail. If the initial estimate is poor, the refinement step may never fully correct it, and iterative render-and-compare loops are notoriously slow at inference time, degrading performance in the fast-changing environment of a live operation.

PICO’s architecture attacks the problem from a different direction. Given a cropped monocular RGB image, a shared encoder based on ResNet-50, a widely used deep residual network pre-trained on ImageNet, extracts visual features. Those features feed two parallel paths. One path runs through a U-Net style decoder, the workhorse of biomedical image segmentation, into two task-specific heads: a segmentation head that produces a binary mask of the tool, and a depth head that outputs a pseudo-depth map of the scene. The other path bypasses the decoder, passing the encoder features through fully connected layers into regression heads that directly output the tool’s rotation and translation. Rotation is regressed as a full 3×3 matrix, which is projected onto the nearest valid rotation in the mathematical group SO(3) using Singular Value Decomposition, a parameterisation the authors chose to avoid the singularities and ambiguities that plague Euler angles and quaternions. Translation is split into the (x, y) pixel coordinates of the tool joint in the image frame and a separate depth value z relative to the camera, each with its own activation function tuned to the range of plausible values.

The genuinely novel ingredient is the pair of geometric consistency losses, or proxy tasks, that bind these outputs together. The projection loss takes the network’s predicted pose, applies it to a randomly sampled set of 3D points on the tool’s mesh model, and projects those points into the 2D image plane using the camera’s intrinsics. The contour of the resulting concave hull yields a projected binary mask, which acts as a pseudo-ground-truth against which the predicted segmentation mask is scored with a Dice loss. If the network’s pose and its segmentation disagree, the loss rises, forcing the model to keep its 2D appearance and its 3D geometry in register. The point-to-point loss works entirely in 3D: the predicted and ground-truth pose transformations are applied to the same sampled model points, and the root mean squared error between corresponding points is minimised. Together with a geodesic loss that measures the true angular distance between rotation matrices, and separate root-mean-squared-error terms for the (x, y) and z translation components, the full multi-task objective supervises the network simultaneously in 2D image space and 3D model space.

The auxiliary tasks proved to have complementary but delicate roles. In ablation studies across four test datasets, adding depth supervision alone yielded the lowest translation errors on three of the four sets, for example cutting error from 13.28 to 8.36 millimetres on the occluded large needle driver set, because it primarily constrains the depth component of the pose. Segmentation supervision, by contrast, sharpened rotation estimates by encoding the tool’s projected shape, improving rotation error from 18.89 to 11.11 degrees on the large needle driver and from 12.95 to 9.67 degrees on the Maryland bipolar forceps. Curiously, combining the two naively did not stack the gains and sometimes hurt performance, but the projection and point-to-point losses, which couple both signals through a single predicted transformation, resolved the trade-off and produced the best overall results.

Benchmarked on the SurgRIPE dataset, introduced at the MICCAI 2022 SurgRIPE challenge and still the only public benchmark with ground-truth 6DoF pose annotations for surgical instruments, PICO ranked second in rotational accuracy across all four test sets, covering two tools, the large needle driver and the Maryland bipolar forceps, each in occluded and unoccluded conditions. It recorded rotation errors of 5.78 degrees on the large needle driver and 21.02 degrees on the occluded forceps, trailing only the top-performing multi-stage entry from ImFusion, which combined SurfEmb surface embeddings with an iterative render-and-compare refinement. PICO’s translational performance remained competitive, particularly under occlusion, where its 8.87-millimetre error on the occluded needle driver far outperformed the 28.09 millimetres of PVNet, a widely cited two-stage method. Against the only other single-stage method in the benchmark, PICO cut rotation errors dramatically, from 27.21 to 5.78 degrees on the large needle driver and from 34.13 to 21.02 degrees on the occluded forceps.

Perhaps the most impressive figure is the runtime. On a consumer-grade NVIDIA Tesla T4 GPU, PICO completes inference at 33.3 frames per second, roughly 30 milliseconds per image, clearing the customary 30 FPS threshold for real-time performance. PVNet, by comparison, manages about 25 FPS even on a GTX 1080ti. At inference time the U-Net decoder is simply discarded, stripping away computational overhead, while a fine-tuned YOLOv5 detector locates the tool in the image, achieving intersection-over-union scores as high as 0.89. The combination of single-step prediction and geometric supervision means no iterative refinement, no correspondence solving and no render-and-compare loop stand between the camera image and the pose estimate.

The authors were unusually candid about the method’s remaining weakness. PICO scored lowest of all benchmarked methods on the ADD metric, which measures the fraction of samples whose average distance between ground-truth and predicted point clouds falls below 10 percent of the instrument’s diameter. An error decomposition revealed why: depth error along the camera axis, with mean values as high as 11.82 millimetres on the occluded forceps set, tracks ADD distance almost perfectly, with Spearman correlations of 0.955 to 0.982, dwarfing the correlations of image-plane error. In other words, the pseudo-depth maps generated by sampling mesh points and assigning them to the nearest pixel, then filling holes with neighbourhood averages, remain too coarse to supervise depth precisely. The team also acknowledges that resizing non-square crops to a fixed 224-pixel square can introduce mild aspect-ratio distortion, and that the scarcity of public surgical pose datasets restricts comparisons to benchmark-reported figures rather than fully reproducible implementations.

Even so, the trajectory of the work is hard to ignore. Accurate, markerless, real-time 6DoF tool pose estimation is a prerequisite for surgical autonomy, robotic proprioception and safe tissue interaction, and PICO demonstrates that an end-to-end network guided by geometric consistency can match multi-stage pipelines without their latency or fragility. The Leeds group’s next steps, refining depth modelling and extending generalisation to unseen instruments and environments through domain adaptation and data augmentation, will determine how quickly this kind of software settles into the operating theatre. But the central message already stands: a neural network that forces its own 2D and 3D views of the world to agree can track a robot’s instruments with the speed and reliability that surgical autonomy demands.

Subject of Research: Real-time markerless 6DoF pose estimation of surgical instruments from monocular images using multi-task learning and geometric consistency losses

Article Title: PICO: Projection-Informed Consistency Optimisation for 6DoF surgical tool pose estimation

Article References: Fothergill, L., Valdastri, P., Jones, D., & Sarikaya, D. (2026). PICO: Projection-Informed Consistency Optimisation for 6DoF surgical tool pose estimation. International Journal of Computer Assisted Radiology and Surgery. https://doi.org/10.1007/s11548-026-03802-0

Image Credits: AI Generated

DOI: 10.1007/s11548-026-03802-0

Keywords: surgical robotics, 6DoF pose estimation, PICO, multi-task learning, geometric consistency, SurgRIPE benchmark, markerless tracking, depth estimation, computer vision, da Vinci, real-time inference, occlusion robustness

The study, led by Tianjun Yang, Xingfeng Xu and Siyang Zuo of Tianjin University together with gastroenterologist Xin Chen of Tianjin Medical University General Hospital, approaches the problem as an exercise in video inpainting: the task of filling in corrupted regions of an image sequence with plausible content drawn from the surrounding frames. Rather than treating each frame in isolation, as many earlier reflection-removal systems did, Lumina-Net exploits the fact that endoscopic video is inherently temporal. As the endoscope moves, the same patch of mucosa is viewed from slightly different angles at different moments, so information hidden behind a glare in one frame is often cleanly visible in its neighbors. The heart of the framework is a spatiotemporal Transformer that uses overlapping tokens, meaning small blocks of visual features that share context across both space and time. By allowing these tokens to aggregate complementary information from consecutive frames, the network can reconstruct the true tissue appearance beneath a reflection instead of simply painting over it with a generic texture.

Two lightweight modules in the decoder distinguish Lumina-Net from its predecessors. The first, Variance-Guided Feature Modulation, or VGFM, tackles a subtle but important problem: specular highlights in endoscopic imagery come in mixed scales, from tiny pinpoints of glare to large saturated blooms that cover a substantial fraction of the field of view. VGFM recalibrates network features using channel statistics, essentially measuring the variance of responses along each feature channel and using that measurement to decide how strongly to amplify or suppress them. This statistical steering allows a single network to handle both fine-grained and coarse-scale artifacts without resorting to separate models or heavy per-scale processing, keeping the added computational cost to a minimum.

The second module, Multi-Scale Energy Free-Space Attention, abbreviated MS-EFSA, is notable for carrying no learned parameters at all. Instead of adding weights that must be trained, it derives spatial attention weights directly from the energy of the features themselves. In practical terms, regions of the image that carry strong, reliable information about tissue structure receive more attention, while ambiguous or corrupted regions are down-weighted. The authors designed this mechanism with one anatomical priority in mind: preserving mucosal structures, the fine vascular and fold patterns of the gastrointestinal lining that clinicians rely on for diagnosis. Attention schemes that merely chase photometric consistency can smooth away exactly these clinically meaningful details. By anchoring attention to feature energy across multiple scales, MS-EFSA encourages the inpainted result to remain faithful to the underlying anatomy rather than producing a visually plausible but structurally hollow reconstruction.

The technical claims were tested on two public benchmarks: HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy, and GastroHUN, a dataset covering a complete systematic screening protocol for the stomach. Quantitatively, Lumina-Net achieved a peak signal-to-noise ratio of 30.20 decibels on these datasets and reduced mean squared error by approximately 5.3 percent compared with the strongest baseline method. While such numbers may sound incremental, in the tightly contested field of image restoration a 5 percent error reduction over a state-of-the-art competitor is meaningful, and PSNR above 30 decibels in a challenging medical domain indicates a reconstruction quality that is difficult to achieve. More striking is the speed: the network runs at 27 frames per second on a single graphics processing unit, placing it at the threshold of real-time video processing and making deployment alongside a live endoscopy workflow a realistic prospect rather than a laboratory aspiration.

Numbers alone, however, do not decide whether a restoration method is fit for clinical use, and the team supplemented the quantitative evaluation with a blinded study in which clinical experts compared the visual quality of outputs from Lumina-Net and competing approaches without knowing which method produced which result. The experts consistently preferred the proposed method, a finding the authors supported with established statistical procedures for ranked comparisons, including Wilcoxon-style rank testing and Kendall’s coefficient of concordance, which measures agreement among multiple raters. This convergence of expert judgment with objective metrics strengthens the case that the improvements are perceptually relevant, not merely artifacts of a particular error function.

Perhaps the most consequential demonstration concerns what happens downstream of the cleaned video. Modern endoscopy research increasingly depends on monocular depth estimation, the task of inferring three-dimensional scene structure from a single camera, which underpins applications such as autonomous scope navigation and robotic-assisted intervention. Glare and flicker are poison for these algorithms, because depth networks learn from photometric consistency between frames and reflections violate the assumptions that make that consistency meaningful. When the researchers applied depth estimation models to sequences processed by Lumina-Net, the resulting depth predictions improved, providing concrete evidence that reflection removal is not just cosmetic but functionally enables the robotic and navigational systems now under development in surgical laboratories worldwide.

The work fits into a research lineage stretching back nearly two decades, from early hand-crafted methods for detecting and inpainting specular highlights, through generative adversarial networks trained to synthesize glare-free tissue, to more recent temporal learning approaches and depth-aware endoscopic video inpainting presented at venues such as MICCAI. What Lumina-Net adds to this progression is a combination of architectural economy and clinical grounding. Its Transformer core borrows from general-purpose video inpainting frameworks such as FuseFormer and joint spatial-temporal transformation networks, but the VGFM and MS-EFSA modules are engineered specifically for the statistics of endoscopic imagery. The collaboration between engineering and clinical departments, funded by the National Natural Science Foundation of China under grant number 62133010, reflects a broader trend in which computer vision researchers and practicing gastroenterologists co-design tools around the actual failure modes of the imaging chain rather than abstract benchmarks.

The practical implications extend well beyond cleaner videos for human viewing. Reliable, glare-free endoscopic video is a prerequisite for the next generation of computer-assisted interventions: self-navigating capsule endoscopes that must map the stomach, surgical robots that need accurate tissue geometry, and diagnostic support systems that flag subtle early-stage lesions before they become advanced cancers. Because Lumina-Net operates at near-real-time speed and its complete code and pretrained weights are slated for release on GitHub upon acceptance, the barrier to integrating it into these pipelines is low. The study relied exclusively on publicly available, anonymized datasets, requiring no new ethics approval, and the authors report no conflicts of interest.

Caveats remain, as they always do with deep learning in medicine. The method was validated on two datasets and its generalization to unusual patient populations, atypical lighting hardware, or pathological tissue with markedly different reflectance properties will require further study. And like all generative restoration systems, inpainting networks must be used with care in diagnostic contexts, since any reconstructed pixel is by definition inferred rather than observed. Still, the combination of statistical rigor, expert validation and demonstrated downstream utility marks Lumina-Net as a serious step toward endoscopic video that is as clean as the underlying anatomy deserves. If the glare can be removed as reliably as this work suggests, both the eyes of the endoscopist and the algorithms of the robotic future may finally see the digestive tract clearly.

Subject of Research: Deep learning-based specular reflection removal in gastrointestinal endoscopy video using temporal learning and feature fusion.

Article Title: Lumina-Net: temporal learning with feature fusion for endoscopic artifact removal

Article References: Lumina-Net: temporal learning with feature fusion for endoscopic artifact removal. (n.d.). https://doi.org/10.1007/s11517-026-03674-1

Image Credits: AI Generated

DOI: 10.1007/s11517-026-03674-1

Keywords: Gastrointestinal endoscopy, Specular reflection removal, Video inpainting, Temporal learning, Transformer, Feature fusion, Depth estimation, HyperKvasir, GastroHUN, Robotic-assisted intervention, Medical image computing, Real-time processing