New temporal distillation module sharpens surgical gesture segmentation

New AI Module Pinpoints the Hidden Seams Between Surgical Gestures

Every operation is a river of gestures — grasping, cutting, suturing, cauterizing — that pour into one another with almost no visible seams. For artificial intelligence, those seams are exactly where machine vision loses the thread. A team of researchers from Inria and Université Côte d’Azur, working with clinicians at CHU Nice and engineers at the medical robotics company Caranx Medical, has now built a module that teaches video models to sense where one surgical gesture ends and the next begins. The technique, called the temporal boundary distillation module, or TBDM, is presented in a study published on 28 August 2026 in the International Journal of Computer Assisted Radiology and Surgery. Across two of the most widely used surgical video benchmarks, the module lifted the performance of existing AI models by margins large enough to set a new state of the art — and it did so using nothing more than ordinary color video from the operating room camera.

Understanding what a surgeon is doing, second by second, has become one of the central ambitions of computer-assisted intervention. Systems that can recognize surgical gestures in real time could warn a robot to adjust its grip, flag a trainee’s hesitation for objective skills assessment, generate operative reports automatically, or hand context-aware support to the surgical team at exactly the right moment. The task is known as surgical gesture segmentation, and it is far harder than it sounds. Coarse steps of an operation, such as the phases of a gallbladder removal, unfold over minutes and are relatively easy to distinguish. Gestures, by contrast, last only seconds, involve subtle hand and instrument motions, and frequently overlap in time as the surgeon’s two hands pursue different goals simultaneously. Fine-grained motion understanding at this scale remains a fundamental challenge for computer vision, and it is precisely at the transitions between gestures that today’s algorithms stumble most often. A model that confuses them produces an operative timeline that neither a reviewer nor a robot can safely trust.

The obstacle is partly physical and partly statistical. Surgical actions do not switch on and off like light bulbs; one gesture dissolves gradually into the next as instruments advance, retract and reposition inside a crowded, reflective field of view. Many leading systems attack the problem by fusing two data streams: the RGB video captured by the endoscope and the kinematic data exported by the robotic system, which records the precise positions and velocities of each instrument arm. That fusion strategy works well in heavily instrumented robotic operating rooms, but it breaks down wherever kinematic logs are unavailable, such as in conventional laparoscopy or archived footage, and it leaves untouched the deeper problem that gesture boundaries are intrinsically ambiguous. When a segmentation model blurs a transition, it fragments a single long action into a stutter of short ones, and its scores collapse on the edit score, a standard metric computed with the Levenshtein edit distance that tolerates small timing offsets but punishes such fragmentation severely.

Computer-vision researchers have long recognized that boundaries deserve special treatment. In general video understanding, architectures such as boundary-sensitive and boundary-matching networks emerged specifically to propose and refine the edges of actions, and modern action-segmentation models built on temporal convolutions and transformers have grown steadily better at delineating activities in cooking, assembly and sports footage. Surgical video, however, is a harsher arena. The camera drifts, smoke from electrocautery occludes the field, tissue deforms unpredictably, and the tool tips responsible for each gesture occupy only a handful of pixels. Several recent systems have therefore leaned on multimodal fusion, pairing video with robot kinematics or adding attention mechanisms that weigh multiple cues. Each added modality, though, brings cost, complexity and a dependence on hardware that most of the world’s operating rooms do not have. The Inria-led group set itself the opposite constraint: squeeze better boundary understanding out of RGB video alone, and do it without slowing down the model that must ultimately run inside the operating room.

The team’s answer is to stop treating boundaries as incidental and to model them explicitly during training — using information that will never be available at test time. Their approach belongs to a family of machine-learning strategies known as learning using privileged information, in which a model is granted extra knowledge while it studies so that it emerges smarter without carrying extra baggage into deployment. During training, the researchers partition each annotated surgical video into temporal regions where a given gesture class is present and regions where it is absent, drawing directly on ground-truth annotations. A dedicated module, the temporal boundary distillation module, then applies cross-attention — the same attention operation that powers modern language models, allowing positions in the video stream to query information from contrasting regions — between these class-present and class-absent temporal regions. By forcing the network to compare stretches of video that contain a gesture with stretches that do not, the module compels the model’s internal features to encode what the moment of transition actually looks like: the in-between evidence that ordinary classifiers gloss over.

The clever part is what happens next. Rather than leaving the heavyweight boundary-aware module bolted onto the network forever, the researchers use it as a teacher. A lightweight projection layer — a small, cheap set of transformations — sits on top of the main backbone and learns to reproduce the boundary-aware features through knowledge distillation, guided by two signals at once: a standard classification loss that keeps its predictions accurate, and a mean-squared-error distillation loss that pushes its internal representations toward the teacher’s transition-sensitive ones. Because the student mimics the teacher’s feature geometry rather than merely copying its predictions, boundary sensitivity is absorbed into the representations themselves instead of being bolted on as a separate detection head. Once training is complete, the teacher is discarded entirely. At inference time, the system runs only the original backbone plus the slim projection layer, which means the accuracy gains arrive with no additional annotations, no extra memory and no measurable computational overhead. Because the module is agnostic to the underlying architecture, it can be attached to temporal convolutional networks, transformers or other segmentation backbones as a plug-and-play upgrade.

The team put that claim to a rigorous test on two demanding benchmarks. The first, CholecT50, is a collection of fifty laparoscopic cholecystectomy videos — recordings of gallbladder removals — that has become a proving ground for fine-grained surgical action recognition. TBDM was attached to a series of baseline segmentation models spanning different architectural families, and the results were strikingly consistent: every metric improved for every baseline. The gains were not marginal refinements. On CholecT50, the module delivered improvements of up to 8.5 points in edit score, one of the most unforgiving measures of segmentation quality, indicating that models equipped with boundary-aware features stopped shredding continuous gestures into spurious fragments and began tracking the true temporal arc of each action. Precision, recall and F1 values at multiple overlap thresholds rose as well, consistent with cleaner temporal boundaries rather than lucky tuning of a single metric. Consistency across so many different architectures suggests the benefit comes from the information the module injects rather than from any quirk of a particular network design.

The second benchmark, RARP-45, raised the stakes. It comprises forty-five videos of robot-assisted radical prostatectomy, a lengthy and technically demanding cancer operation performed with robotic surgical platforms. Here the approach did not merely improve existing baselines; it set new records. The TBDM-equipped system reached a state-of-the-art edit score of 81.4 and an F1 score at the 50 percent overlap threshold, known as F1@50, of 77.9, outperforming prior methods on both measures. The edit score, computed by aligning predicted and ground-truth action sequences with the Levenshtein distance, is notoriously sensitive to boundary errors, which makes the margin especially meaningful. Achieving top results on two procedures as different as gallbladder removal and prostatectomy — and across multiple network architectures — is evidence that the framework captures something general about how surgical gestures change, rather than memorizing the visual habits of a single operation or a single camera setup.

What makes the result commercially and clinically significant is its frugality. Because TBDM operates on RGB video alone, it does not depend on kinematic streams that only robotic systems can provide, which means it can, in principle, run on footage from any standard endoscopic camera — including recordings made years ago in hospitals that never owned a robot. Deployment is equally painless: the distilled student adds essentially no compute, so a hospital server or even an edge device attached to the operating room stack could run gesture recognition in real time without slowing anything down. For surgical training programs, an objective, second-by-second record of gestures could complement today’s largely subjective rating scales. The same capability opens practical doors to automatic operative reports, post-hoc review of adverse events, and the perceptive layer that autonomous surgical assistants will eventually need in order to know, moment by moment, whether they are watching dissection, suturing or something going wrong at the seam between the two.

The study’s authors — a collaboration spanning Inria’s research labs, the university hospital of Nice and Caranx Medical — have released their code publicly on GitHub, and the underlying datasets, CholecT50 and RARP-45, are available from their creators on request, inviting other groups to reproduce and extend the results. The authors describe TBDM as a generalized, plug-and-play framework rather than a bespoke model for one operation or robot. The work was supported by the French government through the 3IA Côte d’Azur program of the National Research Agency and benefited from computing time on the Jean Zay supercomputer operated by GENCI. One honest caveat remains: the module’s boundary sensitivity is instilled during training with ground-truth annotations, so its power depends on having carefully labeled examples of each new procedure. Even so, the lesson of the study is likely to travel beyond the operating room. Wherever actions overlap and melt into one another — in sports analytics, manufacturing or human-robot collaboration — teaching models to attend to the boundary between classes, then distilling that awareness into a lightweight student, may prove the cheapest way to give machines a sense of when one story ends and the next begins.