AI Learns a Patient’s Anatomy in Five Minutes to Align X-rays with 3D Scans

Surgeons and interventional radiologists rely on a deceptively simple question every time they guide a needle, a screw, or a catheter through the body: given a live X-ray image on the monitor and a detailed three-dimensional scan taken before the operation, exactly where is the patient’s anatomy in relation to the imaging beam? Answering that question — a computational problem known as 2D/3D registration — has long been a bottleneck for image-guided surgery and surgical robotics. A team led by researchers at the Massachusetts Institute of Technology and Harvard Medical School now reports in Nature a framework called xvr that solves it with unprecedented speed, generality, and accuracy, adapting a neural network to any individual patient in roughly five minutes and then registering X-rays to their three-dimensional scans in seconds.

The technical challenge is fundamentally one of bridging two very different kinds of images. Before an operation, patients typically receive a computed tomography (CT) or magnetic resonance imaging (MR) scan: a rich three-dimensional volume showing bones, vessels, and soft tissue in exquisite detail. During the operation, however, clinicians work with two-dimensional X-ray fluoroscopy — fast, low-dose, live images that look nothing like the slices of a CT volume. Registration is the task of finding the precise six-degree-of-freedom pose of the imaging system, essentially the position and orientation of a virtual camera, that would make a simulated X-ray of the preoperative scan match the real X-ray on the table. Get that pose right, and the surgeon can overlay the patient’s three-dimensional anatomy onto the live fluoroscopic view, guiding instruments with confidence.

Existing approaches have split into two camps, each with crippling limitations. Intensity-based optimization methods simulate X-rays from the 3D volume at candidate poses and iteratively adjust the pose until the simulated image best matches the real one. These methods are flexible in principle, but their similarity metrics and optimizer settings must be painstakingly tuned for each individual case, and the authors note that this per-individual hyperparameter tuning has prevented them from generalizing across the broad spectrum of fluoroscopy-guided procedures. Deep-learning approaches, meanwhile, promise fast inference but demand enormous manually labelled training datasets — pairs of X-rays and ground-truth poses that are expensive and often impossible to collect — and the resulting networks remain confined to the specific anatomy on which they were trained. A network trained on pelvic X-rays cannot simply be pointed at a spine or a skull.

The xvr framework, developed by Vivek Gopalakrishnan, Neel Dey, Polina Golland and colleagues across MIT, Harvard Medical School, Massachusetts General Hospital, Saint Luke’s Marion Bloch Neuroscience Institute, and Boston Children’s Hospital, sidesteps both traps with a self-supervised strategy. The key insight is that a patient’s own preoperative scan can serve as an unlimited source of training data. Using physics-based, differentiable X-ray rendering — implemented through ray-tracing algorithms including trilinear interpolation and Siddon’s exact radiological path method, with attenuation governed by the Beer–Lambert law — the system synthesizes realistic X-ray images from the patient’s CT or MR volume at randomly chosen virtual camera poses. Because the renderer is differentiable with respect to the camera pose, gradients can flow from an image-similarity loss all the way back to the pose parameters, enabling gradient-based optimization of registration. No manual annotation of the real X-rays is ever required.

On top of this self-supervised recipe, the team built a foundation model: a single neural network pretrained on thousands of whole-body volumetric scans drawn from public datasets including TotalSegmentator, CTPelvic1K, an MR angiography atlas, and whole-body PET/CT data. During pretraining, the network learns a general competence in the inverse problem of pose estimation — inferring from a synthetic X-ray where the virtual camera must have been. The crucial trick is patient-specific fine-tuning. When a new patient arrives, xvr renders synthetic X-rays from that patient’s scan, applies heavy data augmentation such as Gaussian noise, local contrast enhancement, random masking, and simulated collimation to mimic real fluoroscopy, and adapts the foundation model to this specific anatomy in about five minutes. The network thereby inherits broad anatomical knowledge from pretraining while becoming a bespoke expert on the individual on the operating table.

Handling the bewildering variety of real clinical imaging hardware demanded further engineering. X-ray machines differ in detector size, pixel spacing, principal point, and source-to-detector distance, and a naive network would falter when these intrinsic parameters change. xvr solves this with a resampling strategy that warps any incoming X-ray to a set of canonical intrinsics matching the network’s training data, so the same adapted network works regardless of the machine. Rotation representations — a notorious source of instability in pose regression — are handled with care, drawing on recent results on continuous rotation parameterizations. After the network produces its initial pose guess, a refinement stage applies iterative gradient-based optimization, blending two similarity measures: multiscale normalized cross-correlation, whose loss landscape is smooth and forgiving far from the true pose, and gradient normalized cross-correlation, which pinpoints the optimum sharply but only nearby. Averaging the two achieves millimetre-accurate registration, and the refinement corrects the network’s weakest axis — depth, or source-to-object distance, where initial errors of around fifteen millimetres are squeezed down within one to ten seconds.

The scale of the evaluation sets the work apart. According to the authors, this is, to their knowledge, the largest assessment of 2D/3D registration on real fluoroscopy conducted to date, spanning diverse anatomical structures, volumetric imaging modalities, and multiple hospitals. Public benchmarks included DeepFluoro, a femur dataset, and the Ljubljana cerebral angiography gold standard, while additional clinical evaluation drew on CTA and digital subtraction angiography data from Brigham and Women’s-affiliated collections and MRA/DSA data from Boston Children’s Hospital, governed by institutional review board approvals. Across this breadth, xvr achieved high accuracy in seconds and improved on the accuracy of existing methods by an order of magnitude — a striking margin in a field where incremental gains are the norm. Error metrics reported as median target registration error and submillimetre success rates showed the system ranking first across the benchmark datasets.

The clinical implications reach well beyond a single procedure. Image-guided interventions and surgical robotics — from spine surgery, where wrong-level errors remain a documented patient-safety concern, to orthopaedic trauma reconstruction, to neurovascular procedures such as treating vein of Galen malformations in children — all depend on knowing where preoperative three-dimensional anatomy sits relative to the live X-ray beam. Radiation oncology, telerobotic and magnetic-navigation neurovascular interventions, and intraoperative device reconstruction similarly stand to benefit. Because xvr performs rigid registration pan-anatomically, a single tool could serve orthopaedic surgeons, neurosurgeons, interventional radiologists, and radiation therapists alike, and its differentiable rendering engine opens a path toward the harder problem of deformable registration, in which anatomy moves and deforms during the procedure.

Equally significant is the decision to release everything openly. The Python package and command-line interface for xvr, built on the team’s DiffDRR library and PyTorch, are available on GitHub along with scripts to replicate every experiment in the paper. The team has also remixed the public benchmark datasets into standardized formats and released them to the community through a public repository, with permission from the original dataset authors. In a field where reproducibility has been a persistent headache and where proprietary, anatomy-specific systems dominate commercial deployments, an open-source, anatomy-agnostic, annotation-free registration engine could lower the barrier to entry for research groups and hospitals worldwide.

Caveats remain, as they do for any technology crossing from bench to bedside. The framework addresses rigid registration — treating the anatomy as a fixed body — whereas breathing, soft-tissue deformation, and surgical manipulation introduce motion that rigid models cannot capture, though the group’s associated work on polyrigid deformable registration suggests this frontier is next. Clinical datasets used for real-world evaluation cannot be released publicly for privacy reasons, and prospective validation inside live surgical workflows will be the decisive test. Still, the combination of five-minute patient adaptation, second-scale registration, order-of-magnitude accuracy gains across anatomy and hospitals, and fully open software marks a genuine inflection point. Registration, long the quiet obstacle between preoperative planning and intraoperative guidance, may finally be becoming a solved problem — and with it, the vision of routinely available, fluoroscopy-anchored surgical navigation moves measurably closer to the operating theatre.

Subject of Research: Self-supervised patient-specific neural networks for 2D/3D X-ray to volumetric image registration in image-guided interventions

Article Title: Rapid patient-specific neural networks for X-ray to volume registration

Article References: Gopalakrishnan, V., Chlorogiannis, D.-D., Abumoussa, A., Larson, A. M., Haouchine, N., Orbach, D. B., Frisken, S., Dey, N., & Golland, P. (2026). Rapid patient-specific neural networks for X-ray to volume registration. Nature. https://doi.org/10.1038/s41586-026-11045-x

Image Credits: AI Generated

DOI: 10.1038/s41586-026-11045-x

Keywords: 2D/3D registration, X-ray fluoroscopy, neural networks, image-guided surgery, self-supervised learning, foundation model, differentiable rendering, surgical navigation, medical imaging, patient-specific adaptation, interventional radiology, open-source software