From task-specific AI to multimodal foundation models in PET/CT: a readiness framework for clinical translation

Artificial intelligence in nuclear medicine is undergoing a quiet revolution, and a new review from researchers at The University of Texas MD Anderson Cancer Center argues that PET/CT is about to become its most demanding proving ground. Writing in Frontiers in Nuclear Medicine, the team synthesizes five and a half years of literature to map how the field is moving away from narrowly optimized, single-task models toward reusable pretrained representations, promptable segmentation systems, vision-language models, and agentic workflows that orchestrate multiple tools. Their conclusion is striking: the technology is advancing faster than the evidence needed to prove it safe, and the gap between impressive benchmarks and genuine clinical benefit remains wide.

The reason PET/CT poses such a hard test is that it is not just an image. PET intensities carry quantitative and biologic meaning: standardized uptake values, metabolic tumor volume, total lesion glycolysis and kinetic parameters directly influence staging, treatment response assessment and prognosis. A deep learning model can produce a visually plausible scan or a fluent report while corrupting SUVs, suppressing small lesions, confusing physiologic and pathologic uptake, or inventing comparative findings that were never measured. Uptake depends on injected activity, uptake time, reconstruction settings, motion, attenuation and scatter correction, partial-volume effects and the physiology of the radiopharmaceutical itself. Any AI that touches the image chain must therefore be evaluated not only for image similarity but for lesion detectability, uptake bias, repeatability and clinically meaningful thresholds.

The hybrid nature of PET/CT adds another layer of complexity. PET and CT are spatially linked but semantically asymmetric: CT provides high-resolution anatomy and attenuation information, while PET reveals tracer-specific physiology at lower spatial resolution. Naive channel concatenation or independent encoders may underuse this complementarity or allow one modality to dominate. Hybrid acquisition also introduces failure modes absent in single-modality models, from respiratory motion that misregisters uptake and anatomy to metallic implants and contrast that alter CT appearance, and CT components that may be low dose or non-diagnostic. A foundation model must learn correspondence between the two streams while retaining the ability to detect and communicate misalignment.

Tracer diversity is both an opportunity and a source of domain shift. FDG dominates existing AI datasets, but nuclear medicine spans PSMA-targeted agents, somatostatin-receptor tracers, amyloid and tau agents, amino-acid tracers, cardiac perfusion tracers and an expanding theranostic ecosystem. A representation that generalizes across tracers could enable label-efficient learning and rare-indication transfer. Conversely, anatomic uptake that is abnormal for one tracer can be expected for another. Pooling tracers without explicit conditioning risks negative transfer: a model may learn that intense salivary-gland uptake is suspicious from one training context and normal from another, or conflate treatment-related and pathologic patterns. Tracer identity, injected activity, uptake interval and acquisition protocol should be explicit model inputs, and evaluation should report performance by tracer and by tracer-disease combination, including rare ones.

Despite these challenges, task-specific deep learning has already demonstrated promising technical performance. Multicenter low-count studies have shown that deep-learning enhancement can preserve diagnostic confidence and SUV agreement, opening the door to lower injected activity, shorter acquisitions and improved throughput. The autoPET challenge showed that accurate whole-body lesion segmentation is feasible, while public resources such as the FDG-PET/CT-Lesions collection of 1,014 expertly annotated studies and the HECKTOR benchmarks have accelerated progress. A semi-supervised transfer-learning model evaluated 611 FDG and 408 PSMA PET/CT scans, performing segmentation across several malignancies and deriving prognostic measures for multiple downstream endpoints. Federated learning has offered one route to multi-institutional development without centralizing images; in a six-center head-and-neck study, federated transformer models approached centralized segmentation performance. Yet most evidence remains retrospective and restricted by radiotracer, disease, scanner, institution or endpoint, and challenge performance should be interpreted as a benchmark under defined conditions, not evidence of universal clinical readiness.

The technical transition toward foundation models rests on three ideas. Self-supervised learning extracts supervisory signals from the images themselves through masking, reconstruction, contrastive learning or temporal consistency. Language provides scalable semantic supervision because reports contain anatomy, findings, comparisons and diagnostic impressions. Parameter-efficient adaptation allows a shared encoder to support new tasks with fewer labeled examples. PET-native systems began to emerge in 2025: SegAnyPET introduced a three-dimensional promptable segmentation model trained on 5,731 whole-body FDG PET volumes, achieving mean Dice similarity coefficients of 90.49 to 91.05 percent for prompted organ segmentation, though performance dropped for unseen organs and markedly for zero-shot multifocal tumor segmentation. In 2026, an open multicenter FDG PET/CT foundation model used 4,797 scans from four public datasets, with masked-autoencoder initialization improving Dice from 0.4889 to 0.6132 with only 70 labeled training scans and from 0.6414 to 0.7205 with 700. Independent non-FDG and prospective validation remain absent.

Vision-language modeling is among the most rapidly emerging frontiers. PET reports contain lesion locations, tracer avidity, SUV values, comparison with prior examinations and an integrated impression, so a model must connect words to three-dimensional, often multifocal findings. ConTEXTual Net 3D used weak labels derived from report mentions and scan coordinates to train text-guided PET/CT visual grounding, reaching an F1 of 0.80 against 0.22 to 0.53 for baseline models, although report omissions and automatically generated labels can introduce systematic bias. PET2Rep established a benchmark of 565 whole-body PET/CT-report pairs and evaluated 30 models; its best reported overall reference score was 26.8 percent, below the 27.5 percent of a fixed-template baseline, underscoring how far current report-generation models still are from clinical usefulness. PETAR aligned PET, CT, lesion masks and 11,356 localized descriptions from 5,126 examinations, with five physicians preferring or tying its outputs in 69 of 116 blinded comparisons, but numeric values were masked during training and quantitative errors excluded from reader evaluation, limiting conclusions about SUV or lesion-size fidelity.

Four evidence gaps are conspicuous. First, most PET foundation models focus on FDG oncology; evidence for neurodegenerative, cardiac, pediatric, infection and theranostic applications is sparse. Second, longitudinal reasoning is underdeveloped even though response assessment depends on consistent comparison across time; generalist longitudinal PET benchmarks remain absent. Third, the relationship between text generation and quantitative correctness is unresolved: a report may correctly mention a lesion but misstate its SUV, size, temporal change or anatomic location. Fourth, prospective human-AI evidence is rare. The field has many retrospective accuracy studies but few silent deployments, crossover reader trials, time-motion studies or patient-outcome evaluations. Medical vision-language models can also generate convincing but unsupported details; in one evaluation of GPT-4V across radiologic images, false-positive rates were high and performance remained well below expert readers.

To organize the evidence requirements, the authors propose MOSAIC-PET, a heuristic six-domain evaluation framework encompassing Molecular fidelity, Organ and anatomic grounding, Semantic faithfulness, Adaptability, Integration and clinical impact, and Continuous governance. The framework is explicitly unvalidated: it has not undergone Delphi consensus or formal testing and is intended to organize PET-specific evidence requirements rather than function as a certification score. The near-term translational priority, they argue, is the development of modular, auditable systems in which validated quantitative tools and evidence-linked language outputs remain under specialist oversight, progressing from retrospective external testing to prospective silent evaluation, assisted-use reader studies, clinical-impact assessment and continuous lifecycle monitoring. Regulatory alignment with CLAIM 2024, TRIPOD+AI, FUTURE-AI, DECIDE-AI and the FDA’s 2025 guidance on AI-enabled device software functions should treat authorization as an ongoing process rather than a one-time event.

The reviewed evidence supports a transition in development strategy, not yet a transition to routine clinical practice. The most defensible near-term architecture is modular and claim-specific: PET physics, quantitative tools, image-text grounding and language generation should remain separately testable, evidence linked and overseen by nuclear medicine specialists. A credible PET/CT foundation-model ecosystem should preserve quantitative molecular meaning, expose uncertainty, ground language in images and measurements, generalize across clinically relevant domains and remain auditable throughout its lifecycle. The field now has a proposed structure for organizing that evidence; whether it can close the gap between technical promise and patient benefit will depend on multitracer and longitudinal corpora, PET-specific factuality benchmarks, and prospective studies that move beyond retrospective accuracy toward genuine clinical impact.

Subject of Research: From task-specific AI to multimodal foundation models in PET/CT: a readiness framework for clinical translation

Article Title: From task-specific AI to multimodal foundation models in PET/CT: a readiness framework for clinical translation

Article References: Muneer, A., Saad, M. B., Xu, H., & Wu, J. (2026). From task-specific AI to multimodal foundation models in PET/CT: a readiness framework for clinical translation. Frontiers in Nuclear Medicine, 6, Article 1956567. https://doi.org/10.3389/fnume.2026.1956567

Image Credits: AI Generated

DOI: 10.3389/fnume.2026.1956567

Keywords: task-specific, multimodal, foundation, models, readiness, framework, clinical, translation, scientific research