A conventional CT scan contains far more information than is visible in any single screen display. CT attenuation is measured in Hounsfield units, a scale that distinguishes materials according to how strongly they weaken X-rays. Air-filled lung tissue occupies a very low range, soft tissues cluster around intermediate values, and dense bone lies at much higher values. Radiologists therefore adjust the window width and level—the range and midpoint of Hounsfield units mapped to grayscale—to emphasize particular anatomical structures. A lung window reveals faint differences in aerated tissue but can make soft-tissue contrast difficult to assess. A bone window highlights fractures and mineralized structures while suppressing many soft-tissue details. A soft-tissue window improves visualization of organs and masses but may obscure subtle lung or bone abnormalities. The new system attempts to preserve these complementary signals in one synthesized representation.

Creating a reliable target image for such a system is not straightforward because an all-in-one image is not directly acquired by the scanner. It is a secondary visual representation assembled from multiple window settings, and radiologists may disagree about how regions should be blended or prioritized. Instead of claiming to possess a definitive “ground truth,” the researchers constructed what they call a silver-standard reference. Their hybrid pipeline combined conventional image-processing operations with deep-learning segmentation to identify bone, lung and soft-tissue regions. Information from the different windows was then fused according to the segmented anatomy. The resulting images served as practical training references: not perfect physical measurements, but carefully engineered targets that encode the tissue-specific appearance clinicians need to examine.

The segmentation stage used TransU-Net, a model that combines the U-Net architecture’s ability to preserve fine spatial detail with transformer-based mechanisms capable of modeling relationships across a larger image. U-Net-like networks process an image through an encoder that extracts increasingly abstract features and a decoder that restores resolution, using skip connections to retain boundaries and local structure. Transformers add attention operations, allowing the network to weigh relationships between distant regions rather than relying only on neighboring pixels. That combination is useful in CT, where the identity of a structure may depend on its broader anatomical context. Tested on the public CT-ORG dataset, TransU-Net achieved a mean Dice similarity coefficient of 0.934. The Dice score measures overlap between predicted and reference regions, with 1 representing perfect agreement, indicating that the model could separate the principal anatomical categories with high consistency in the evaluation.

Once the regions had been identified and the silver-standard images generated, the investigators trained a Pix2Pix conditional generative adversarial network to perform the conversion directly from routine CT input. A GAN contains two competing neural networks: a generator creates an output image, while a discriminator attempts to distinguish generated images from reference images. During training, the generator learns not only to reproduce the overall intensity pattern but also to create images that appear structurally plausible to the discriminator. Pix2Pix is conditional because its output is guided by a corresponding input image rather than generated from random noise alone. In this study, that arrangement allowed the model to learn a mapping from standard DICOM-derived CT images to the fused all-in-one format. The researchers compared Pix2Pix with CycleGAN and AttentionGAN, two other image-translation approaches.

Pix2Pix produced the strongest numerical results among the tested models. Its structural similarity index measure, or SSIM, reached 0.99, suggesting close agreement in luminance, contrast and local structure with the reference images. Its peak signal-to-noise ratio was 43.42 ± 1.39 decibels, a high value indicating relatively low pixel-level reconstruction error. The model also recorded a Fréchet inception distance of 11.03 ± 8.64. FID estimates how closely the distribution of generated images resembles that of reference images in a feature space; lower values generally indicate greater similarity. These metrics do not prove that every clinically important abnormality has been preserved, but together they suggest that the generated images were visually and structurally close to the engineered targets. The results also illustrate why the reference-construction step matters: the model can only learn the tissue priorities and appearance conventions represented in the data used for training.

Expert reviewers found the synthesized images to have good overall quality and judged the depiction of abdominal, lung and bone structures favorably. The researchers then conducted a preliminary lesion-detection assessment using 64 follow-up CT examinations from patients with breast cancer. Accuracy was 85.94 percent for lung lesions and 87.50 percent for bone lesions. Those figures indicate that the generated images retained substantial diagnostic information in two clinically important categories, particularly because lung and skeletal abnormalities are often best appreciated in different conventional windows. The study does not establish that the AI-generated display is equivalent to a full radiologist review of the original scan, nor does it show that the system improves patient outcomes. Rather, it provides an early feasibility test of whether one synthesized image can preserve enough information to support lesion recognition.

The central technical challenge is balancing information compression against diagnostic fidelity. A single display must combine intensity ranges that are ordinarily separated because the grayscale dynamic range of a monitor is limited. Segmentation gives the algorithm a way to apply different transformations in different anatomical regions: lung information can be emphasized inside the thorax, bone detail can be enhanced around the skeleton, and soft-tissue contrast can be maintained in organs and masses. Yet anatomical boundaries are not always clean, and pathology can cross or distort the expected regions. A tumor, hemorrhage, atelectasis or treatment-related change may not conform to the categories used by a fusion rule. Generative models also learn statistical patterns from their training data, creating a need for careful validation to ensure that image synthesis does not smooth, distort or invent clinically relevant features. For that reason, the original CT data and conventional windows remain essential references rather than being rendered obsolete by the synthesized view.

The work was conducted by Yunjung Hong and Jiyoung Woo of Soonchunhyang University’s Department of ICT Convergence and Sanghyeok Lim of Soonchunhyang University Hospital Bucheon in the Republic of Korea. The retrospective hospital data were anonymized, and the study received institutional review-board approval; the public CT-ORG dataset was also anonymized. Supported by Soonchunhyang University and Korean National Research Foundation grants, the project points toward a possible change in how CT information is presented rather than how it is acquired. If validated across larger and more diverse datasets, an all-in-one display could help radiologists survey multiple tissue types more efficiently before examining suspicious regions in the original window settings. The authors describe the framework as a potential aid to multi-window interpretation, not an autonomous diagnostic system. Its next test will be whether the apparent visual convenience and strong reconstruction scores translate into reliable performance across scanners, diseases, institutions and real-world clinical workflows.

CT-RATE is remarkable not only for its size but also for its clinical depth and diversity. The dataset encompasses 25,692 non-contrast chest CT scans sourced from 21,304 unique patients, ensuring a robust sample representative of a wide spectrum of clinical presentations. Each CT volume is meticulously aligned with its respective radiological narrative, providing rich multimodal data that enables models to learn complex correlations across visual and textual medical information. This comprehensive resource aims to catalyze progress in AI-assisted diagnostics, bridging the gap between radiological imaging interpretation and natural language understanding.

Leveraging this new dataset, the research team developed CT-CLIP, a novel contrastive learning framework specifically tailored for 3D CT volumes. CT-CLIP extends the principles of contrastive language–image pretraining (CLIP), a paradigm celebrated for enabling flexible, zero-shot learning in natural image domains, into the specialized terrain of volumetric medical imaging. Uniquely, it undertakes the challenge of mapping entire 3D CT structures into a semantically rich joint embedding space shared with radiology reports, enabling versatile downstream applications without the necessity for task-specific retraining.

The transformative potential of CT-CLIP is multifaceted. In the critical application of multi-abnormality detection, it demonstrates superior performance, outstripping state-of-the-art fully supervised models across all major metrics. This suggests that CT-CLIP not only generalizes across diverse pathological features but also robustly captures subtleties within the volumetric data that traditional models might overlook. Additionally, CT-CLIP’s prowess in case retrieval promises to revolutionize clinical workflows by rapidly finding analogous historical cases, a capability that can substantially augment diagnostic confidence and medical education.

However, the research journey did not stop at diagnosis and classification. Building upon the vision encoder of CT-CLIP, the investigators introduced CT-CHAT, a vision–language foundational chat model uniquely designed for interactive dialogue involving 3D chest CT volumes. By integrating CT-CLIP with a pre-trained large language model, CT-CHAT merges visual understanding with advanced conversational capabilities. This synergy allows clinicians and researchers to query complex imaging data in natural language, receiving contextually relevant and clinically insightful responses grounded in the volumetric data.

The training of CT-CHAT involved an extraordinary scale of fine-tuning, utilizing over 2.7 million question–answer pairs generated from the CT-RATE dataset. This massive supervised training corpus laid the foundation for a sophisticated AI assistant adept at nuanced interpretation and communication, a crucial step toward realizing AI as an integral partner in clinical decision-making. Importantly, CT-CHAT exemplifies the necessity for specialized methodologies and architectures when transitioning AI systems from two-dimensional imaging to the more information-dense and computationally challenging domain of 3D medical imaging.

The integration of CT-RATE, CT-CLIP, and CT-CHAT together forms a comprehensive ecosystem that addresses previously insurmountable challenges in medical AI for 3D imaging. Open-sourcing these assets marks a pivotal move towards democratizing access and fostering innovation across research communities and clinical institutions worldwide. This openness is expected to accelerate the development of new diagnostic tools, educational platforms, and decision support systems that harness the full potential of volumetric imaging and natural language processing.

Moreover, the methodological advancements embodied by CT-CLIP demonstrate a crucial paradigm shift. Where traditional AI approaches often require extensive labeled datasets tailored for each specific diagnostic task, the contrastive learning framework promotes the development of foundation models capable of zero-shot adaptation. This not only streamlines the research and deployment pipeline but also significantly lowers the barrier for adopting AI solutions in diverse healthcare settings where annotated data may be scarce.

CT-CHAT’s conversational interface exemplifies how AI can move beyond static image interpretation toward dynamic and interactive clinical workflows. By allowing clinicians to engage in dialogue with the AI about patient scans, this model enables a more intuitive exploration of the data, providing explanations, highlighting abnormalities, or retrieving related case histories on demand. Such a system empowers radiologists to leverage AI as a collaborative partner rather than a black-box tool, enhancing transparency and trust in AI-generated insights.

The comprehensive scale and integration of multimodal data in CT-RATE are particularly noteworthy given the inherent complexity of aligning 3D imaging with unrestricted textual narratives. Radiology reports are rich in nuance, imbued with clinical context, diagnostic reasoning, and descriptive details that are challenging for AI to parse. By creating a dataset that tightly couples volumetric data with authentic, contemporaneous reports, the researchers have provided an invaluable training ground for models to learn complex multimodal relationships, pushing beyond the limitations of purely vision- or text-based datasets.

Looking ahead, the release of the CT-RATE dataset alongside CT-CLIP and CT-CHAT is poised to catalyze a new era of innovation in medical imaging AI. Researchers will now be able to develop more generalized, interpretable, and clinically useful AI tools that address not only detection but also explainability, usability, and integration into real-world workflows. This holistic approach is essential to surmount longstanding barriers, translating technical breakthroughs into tangible clinical benefit and improved patient outcomes.

Furthermore, the emphasis on open-source release fosters a spirit of collaborative advancement, inviting the global research community to build upon this foundational work. As AI models trained on CT-RATE grow more sophisticated, the potential to uncover previously unnoticed imaging biomarkers or to automate complex diagnostic tasks increases, thereby expanding the frontiers of precision medicine. The paradigm of multimodal, foundation model pretraining illustrated here is likely to inspire analogous developments in related domains such as MRI, ultrasound, and other medical imaging modalities.

In conclusion, this landmark study charts a visionary path forward, demonstrating that multidisciplinary synergy between radiology, AI, and data science can be harnessed to unlock the diagnostic potential buried in vast repositories of 3D imaging data. From the meticulous construction of a large-scale multimodal dataset to the development of advanced foundation models and interactive AI agents, the work highlights how focused innovation can transcend data scarcity and computational challenges. It heralds a future where AI not only augments the capabilities of clinicians but revolutionizes the very nature of medical imaging analysis.

As the clinical community embraces these tools, the implications extend beyond diagnostics to include personalized medicine, disease monitoring, and education. The integration of CT-CHAT as an intelligent conversational interface exemplifies the move toward democratizing expert knowledge, making sophisticated image interpretation accessible to a wider spectrum of healthcare providers. Collectively, these advances underscore the profound and transformative impact that well-designed, large-scale multimodal AI systems can have on the practice of medicine in the 21st century.

Subject of Research: Development of generalist foundation models for 3D computed tomography through large-scale multimodal datasets.

Article Title: Generalist foundation models from a multimodal dataset for 3D computed tomography.

Article References:
Hamamci, I.E., Er, S., Wang, C. et al. Generalist foundation models from a multimodal dataset for 3D computed tomography. Nat. Biomed. Eng (2026). https://doi.org/10.1038/s41551-025-01599-y

Image Credits: AI Generated

DOI: https://doi.org/10.1038/s41551-025-01599-y