Artificial intelligence systems built to understand medical scans may be far better at spotting broad, organ-wide disease than detecting the tiny lesions that can matter most to cancer patients, according to a systematic evaluation of foundation models for computed tomography. The study tested five widely available medical-imaging models on the task of deciding whether six abdominal organs were normal or abnormal. The result was a sharp warning for the rapidly expanding field of medical AI: the model used to create an organ’s representation mattered considerably more than the sophistication of the method used to combine information from different regions of the scan. Even the most advanced attention-based approaches failed to close a persistent gap between diffuse abnormalities, such as cirrhosis, and focal abnormalities, such as metastases or small tumors.
Foundation models are large neural networks pre-trained on extensive collections of medical images, often without detailed human labels. Instead of learning one narrowly defined diagnostic task, they are designed to produce numerical descriptions, known as feature embeddings, that can later be reused for many applications. In principle, this approach could reduce the enormous amount of labeled data required to train a medical AI system. A hospital might take a frozen, pre-trained model, extract features from its CT scans, and add a relatively simple classifier to recognize disease. But the quality of those reusable representations depends on what the model learned during pre-training and how effectively the information is converted into a prediction for an entire organ.
The researchers from Radboudumc in the Netherlands examined that process using more than 2,000 abdominal CT scans from the AMOS-MM dataset, paired with organ-level abnormality labels generated by the LEAVS system. LEAVS uses a large language model to extract findings from radiology reports, and its abnormality-type extraction has an average F1 score of 0.89 against human annotations. The investigators excluded 200 scans from model development and used them as a human-annotated test set. They assessed the spleen, gallbladder, large bowel, left kidney, right kidney and liver, while stomach, pancreas and small bowel were removed from the main analysis because severe class imbalance made reliable comparisons difficult.
For every scan, the team first used TotalSegmentator, an automated anatomical-segmentation system, to locate each organ. The organ was then enclosed in a three-dimensional bounding box, which was divided into overlapping patches. The foundation models converted these local patches into feature vectors—lists of numbers encoding patterns such as texture, shape and possibly disease-related visual signals. Those patch-level vectors had to be assembled into one organ-level representation before a classifier could determine whether the organ was abnormal. The researchers compared simple statistical operations, including mean, median, maximum and standard deviation pooling, with a combined mean-and-standard-deviation representation. They also tested attention-based multiple instance learning, in which the system learns to assign different importance to different patches, as well as a hierarchical global transformer used by one of the models.
The five principal foundation models represented different philosophies of medical AI. Some were trained on three-dimensional CT volumes, allowing them to learn relationships between adjacent slices and volumetric structures. Others were based on two-dimensional images or trained across multiple imaging modalities. Their pre-training objectives also varied, including self-distillation, contrastive learning and alignment between images and radiology reports. The models included SPECTRE, TAP-CT, CT-FM, UMedPT and Curia, with Merlin included as a reference because its large input window can contain an entire organ without the need for patch aggregation. To keep the comparison focused, the investigators trained linear classifiers and two increasingly expressive multilayer perceptrons on the resulting embeddings rather than fine-tuning the foundation models themselves.
SPECTRE produced the strongest overall performance, achieving an area under the receiver operating characteristic curve, or AUC, of 0.714 when mean pooling was used. AUC measures how well a system ranks abnormal cases above normal ones; a value of 0.5 corresponds to random guessing, while 1.0 represents perfect separation. SPECTRE significantly outperformed its version without vision-language alignment, which achieved an AUC of 0.625, and TAP-CT, which reached 0.605. CT-FM, Merlin and UMedPT occupied a middle range, while Curia performed worst, with an AUC of 0.468—slightly below random performance in this frozen feature-extraction setting. Across the comparison, three-dimensional CT-native systems generally outperformed models trained primarily on two-dimensional or multimodal data, although the authors emphasize that architecture, training data and pre-training objectives were intertwined and could not be isolated completely.
The most surprising finding was that increasingly elaborate ways of combining patches did not deliver a meaningful advantage. None of the tested aggregation strategies significantly outperformed simple mean pooling. The best alternative improved AUC by only 0.008 compared with the mean, with a 95 percent confidence interval ranging from −0.002 to 0.022. The same pattern held for focal abnormalities. Attention-based multiple instance learning did sometimes distribute its weights unevenly across patches, suggesting that the model was not always treating every region identically, but those learned preferences did not translate into better generalization on unseen scans. In some models, attention weights were nearly uniform, effectively behaving like mean pooling. SPECTRE’s hierarchical global transformer also failed to improve performance, producing a difference of −0.033 relative to mean aggregation, with uncertainty that included no difference.
Disease location and scale proved more consequential than the aggregation algorithm. When the researchers separated diffuse from focal abnormalities, only the liver and gallbladder had enough examples of each type for a reliable analysis. Among models that performed better than chance, every one classified diffuse abnormalities more accurately than focal abnormalities. The largest difference in AUC was 0.108, with a 95 percent confidence interval of 0.080 to 0.135. The researchers then gave the downstream classifier more power by replacing the linear probe with multilayer perceptrons containing one or two hidden layers. That additional capacity did not improve focal-abnormality performance, indicating that the limitation was probably not simply an overly weak classifier. In illustrative liver cases, models confidently recognized diffuse changes consistent with cirrhosis, were less certain about a lesion measuring approximately 48 by 40 millimeters, and often assigned low abnormality probabilities to a lesion of about 13 by 11 millimeters.
The technical explanation may lie in what the models are rewarded for learning during pre-training. Radiology reports commonly describe a scan or an organ in global terms, but they do not always specify the exact coordinates, size or boundaries of every abnormality. A model trained to align an entire CT examination with its report may therefore learn that a liver has a generally abnormal appearance without learning which small cluster of voxels contains the lesion. In a large organ, a focal abnormality may occupy only a tiny fraction of the total volume. If the representation averages information across many patches, the signal from that lesion can be diluted by surrounding normal tissue. Attention mechanisms can theoretically counteract this dilution by focusing on informative regions, but they cannot recover spatially discriminative information that the foundation model never encoded in its patch embeddings.
The study also found no statistically significant evidence that overlap between pre-training and evaluation scans explained SPECTRE’s advantage. SPECTRE and UMedPT had been trained partly on AMOS22 data, which overlaps with about 29 percent of the scans in the study, including some test cases. After removing those overlapping scans, SPECTRE’s performance changed by 0.022, with a confidence interval from −0.005 to 0.062, while UMedPT changed by −0.004. Neither difference was statistically significant. Nevertheless, the investigators caution that removing identical scans cannot eliminate every possible source of dataset contamination: institutions, scanners, patient populations and acquisition protocols may still overlap. The findings should also not be interpreted as clinical validation. The evaluation used one abdominal CT dataset, report-derived labels and frozen encoders, not prospective patients or fully fine-tuned diagnostic systems.
The implications are both practical and urgent. For developers using foundation models as fixed feature extractors, choosing a strong encoder may be more valuable than adding a complex aggregation module. Mean pooling, despite its simplicity, provides a competitive baseline and is easier to implement, interpret and test. But the results also expose a potentially dangerous blind spot for systems intended to support oncology, where small, localized lesions can be more clinically consequential than diffuse organ changes. Future models may need pre-training objectives that explicitly preserve spatial detail, such as lesion-focused contrastive learning, anatomically informed masking or synthetic insertion of focal lesions. Spatially grounded supervision could teach a network not only that an abnormality exists but where it is located, although such approaches may require more annotation and sacrifice some scalability. Until these strategies are tested across independent datasets and with fine-tuning, a high overall score from a medical foundation model should not be taken as evidence that it can reliably detect the smallest and most important abnormalities.
