Multimodal Cell Foundation Models Require Synergistic Integration Beyond Linear Alignment

Integrating biological measurements across distinct modalities and physical scales into unified quantitative frameworks of cellular function represents a central frontier in computational biology. Although specialized unimodal foundation models have demonstrated remarkable predictive power, the acute scarcity of large-scale paired datasets severely limits the end-to-end training of joint multimodal architectures. This constraint has favored the development of compositional foundation models, which fuse fixed, pretrained unimodal expert encoders through lightweight learned interface networks. In these modular systems, unimodal encoders remain entirely frozen, meaning that all multimodal synthesis is determined exclusively by the chosen fusion mechanism. However, it has remained fundamentally unclear whether existing fusion strategies extract genuinely complementary biological information beyond the strongest unimodal representation or merely aggregate redundant signals across views. A study published in Nature Machine Intelligence introduces a theoretical and empirical framework showing that standard alignment objectives collapse into linear redundancy recovery, establishing that effective multimodal cellular modeling requires shifting toward synergy-driven integration interfaces that systematically exploit cross-modal interactions.

To rigorously diagnose when multimodal fusion provides true additive predictive value, the research team established a mathematical metric termed the Synergistic Information Score, which is formally grounded in partial information decomposition. Under partial information decomposition, the mutual information shared between a downstream target variable and two source representations is decomposed into redundant, unique, and synergistic components. Redundant information is accessible from either representation alone, unique information is provided exclusively by a single modality, and synergistic information emerges only when both modalities are evaluated jointly. Implementing the minimum mutual information assumption, the authors defined the Synergistic Information Score as the relative information gain achieved by a fused joint representation beyond the maximum mutual information captured by the strongest individual unimodal baseline. In practical implementation, calculating mutual information in high-dimensional feature spaces is intractable, so the framework approximates these quantities using empirical performance metrics obtained from linear probes, such as macro-averaged F1 scores for discrete classification tasks and coefficients of determination for continuous regression benchmarks.

Restricting downstream evaluation strictly to linear probes is a deliberate methodological design choice that directly measures linearly accessible synergy. When foundation model encoders are kept frozen, downstream applications rely on representations whose task-relevant structure is immediately usable without requiring complex nonlinear decoders that defer the integration burden to downstream stages. Within this operational diagnostic framework, a positive Synergistic Information Score confirms that a fusion interface successfully renders cross-modal synergistic interactions linearly accessible to downstream probes. Conversely, a near-zero score indicates that the target task is dominated by redundant features already captured by unimodal experts, while a negative score indicates fusion failure or representational degradation below the unimodal baseline. Furthermore, under conditions of substantial modality imbalance, the metric is intentionally conservative: a fused model must strictly surpass the performance of the dominant unimodal expert to yield a positive score. A low score therefore diagnoses an absence of accessible incremental gain rather than implying that the weaker modality lacks biological relevance.

The study provides a formal theoretical explanation for why standard multimodal alignment methods frequently stall at comparable performance plateaus, identifying a fundamental constraint termed the spectral ceiling. Extending foundational self-supervised learning theory to multimodal fusion with frozen encoders, the authors demonstrate that a broad family of alignment objectives reduces mathematically to an eigenvalue or singular value decomposition problem over the empirical cross-covariance matrix computed between aligned representations. When linear projection mappings operate under standard variance or whitening constraints designed to prevent trivial representational collapse, trace maximization problems are solved via the Eckart-Young theorem by the top singular vectors of the whitened cross-covariance. Consequently, exact spectral alignment methods, including canonical correlation analysis, variance-invariance-covariance regularization, and Barlow Twins, act strictly as alignment interfaces that optimally isolate shared linear correlations within the linear subspace. By their mathematical formulation, these spectral objectives cannot capture nonlinear cross-modal interactions or synergistic biological states that reside outside this linear subspace, whereas non-spectral integration interfaces break this eigenvalue structure.

To validate the distinction between spectral alignment and non-spectral integration, the researchers constructed a controlled five-level synthetic data generator that systematically modulated noise, structured private variation, and nonlinear coupling between two projected modalities from a shared Gaussian latent variable. At the initial level with perfectly linear coupling and infinite signal-to-noise ratios, all benchmarked methods accurately recovered the ground-truth shared signal. However, as the synthetic data progressed through intermediate noise degradation and structured private noise to moderate nonlinear coupling involving hyperbolic tangent and sinusoidal functions, and ultimately to strong rotated nonlinear coupling with asymmetric private noise, spectral methods degraded precipitously in canonical cosine similarity to the generative shared directions. In contrast, non-spectral integration methods, such as deep canonical correlation analysis, neural Deep InfoMax, Bootstrap Your Own Latent, Simple Siamese, and the Compositional Multimodal Model, degraded at substantially slower rates, successfully preserving high canonical cosine recovery under severe nonlinearity because their asymmetric predictors or synergy-aware loss terms avoided reduction to symmetric trace maximization.

The benchmarking framework evaluated ten distinct multimodal fusion interfaces across three diverse spatial transcriptomics datasets, combining histopathology image patches with spatial gene expression profiles. Image embeddings were derived from the frozen UNI-200M pathology foundation model, while spatial transcriptomic embeddings were generated using the frozen Nicheformer foundation model trained on SpatialCorpus-110M. Five biologically grounded downstream tasks were interrogated: cell-type composition regression using ridge regression, niche-type classification using multinomial logistic regression, spatial consistency, niche consistency, and spatial neighbourhood prediction across physical distances. Crucially, pairwise consistency diagnostics demonstrated that multimodal fusion preserved the intrinsic biological manifold and spatial organization learned during foundation model pretraining, with both unimodal and fused representations maintaining area under the receiver operating characteristic curve values exceeding 0.8. Multimodal fusion models matched or marginally exceeded unimodal consistency scores, demonstrating that downstream performance variations stemmed from information synthesis rather than topological distortion or representational degradation across tissue sections.

Empirical evaluations revealed that multimodal fusion is inherently task-dependent, demarcating distinct information regimes governed by spatial resolution and biological measurement properties. In breast cancer and pulmonary fibrosis lung atlases profiled via high-resolution Xenium in situ technology, tissue morphology and gene expression exhibit tight local correspondence at matched resolution. For niche classification in the breast dataset, leading fusion methods yielded only modest positive Synergistic Information Scores, with canonical correlation analysis reaching 0.058, the Compositional Multimodal Model reaching 0.045, and simple concatenation remaining at 0.011, reflecting a redundancy-dominated regime. Similarly, in the lung cell-type composition task, cellular proportions estimated from Xenium cell annotations align directly with gene expression. Here, the unimodal gene expression baseline proved overwhelmingly dominant with a coefficient of determination of 0.684, causing Synergistic Information Scores across all fusion interfaces to become negative, including a score of negative 0.028 for the Compositional Multimodal Model, confirming that multimodal integration was completely unnecessary.

In marked contrast, the developmental human thymus dataset exposed a cross-modal-dependent regime driven by intrinsic measurement resolution mismatch. In this dataset, gene expression was profiled using Visium spatial transcriptomics, which captures transcripts across coarse spots encompassing multiple cells, creating inherent spatial ambiguity relative to high-resolution histopathology. This resolution mismatch prevented single modalities from saturating task-relevant information and distributed complementary spatial signals across views. Consequently, multimodal models achieved consistently positive Synergistic Information Scores for thymus niche classification, with non-spectral integration methods delivering the largest synergistic gains, led by deep canonical correlation analysis at 0.142 and the Compositional Multimodal Model at 0.120. Simple concatenation yielded a smaller score of 0.072, while linear canonical correlation analysis remained confined to 0.057. A class-wise thymus case study further revealed that these synergistic improvements were concentrated within anatomically intricate capsular and corticomedullary compartments where visual morphology and coarse transcriptomic signals interact nonlinearly to define cellular microenvironments.

The cross-modal-dependent regime expanded dramatically when models were evaluated on spatial neighbourhood prediction tasks requiring extrapolation beyond immediate observation. In the lung dataset, unimodal representations performed robustly when predicting niche identities or cell compositions at short centre-to-neighbour distances of 10 micrometres or less, yielding Synergistic Information Scores near zero. However, as the physical separation increased to between 20 and 60 micrometres, local histological correspondence weakened, and unimodal performance decayed rapidly, particularly for the gene expression baseline in continuous neighbour cell-type composition regression. In response to this growing spatial ambiguity, the Synergistic Information Score rose steeply from approximately zero to greater than 0.15. Across all three datasets, neighbourhood prediction scores converged to positive values between 0.03 and 0.07 at extended spatial ranges, demonstrating that tissue-scale structural inference fundamentally shifts tasks away from local unimodal redundancy toward cross-modal dependence, where synergy-aware integration becomes indispensable.

Finally, a scaling analysis comparing unimodal expert fine-tuning against multimodal interface training established critical practical principles for deploying cell foundation models under finite annotation budgets. Fine-tuning the gene expression encoder on downstream training fractions ranging from 1 percent to 100 percent while keeping the image encoder frozen demonstrated rapid performance saturation in unimodal-sufficient tasks. In lung niche classification and cell-type regression, fine-tuning just 1 percent of labeled data yielded substantial performance gains that matched or exceeded multimodal baselines, driving Synergistic Information Scores consistently negative. Adapting the dominant unimodal expert proved to be the most sample-efficient strategy for redundancy-dominated tasks, rendering multimodal fusion an unnecessary parameter overhead. Conversely, in the resolution-mismatched thymus dataset, the synergy-aware Compositional Multimodal Model maintained consistently positive Synergistic Information Scores across all fine-tuning fractions. These findings establish the Synergistic Information Score as an essential diagnostic workflow to determine whether computational pipelines should prioritize targeted unimodal refinement or invest in synergy-driven multimodal integration.

Subject of Research: Multimodal cell foundation models and synergistic information integration in spatial biology

Article Title: Multimodal cell foundation models require synergistic integration beyond alignment

Article References: Richter, T., Zimmermann, E., Hall, J., Theis, F. J., Raghavan, S., Winter, P. S., Amini, A. P., & Crawford, L. (2026). Multimodal cell foundation models require synergistic integration beyond alignment. Nature Machine Intelligence. https://doi.org/10.1038/s42256-026-01328-3

Image Credits: AI Generated

DOI: 10.1038/s42256-026-01328-3

Keywords: computational biology, cell foundation models, multimodal integration, spatial transcriptomics, histopathology, partial information decomposition, synergistic information score, machine learning, Multimodal, cell, foundation, models