Source-linked AI summary
Synergistic Information Disentanglement for Omni-modal Slide Representation Learning in Computational Pathology
Mingxin Liu, Chengfei Cai, Anwen Lu, Pengbo Xu, Jun Li, Jinze Li, Depin Chen, Jun Xu
TL;DR
Multimodal CPath representation learning can under-use complementary information by aligning modalities around shared redundancy, overlooking synergistic diagnostic signals. Φ-Omni addresses this with an SIB module and ΦID objective grounded in PID theory, and reports superior few-shot performance across independent datasets and tasks against supervised and SSL baselines.
Problem
Existing multimodal CPath methods rely on contrastive alignment that can overlook synergistic information absent from individual modalities, while pathology reports remain under-utilized.
Method
Φ-Omni uses domain-specific encoders, an SIB module, and a ΦID objective to distill high-order cross-modal interactions while minimizing shared redundancy.
Results
Φ-Omni achieves superior few-shot performance across eight tasks and five independent cohorts compared with supervised MIL and SSL baselines.
Takeaways & Limitations
The framework supports slide representations that transfer to unimodal few-shot inference while retaining synergistic information from omni-modal pretraining.
Abstract
from arXiv · showhide
In computational pathology (CPath), developing omni-modal self-supervised learning (SSL) models that integrate histology, genomics, and clinical reports enables transferable representation learning for whole slide images (WSIs). Existing approaches implicitly force heterogeneous modalities into a uniform latent space by contrastive alignment, causing modality collapse where unique, synergistic diagnostic signals (termed as $\mathrmΦ$) are discarded in favor of trivial redundancy. We hypothesize that the strongest task-agnostic SSL training signal stems from distilling the synergistic interactions over merely aligning shared redundancy. To this end, we introduce \textsc{$\mathrmΦ$-Omni}, a synergistic information disentanglement framework grounded in Partial Information Decomposition (PID) theory for slide representation learning. Unlike standard contrastive approaches, \textsc{$\mathrmΦ$-Omni} employs a Synergistic Information Bottleneck (SIB) regulated by the proposed $\mathrmΦ\text{ID}$ objective, which explicitly suppresses marginal redundancy while maximizing irreducible synergy, thereby distilling high-order cross-modal interactions. Following pretraining on breast ($n$=1031) and lung ($n$=919) cohorts, \textsc{$\mathrmΦ$-Omni} demonstrates superior few-shot performance across five independent external datasets spanning eight tasks compared to supervised and SSL baselines. Source code is available here.
1 Introduction
Φ-Omni addresses limitations of multimodal slide representation learning by disentangling synergistic cross-modal information rather than relying only on redundancy-driven alignment. Its SIB and ΦID components support compact omni-modal pretraining and unimodal few-shot inference.
- Background: The framework builds on a CPath workflow in which WSIs are tessellated, patch features are extracted, and embeddings are aggregated for transferable slide representations.SSL slide embeddings can support zero- or few-shot transfer without task-specific fine-tuning.
- Motivation: Contrastive alignment can overlook synergistic diagnostic information that emerges only from jointly observing pathology, genomics, and clinical reports.Existing methods may under-utilize pathology reports and favor shared redundancy, despite diagnosis relying on complementary cross-modal cues.
- Method: Φ-Omni encodes omni-modal data with domain-specific encoders and integrates the embeddings through a Synergistic Information Bottleneck.The SIB module is designed to distill high-order cross-modal interactions into the slide representation.
- Method: The ΦID objective maximizes synergistic interactions while minimizing shared redundancy in the omni-modal latent space.This replaces redundancy alignment with synergistic disentanglement as the central SSL objective.
- Evaluation: Φ-Omni validates superior few-shot generalizability across eight diagnostic tasks and five public independent cohorts against supervised MIL and SSL baselines.The frozen slide encoder supports unimodal few-shot adaptation at inference.
2 Method
Φ-Omni encodes pathology, genomics, and report data with domain-specific encoders, then uses SIB and ΦID to suppress redundancy while distilling synergistic cross-modal information. Its training combines symmetric contrastive alignment with a ΦID regularizer that compresses marginal information and expands joint information.
- Omni-modal Feature Encoding: Domain-specific encoders project pathology slides, genomic profiles, and text reports into modality-specific embeddings for omni-modal learning.The pathology pipeline uses WSI patch embeddings and multiple-instance aggregation; genomics and reports are separately projected into embeddings.
- Synergistic Information Bottleneck: SIB maps concatenated unimodal embeddings from 3d to d through a 3d → 2d → d bottleneck, filtering modality-specific noise while retaining synergistic diagnostic signals.The bottleneck is implemented with projection layers and Layer Normalization.
- Synergistic Information Disentanglement: PID decomposes mutual information into redundancy, unique modality information, and synergy, motivating Φ-Omni’s focus on information available only through joint multimodal observation.The method treats standard mutual-information maximization as insufficient because it can favor shared redundancy and modality collapse.
- Synergistic Information Disentanglement: ΦID maximizes the whole-minus-sum synergistic residual while minimizing marginal redundancy inferred from individual modalities.The objective uses projected marginal and joint representations to separate complementary cross-modal interactions from unimodal information.
- Gaussian Canonical Projector: GCP maps heterogeneous embeddings into a smooth Gaussian latent space, enabling tractable mutual-information estimation through differential-entropy approximations.The joint covariance log-determinant represents feature volume, which is expanded to prevent representation collapse; ε = 10^-5 provides numerical stability.
- Pretraining Objective: Pretraining combines symmetric pathology-centered InfoNCE alignment with ΦID regularization as L = LSymCL + γLΦID, using γ = 0.2.The contrastive term aligns pathology with genomics and reports, while ΦID penalizes trivial redundancy and encourages complementary features.
3 Experimental Setups and Results
Φ-Omni is evaluated through omni-modal pretraining, few-shot transfer across independent cohorts, and ablations of architecture, fusion, and loss design. Results report broad gains over MIL and SSL baselines, with synergy disentanglement improving performance across six tasks.
- Study Design: Pretraining uses strictly triplet-aligned pathology WSIs, genomic profiles, and pathology reports, while inference requires only WSIs.Breast and lung cohorts contain 1,031 and 919 cases, respectively.
- Few-shot Results: Φ-Omni surpasses all MIL baselines in 8/8 tasks, including gains of +4.3%, +1.2%, and +1.1% over ABMIL for molecular prediction, breast subtyping, and lung subtyping.Φ-Omni uses linear probing, whereas MIL models are trained from scratch in the few-shot setting.
- Few-shot Results: Φ-Omni outperforms SSL models on most tasks, including +3.2% over TANGLE on BRACS and +5.7% over UNIv2 on CPTAC-LSCC.The advantage is reported consistently across all k values and tasks.
- Ablation Study: SIB outperforms Trilinear and Concat fusion across all six tasks, while the selected ABMIL pre-attention architecture improves over MLP, TransMIL, and ABMIL.These comparisons constitute the architecture ablation.
- Ablation Study: +4.4% on BRACS-Subtyping and +7.2% on BRCA-PIK3CA result from combining SymCL with ΦID rather than using SymCL alone.The ablation identifies ΦID as indispensable for the reported synergistic improvements.
- Interpretability Analysis: Attention heatmaps align high-attention regions with tumor areas, while t-SNE shows three separated modality clusters with preserved cross-modal connections.The analysis covers lung and breast WSIs and pathology, transcriptomics, and text-report embeddings.
4 Conclusion
Φ-Omni shifts omni-modal slide representation learning from redundancy-driven contrastive alignment toward synergistic information disentanglement. Its interpretability analysis reports tumor-focused attention and separated modality clusters, while the paper concludes that synergy distillation supports data-efficient multimodal pathology frameworks.
- Conclusion: Φ-Omni uses PID-derived SIB and ΦID mechanisms to extract high-order diagnostic insights absent from individual modalities.The conclusion frames this as a shift from redundancy-driven alignment to synergistic information disentanglement.
- Conclusion: The paper reports state-of-the-art few-shot performance across six benchmarks together with biologically grounded interpretability.The conclusion connects these outcomes to omni-modal histology, genomics, and pathology reports.
- Conclusion: The authors position synergistic interaction distillation as a basis for data-efficient and robust multimodal computational pathology frameworks.This statement is presented as the work's broader supported implication.
- Interpretability: Attention heatmaps show high-attention regions corresponding to tumor areas in randomly selected lung and breast WSIs.Red and blue indicate high and low attention, respectively.
- Interpretability: t-SNE visualization separates pathology, transcriptomics, and text-report embeddings into distinct clusters while retaining cross-modal connections.The visualization is presented as evidence against modality collapse and for preserved biological and clinical correlations.