Source-linked AI summary
Disentangled Shared Representations Improve Morpho-Transcriptomic Integration
Julian Ostermaier, Swann Ruyter, Reuben Dorent, Daniel Racoceanu
TL;DR
Standard multimodal models for paired H&E and spatial transcriptomics do not explicitly separate shared from modality-specific information. This paper compares VAE-based and contrastive models with standard or disentangled representations, finding that contrastive models generally perform better downstream while disentanglement improves selected readouts variably.
Problem
Standard multimodal models share a latent space without controlling which information is shared or modality-specific in paired H&E and ST data.
Method
The study compares VAE-based and contrastive models with standard or disentangled representations across two cancer cohorts using reconstruction and probing evaluations.
Results
Contrastive models achieved higher downstream probe performance than VAE-based models, while disentanglement improved selected readouts inconsistently across tasks, directions, panel sizes, and strengths.
Takeaways & Limitations
Disentangling shared and modality-specific information can improve selected morpho-transcriptomic representation readouts, but its benefits are not uniform.
Abstract
from arXiv · showhide
Spatial transcriptomics (ST) enables the simultaneous profiling of gene expression and tissue morphology, creating an opportunity to learn multimodal representations capturing shared morpho-transcriptomic structure. However, standard multimodal models often compress modalities into a common latent space without explicitly separating shared and modality-specific sources of variation, which may limit downstream utility. We investigate whether explicit disentanglement of shared and private latent components improves multimodal representation learning for paired Hematoxylin \& Eosin (H\&E) and ST data. We compare VAE-based and contrastive approaches, each in standard and disentangled variants, across two cancer cohorts under matched experimental conditions. Representations are evaluated using cross-modal reconstruction, downstream probing and cross-modal probe transfer. The experiments suggest two main trends. First, contrastive objectives yield higher downstream probing performance than VAE-based models. Second, disentangled variants improve the selected reconstruction and probing metrics, although the gains depend on the model family, task, direction, and disentanglement strength. Overall, our results suggest that explicitly factorizing shared and modality-specific information can improve multimodal representation learning for spatial transcriptomics and provides a useful evaluation framework for future foundation models.
1 Introduction
The introduction motivates multimodal representation learning from paired H&E morphology and spatial transcriptomics, emphasizing that explicit separation of shared and modality-specific information may improve joint representations. The study therefore compares standard and disentangled VAE and contrastive models under matched conditions using reconstruction and probing evaluations.
- Motivation: Paired H&E morphology and gene expression from spatial transcriptomics enable multimodal representations of tissue organization.ST preserves tissue architecture while providing spatially resolved molecular profiling from the same tissue section.
- Motivation: Compact joint representations could encode morpho-transcriptomic organization in a low-dimensional latent space, paralleling transferable tissue features from histopathology foundation models.The motivation draws on evidence that compact image representations capture transferable features across tasks and cohorts.
- Related work: Existing multimodal methods use contrastive, autoencoder-based, or generative objectives but do not control which information is shared versus modality-specific.This limitation motivates explicit disentanglement of shared and private components.
- Study design: The study compares VAE and contrastive models in standard shared-latent and disentangled variants under matched experimental conditions.Representations are evaluated across multiple tissue cohorts using cross-modal reconstruction and downstream probing.
2 Methods
The study compares VAE-based and contrastive multimodal representation models, including disentangled variants, across two cancer cohorts using matched reconstruction and probing evaluations. Models learn shared representations while explicitly testing separation of shared and modality-specific information.
- Model comparison: Five multimodal representation models spanning VAE-based and contrastive families, with standard and disentangled variants, were compared across two cancer cohorts.The unified framework evaluates representations through cross-modal reconstruction and downstream probing.
- VAE models: MMVAE encodes modalities independently into a shared latent space, while MMVAE+ explicitly separates shared and modality-specific information.MMVAE+sg further blocks same-view reconstruction gradients from reaching the shared latent variable to reduce modality-specific leakage.
- Contrastive models: Contrastive models align paired multimodal observations with symmetric InfoNCE, whereas disSSL adds a penalty reducing modality-specific information retained in the shared representation.The disentanglement strength is controlled by β, with β = 0 recovering standard contrastive alignment.
- Training and cohorts: All models learned a 32-dimensional shared latent representation using leave-one-slide-out cross-validation with matched architectural and training choices where applicable.Experiments used Python, PyTorch, and NVIDIA V100 GPUs.
- Evaluation: Evaluation used frozen encoders and fixed probing architectures for cross-modal reconstruction and downstream prediction, with probing repeated five times using different random seeds.Reconstruction metrics were top-50 PCC for HE→G and cosine similarity for G→HE; downstream probes used tissue-type and tissue-niche labels on Colon.
3 Results
Disentangled models showed conditional improvements in cross-modal reconstruction and downstream probing rather than uniform gains. Contrastive models generally produced more task-useful shared representations than VAE-based models, while transfer effects depended on task and modality.
- Cross-modal Reconstruction Results: MMVAE+ and MMVAE+sg had higher mean reconstruction performance than MMVAE across cohorts, panel sizes, and directions, but few comparisons were significant after correction.The authors characterize this as a descriptive trend rather than a uniform effect.
- Cross-modal Reconstruction Results: For HE→G reconstruction, disSSL exceeded CLIP on GBM at K = 500 and K = 1000, while Colon results varied by panel size and G→HE showed no consistent improvement.On Colon, disSSL matched or slightly exceeded CLIP at K = 200 and K = 1000, whereas CLIP remained higher at K = 500; most comparisons were nonsignificant after correction.
- Downstream Probing Results: Contrastive models achieved higher cross-modal probe AUCs than VAE-based models for both tissue-niche and tissue-type classification.MMVAE performed worst, while MMVAE+ and MMVAE+sg improved over MMVAE but generally remained below CLIP and disSSL.
- Downstream Probing Results: MMVAE+ and MMVAE+sg improved cross-modality probing across both label sets, but transfer effects reversed by task: MMVAE+sg helped tissue-niche classification while disentangled variants lowered tissue-type transfer AUCs.The reversal suggests stronger factorization may retain less morphology-associated information for tissue-type transfer.
- Downstream Probing Results: DisSSL generally matched or exceeded CLIP for cross-modality probes up to intermediate β, with weaker gains at K = 200, and improved transfer in several conditions.Effects depended on β, panel size, and task.
4 Conclusion
The study compared standard and disentangled VAE-based and contrastive models for paired H&E and ST data under a common evaluation protocol. Disentangled VAE variants improved mean cross-modal reconstruction, while contrastive models achieved higher downstream performance.
- Model comparison: The evaluation compared VAE-based and contrastive models with standard shared latent spaces against disentangled variants for paired H&E and ST data.All models were assessed under a common evaluation protocol.
- Reconstruction: Disentangled VAE variants improved mean cross-modal reconstruction relative to MMVAE.Statistically significant gains were limited to a subset of configurations.
- Downstream evaluation: Contrastive models achieved higher downstream probing performance than VAE-based models.
Disclosure of Interests · Supplementary Material
The supplementary material documents evaluated models, statistical testing, shared training settings, retained HEST samples, downstream label-generation procedures, and an additional tissue-niche classification result. It also includes a model-overview table and supplementary analyses spanning Colon, GBM, and colorectal cancer cohorts.
- Model Overview: Table 2 summarizes the representation learning models evaluated in the study.
- Statistical Tables: Table 3 reports statistically significant Wilcoxon signed-rank results after Benjamini–Hochberg FDR correction across Colon and GBM datasets.The analysis uses slide-level metric values averaged across repeated runs, with significance defined at p < 0.05 after adjustment.
- Training Configurations: Table 4 lists training hyperparameters shared across all model configurations.
- Sample Overview for HEST Colorectal Cancer Cohort: Table 5 describes retained HEST colorectal cancer samples after low-quality filtering and enforcing one sample per patient.HEST ID denotes each sample’s unique identifier in the HEST database.
- Downstream Task Label Generation Colorectal Cancer Cohort: Pathologist annotations were standardized by normalizing formatting variants and merging biologically equivalent tissue-region labels.
- Downstream Task Label Generation Colorectal Cancer Cohort: Cell-type counts were converted to per-spot proportions, CLR-transformed, and jointly clustered across slides to assign consistent tissue-niche labels.
- Additional Results on Downstream Tasks: Figure 3 evaluates tissue-niche classification from H&E to gene expression using AUC across gene-panel sizes K and five repeated random-seed runs.Non-disentangled models are shown as stars, disentangled versions as circles, and the gray arrow spans CLIP from β = 0 toward increasing disentanglement strength.