Source-linked AI summary
Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learning
Ronald Xie, Kuan Pang, Sai W. Chung, Catia T. Perciani, Sonya A. MacParland, Bo Wang, Gary D. Bader
TL;DR
Spatially resolved gene-expression profiling from H&E images is valuable but expensive and technically demanding. BLEEP learns a contrastive joint embedding from paired image-expression data and imputes query-patch expression from reference profiles. On human liver 10x Visium data, it substantially outperformed existing methods across multiple gene sets, while preserving biological heterogeneity.
Problem
Spatial transcriptomics can be low-throughput, expensive, time-consuming, and dependent on specialized equipment, motivating prediction of spatial expression from histology images.
Method
BLEEP uses contrastive learning to align paired histology patches and expression profiles in a low-dimensional joint embedding, then imputes query expression from nearby reference profiles.
Results
BLEEP achieved significantly higher correlation than existing methods across marker, highly expressed, and highly variable genes, with improvements of 39% to 120% over the second-highest method.
Takeaways & Limitations
BLEEP preserved biological heterogeneity and recaptured and denoised gene-gene correlations in predicted expression profiles.
Takeaways & Limitations
Overall absolute correlations remained low for many genes, partly because morphology may poorly correlate with some genes and experimental artifacts can add non-biological variance.
Abstract
from arXiv · showhide
Histology imaging is an important tool in medical diagnosis and research, enabling the examination of tissue structure and composition at the microscopic level. Understanding the underlying molecular mechanisms of tissue architecture is critical in uncovering disease mechanisms and developing effective treatments. Gene expression profiling provides insight into the molecular processes underlying tissue architecture, but the process can be time-consuming and expensive. We present BLEEP (Bi-modaL Embedding for Expression Prediction), a bi-modal embedding framework capable of generating spatially resolved gene expression profiles of whole-slide Hematoxylin and eosin (H&E) stained histology images. BLEEP uses contrastive learning to construct a low-dimensional joint embedding space from a reference dataset using paired image and expression profiles at micrometer resolution. With this approach, the gene expression of any query image patch can be imputed using the expression profiles from the reference dataset. We demonstrate BLEEP's effectiveness in gene expression prediction by benchmarking its performance on a human liver tissue dataset captured using the 10x Visium platform, where it achieves significant improvements over existing methods. Our results demonstrate the potential of BLEEP to provide insights into the molecular mechanisms underlying tissue architecture, with important implications in diagnosis and research of various diseases. The proposed approach can significantly reduce the time and cost associated with gene expression profiling, opening up new avenues for high-throughput analysis of histology images for both research and clinical applications.
1 Introduction
Spatial transcriptomics can connect histology with gene expression but remains costly, time-consuming, and technically constrained. BLEEP addresses these challenges by aligning paired image and expression data in a joint space and imputing expression for query patches, outperforming existing methods on human liver tissue.
- Motivation: Spatial transcriptomics methods bridge histology imaging and gene expression but are often low-throughput, low-content, expensive, and technically demanding.These methods also require specialized equipment and extensive domain expertise.
- Motivation: Predicting spatially resolved expression from histology is challenging because image features may not fully determine expression profiles.The paper describes the problem as ill-posed and notes limited success from existing methods.
- Approach: BLEEP uses contrastive learning to align paired image and expression representations in a low-dimensional joint embedding space.The learned space supports expression imputation from a reference dataset for any query image patch.
- Results: On a human liver 10x Visium dataset, BLEEP significantly outperformed HisToGene and ST-Net in average correlation across marker, highly expressed, and highly variable genes.The evaluation also reports preservation of heterogeneity and recovery and denoising of gene-gene correlations.
- Contributions: The contrastive objective encourages shared cross-modal features, while query-reference imputation uses nearby reference profiles rather than predicting genes independently.The paper reports resilience to experimental artifacts within and across samples alongside improved performance.
- Contributions: BLEEP is presented as the first bi-modal embedding framework for expression prediction from histology images, with potential to reduce profiling time and cost.The claimed applications include high-throughput histology analysis in research and clinical settings.
2 Related Works
Prior methods differ in spatial resolution, evaluation coverage, and modeling assumptions, while BLEEP adapts contrastive joint embedding to paired histology and expression data. Its query-reference interpolation produces expression predictions directly from the learned shared space.
- Existing methods: ST-Net and HisToGene frame spatial expression prediction as feed-forward regression, using convolutional or vision-transformer image encoders.ST-Net uses ResNet50 with a fully connected layer, whereas HisToGene uses a vision transformer and larger field of view.
- Existing methods: Existing approaches include tissue-level predictors and spatially resolved methods, but the latter have faced limited panels, missing quantitative evaluation, or overfitting concerns.Tissue-level methods may correlate well while lacking spatially resolved predictions.
- Modeling assumptions: Spatial relations used by HisToGene and Hist2ST may not be necessary for tissues with distinct, repetitive patterns such as human liver tissue.The paper questions whether hard-coded spatial proximity helps in such settings.
- Contrastive embedding: BLEEP draws on CLIP-style contrastive learning to align Visium expression profiles with their spatially paired image patches in a joint embedding.The paired image patch spans roughly 55µm.
- Contrastive embedding: Unlike image-text models that require a domain-specific decoder, BLEEP directly interpolates in the joint embedding space to produce expression predictions.This distinguishes its prediction mechanism from CLIP’s image-text usage.
- Reference-based imputation: BLEEP’s query-reference imputation is partly inspired by SeuratV3 but predicts spatial expression from histology morphology without requiring a shared expression panel.SeuratV3 calculates profiles from a linear combination of nearby reference anchors given a query.
3 Methods
BLEEP learns aligned image–expression embeddings from paired liver tissue data, then predicts query-patch expression by combining nearby reference profiles in that space.
- Data and preprocessing: The dataset comprises four human liver tissue slices captured with the 10x Visium platform, with one slice held out for testing after preprocessing and batch correction.Expression profiles were normalized, highly variable genes were selected, and Harmony was used for batch correction.
- Learning bimodal embedding: BLEEP encodes paired H&E image patches and normalized expression profiles into h-dimensional representations using separate image and expression encoders.The image encoder processes patches, while the expression encoder maps gene profiles into the shared representation space.
- Learning bimodal embedding: Contrastive learning aligns paired image and expression representations while accounting for similarities among samples within the batch.The similarity-adjusted target uses internal image and expression similarities before cross-entropy loss is applied in both directions.
- Learning bimodal embedding: BLEEP uses a pretrained ResNet50 image encoder and a 256-dimensional fully connected expression encoder with projection heads for contrastive alignment.The image features are projected to the same dimension as expression features before applying a CLIP-like contrastive loss.
- Query-reference imputation: For query-reference imputation, H&E images are split into patches, embedded, matched to k nearest reference profiles by Euclidean distance, and reconstructed through a weighted linear combination.The reference profiles associated with the nearest embeddings provide the expression prediction for each query patch.
4 Experiments
BLEEP outperforms existing methods across gene sets and better preserves expression heterogeneity, gene-gene correlations, and spatial patterns. Its predictions remain affected by low overall correlations and choices in reference-profile averaging.
- Benchmark results: 120%, 39% and 90% increases in r value over the second-scoring method were achieved for MG, HEG and HVG sets, respectively.BLEEP had the highest correlation with ground-truth expression across all three gene sets.
- Benchmark results: BLEEP consistently achieved higher correlations than HisToGene and ST-Net for the top predicted spatially zonated genes.These included CYP3A4, CYP1A2, CYP2E1, GLUL and FABP1.
- Biological heterogeneity: BLEEP narrowly outperformed the baselines for mean expression while retaining biological heterogeneity that the regression methods failed to recapitulate.BLEEP generally underestimated variance, but fixed-scale visualizations preserved sufficient heterogeneity; its predictions also captured gene-gene correlations.
- Robustness and batch effects: All three methods were robust to low-quality regions and captured tissue zonation, while BLEEP was least prone to introducing batch effects during prediction.HisToGene had the best clustering agreement for this dataset, with BLEEP and ST-Net close behind.
- Ablation experiments: K = 10 reduced prediction quality, whereas larger K values offered small improvements for HVG and HEG but could further reduce variance through averaging.The authors retained K = 50 as the default because MG showed little improvement with increasing K.
- Ablation experiments: Averaging multiple reference profiles was preferred because the nearest match was usually not the best prediction, although averaging may remove genuine biological signals.Smoothing the contrastive objective produced a modest performance increase and may improve retrieval of the top K reference profiles.
5 Discussion and Conclusion
BLEEP constructs a joint image–expression embedding and uses query-reference imputation to predict spatial gene expression from histology images. It improves correlation with original expressions, preserves heterogeneity, and may reduce profiling time and cost, while averaging can smooth genuine abrupt biological signals.
- Framework: BLEEP embeds paired image and expression features jointly, then uses the k most similar reference expression profiles to impute each query image.This query-reference process predicts an entire expression profile jointly rather than each gene independently.
- Robustness: BLEEP prediction is reported to be resilient to experimental artifacts within and across samples.The framework addresses the expression-prediction task through shared modality features and joint profile imputation.
- Results: BLEEP preserves biological heterogeneity and captures existing gene-gene correlation patterns while accentuating subtler positive and negative correlations.The predicted expressions also show lower variance than the original dataset, consistent with noise reduction through imputation.
- Limitation: Averaging during imputation may remove genuine abrupt biological signals and produce artificially smoothed patterns, particularly for genes poorly correlated with image features.The authors suggest that larger reference datasets may mitigate this issue by reducing query-to-reference distances.
- Results: 39% to 120% improvements over the second-highest scoring method were observed across marker, highly expressed, and highly variable gene sets.Correlation with actual expression profiles was significantly higher across MG, HEG, and HVG evaluations.
- Implications: BLEEP could reduce the time and cost of gene expression profiling and support higher-throughput histology-image analysis in research and clinical applications.This is presented as the framework’s potential practical impact.