PaperlayerView exact revision on arXiv

Source-linked AI summary

PaSTel: Anchoring Histology in Spatial Transcriptomics via Multi-Scale Hierarchical Bio-Prior Contrastive Pretraining

Azim Dehghani Amirabad, Junchao Zhu, Pushpak Pati, Walid Abdelmoula, Tommaso Mansi, Rui Liao

arXiv:2608.14924v1cs.CVcs.AI

TL;DR

Spatial transcriptomics pretraining still struggles to select discriminative genes and model spatial dependencies beyond independent spot-patch alignment. PaSTel addresses these gaps with hierarchical biological priors and consistently outperforms existing vision and vision-omics encoders across downstream tasks, especially in data-scarce and heterogeneous tissue settings.

  • Problem

    Existing spatial-transcriptomics alignment methods use gene selections dominated by housekeeping signals and often overlook dependencies among neighboring spots.

  • Method

    PaSTel hierarchically combines TF-IDF gene selection, KEGG pathway anchoring, and region-level aggregation to align histology with multiscale biological structure.

  • Results

    PaSTel consistently outperforms strong vision and vision-omics baselines across gene prediction, few-shot learning, zero-shot clustering, and image-to-ST retrieval tasks.

  • Takeaways & Limitations

    The results support multiscale biological priors as a basis for robust and transferable spatial-transcriptomics representations, with strongest gains in data-scarce and heterogeneous tissues.

Abstract

from arXiv · show

Spatial transcriptomics (ST) links tissue morphology with molecular programs, motivating multimodal pretraining methods that align histology images with gene expression. However, existing approaches suffer from two key limitations: spatially informative gene selection is often dominated by ubiquitous housekeeping genes, leading to weakly discriminative representations, and independent spot-patch alignment fails to capture spatial dependencies that are critical for tissue organization. To address these challenges, we introduce PaSTel, a hierarchical multimodal pretraining framework that integrates biological priors at three levels. At the spot level, TF-IDF reweighting is used to identify spatially informative genes; at the functional level, curated KEGG pathways serve as anchors for encoding global biological semantics; and at the regional level, spatial clustering aggregates neighboring spots to model meso-scale tissue structure. Across multiple downstream tasks, PaSTel consistently outperforms existing vision and vision-omics encoders, demonstrating that incorporating multiscale biological priors yields more informative and transferable representations for spatial transcriptomics.

1. Introduction

Spatial transcriptomics provides molecular information aligned with histology but remains costly and difficult to scale, while existing vision-omics methods face limited generalization and incomplete spatial modeling. PaSTel addresses these challenges through hierarchical biological priors spanning informative genes, functional pathways, and meso-scale regions, producing transferable representations across spatial transcriptomics tasks.

  • Motivation: Spatial transcriptomics aligns molecular atlases with histological structure, but its high cost and technical complexity hinder clinical and large-cohort scalability.Pathology slides offer routinely captured morphological patterns associated with underlying expression programs, motivating histology-based molecular inference.
  • Limitations of Existing Methods: Early vision-driven methods generalize poorly across tissues, diseases, and sequencing platforms, motivating contrastive multimodal pretraining to align histology patches with gene expression.Recent multimodal frameworks aim to produce more transferable vision-omics representations.
  • PaSTel Framework: PaSTel uses TF-IDF reweighting at the spot level to suppress ubiquitous genes and emphasize spatially informative keywords, producing more discriminative representations.This biologically informed alignment improves the specificity and interpretability of spot representations.
  • PaSTel Framework: PaSTel adds curated KEGG pathway supervision and region-aware aggregation to encode global functional programs and meso-scale spatial dependencies.The framework progressively aligns pathway anchors with spots and groups spots into coherent regions to model long-range tissue organization.
  • Results: PaSTel consistently outperforms existing pretrained vision and vision-omics models across gene expression prediction, few-shot learning, zero-shot spatial clustering, and image-to-ST sentence retrieval.Its unified representation captures local molecular identity, global functional programs, and spatial tissue organization.

2. Methods

PaSTel is a hierarchical multimodal pretraining framework that aligns histology images with gene expression using biological priors at spot, functional, and regional levels. It combines TF-IDF gene selection, KEGG pathway supervision, and spatial region modeling to capture molecular identity, functional semantics, and tissue organization.

  • Overall framework: PaSTel jointly supervises spot-level molecular identity, functional programs, and regional tissue organization when aligning histology images with gene expression.The framework applies biological priors at three hierarchical levels.
  • Spot-level supervision: TF-IDF reweighting selects top-K genes per spot, emphasizing locally abundant and globally informative genes over ubiquitous housekeeping signals.The resulting gene sentence S(s_j) provides sparse, discriminative representations for multimodal alignment.
  • Functional-level supervision: Curated KEGG pathways provide biological anchors through normalized gene-set overlap scores and pathway prototype embeddings aligned with vision embeddings.The overlap-based soft supervision enables global functional semantics to be encoded alongside local expression patterns.
  • Regional-level supervision: Neighboring spots are grouped into regions using a hybrid distance that combines spatial proximity with feature similarity.A k-nearest-neighbor graph with weights W_ij = exp(−D_ij) is clustered by Leiden, with assignments periodically updated during training.
  • Joint optimization: Training combines symmetric InfoNCE for spot alignment, KL-based pathway supervision, and region-prototype alignment with weights (λ_1, λ_2, λ_3) = (1, 0.25, 0.25).Region prototypes are obtained by averaging embeddings within each cluster.

3. Data and Experiments

Experiments use leakage-controlled HEST-1K cohorts, with four held-out cohorts reserved for downstream evaluation and the remaining Visium cohorts used for pretraining. The framework is evaluated across four downstream tasks using a specified patch, gene-selection, encoder, and training setup.

  • Dataset: HEST-1K is split into disjoint pretraining and evaluation sets, holding out Breast Cancer, HER2, human DLPFC, and Kidney cohorts for downstream tasks.All remaining Visium-based cohorts are used only for pretraining.
  • Implementation: Experiments use 224 × 224 image patches, the top-250 TF-IDF genes per spot, and ViT-B/16 and transformer-based text encoders within OpenCLIP.Each gene is treated as an individual token.
  • Implementation: Training uses AdamW for 50 epochs on 8 A100 GPUs, enabling region-level aggregation after epoch 25.The region-level component is introduced during the latter half of training.
  • Tasks: The evaluation covers few-shot gene prediction, zero-shot spatial clustering, gene reconstruction, and image-to-ST retrieval.These tasks assess predictive performance and representation transferability.

4. Results

PaSTel consistently outperforms vision and vision-omics baselines across held-out spatial transcriptomics datasets and training-data regimes. It also achieves the strongest zero-shot clustering, gene-expression prediction, and biomarker prediction results described in the supplied results.

  • Limited-supervision adaptation: PaSTel consistently outperforms DenseNet121 and pathology-pretrained encoders across held-out datasets and training-data ratios, with the largest gains in low-data regimes.Baselines include CONCH, UNI, OmiCLIP, and UMPIRE under a unified comparison protocol.
  • Zero-shot clustering: Without finetuning or gene-expression access, PaSTel achieves the best vision-only zero-shot clustering performance on DLPFC laminar structures and HER2 tumor regions.The results indicate spatially coherent and semantically meaningful visual representations.
  • Cross-Validation: PaSTel achieves the best cross-validation performance across all datasets, including a Kidney PCC of 0.2627 versus OmiCLIP (0.2275).The evaluation predicts the top 300 highly variable genes using PCC, MSE, and MAE, and compares regression-based and retrieval-based methods.
  • Biomarker Prediction: For spatial prediction of IGLL5, PaSTel achieves the highest PCC (0.605 and 0.456) and recovers fine-grained spatial expression patterns.Competing methods produce weak or negative correlations.

5. Conclusion

PaSTel is a hierarchical multimodal pretraining framework that integrates biological priors into spatial transcriptomics vision–language alignment through gene selection, pathway anchoring, and region aggregation. It consistently outperforms strong baselines, especially in data-scarce and heterogeneous tissues, while ablations show that each component supports robust and transferable representations.

  • Framework: PaSTel integrates TF-IDF-based gene selection, pathway-level anchoring, and region-level aggregation into hierarchical vision–language alignment for spatial transcriptomics.The framework incorporates biological priors at spot, functional, and regional levels.
  • Results: PaSTel consistently outperforms strong vision and vision-omics baselines across a range of downstream tasks, with the largest improvements in data-scarce and heterogeneous tissue settings.These settings show the most pronounced gains reported in the conclusion.
  • Ablation: Ablation studies show that each component contributes meaningfully, highlighting multiscale biological structure as important for robust and transferable representations.The conclusion attributes representation robustness and transferability to modeling biological structure across multiple scales.

A. Appendix: Detailed Method · A.1. TF-IDF Formulation Details

PaSTel’s TF-IDF formulation stabilizes spot-level gene-expression counts, downweights genes common across spots, and emphasizes spatially specific genes. Top-K genes selected by TF-IDF weight are then used to construct gene sentences for each spot.

  • A.1. TF-IDF Formulation Details: The method first applies log(1 + x) to count-based gene-expression data to stabilize variance.This transformation precedes TF-IDF weighting.
  • A.1. TF-IDF Formulation Details: Term frequency measures a gene’s relative prominence within an individual spot.TF captures within-spot gene prominence.
  • A.1. TF-IDF Formulation Details: Inverse document frequency captures how rare a gene is across all spots.IDF provides the cross-spot rarity component of the weighting scheme.
  • A.1. TF-IDF Formulation Details: Genes frequently expressed across many spots receive lower IDF weights.The indicator function 1[·] is used in the IDF formulation.
  • A.1. TF-IDF Formulation Details: Spatially specific genes are emphasized through the TF-IDF weighting scheme.The weighting favors genes that are rare across spots rather than ubiquitous genes.
  • A.1. TF-IDF Formulation Details: Gene sentences are constructed by selecting the top-K genes per spot according to wj,i.The TF-IDF weights determine which genes are retained for each spot’s sentence representation.

A.2. Pathway Supervision Details

PaSTel uses curated KEGG pathways as biological priors for functional supervision. Pathway–spot relevance is computed from gene-set overlap, filtered by a threshold, and used as soft supervision with mean-pooled pathway prototypes.

  • Curated KEGG pathways provide functional supervision by linking each pathway’s gene set with a spot-specific gene set through normalized overlap.
  • The threshold ρ removes low-relevance pathway–spot pairs to reduce noisy supervision.
  • Pathway–spot overlap ω(P, S) is treated as a soft supervision signal because spots can participate in multiple biological processes.
  • Each pathway is represented by a prototype embedding obtained through mean pooling, summarizing pathway-level functional semantics.

A.3. Region Clustering Details

PaSTel models spatial structure by combining spatial proximity with multimodal similarity, then uses a weighted neighborhood graph and Leiden clustering to assign spots to regions. Region assignments are periodically recomputed as embeddings update during training.

  • A.3. Region Clustering Details: The region distance combines Euclidean spatial proximity with cosine dissimilarity between concatenated image and gene embeddings.The multimodal embedding is represented as [v_i; t_i].
  • A.3. Region Clustering Details: Spatial and feature distances are normalized to [0, 1] and combined with equal weights, α = β = 0.5.The normalization terms d_min, d_max, c_min, and c_max ensure both components share the same range.
  • A.3. Region Clustering Details: A weighted k-nearest neighbor graph is constructed from the combined region distance.
  • A.3. Region Clustering Details: Leiden clustering maximizes modularity to produce region assignments r_i for each spot.Clustering is recomputed periodically during training to reflect updated embeddings.

A.4. Multi-Level Loss Details … A.7. Ablation Study

PaSTel combines spot-, pathway-, and region-level objectives with training stabilizers, then demonstrates retrieval gains and ablation evidence for its hierarchical biological priors. The ablations identify regional alignment, pathway-overlap thresholds, loss weighting, and 250-token gene sentences as important design choices.

  • A.4. Multi-Level Loss Details: PaSTel uses symmetric InfoNCE for spot-level alignment, soft target and predicted pathway distributions with saturating KL divergence, and symmetric InfoNCE for regional embeddings.These objectives supervise spot, functional, and regional representations at multiple biological scales.
  • A.5. Training Details: Region assignments are updated periodically, while temperature scaling is applied to spot-level and pathway-level objectives to stabilize optimization.The training procedure adapts regional assignments as representations evolve.
  • A.6. Image-to-ST Sentence Retrieval: On Breast Cancer, PaSTel raises Recall@5% from 15.63 for OmiCLIP to 19.88 and ranks first or second across every recall threshold on Kidney.The retrieval task tests whether histology images match corresponding gene-expression descriptions.
  • A.6. Image-to-ST Sentence Retrieval: PaSTel processes gene sentences up to 250 tokens, while OmiCLIP and PLIP use 77-token inputs; atomic gene tokens preserve identity and broaden gene coverage.Standard BPE tokenization typically represents only 20–30 distinct genes within a 77-token budget.
  • A.7. Ablation Study: The ablation study removes pathway supervision and regional alignment and compares TF-IDF selection with Top Selection and HVG Selection under fixed 250-token sentences.Top Selection ranks genes by raw spot expression, whereas HVG Selection uses a slide-level high-variance gene set shared across spots.
  • A.7. Ablation Study: Removing regional alignment lowers Breast Cancer PCC from 0.2480 to 0.2162 and BC Recall@5% from 19.88 to 17.16, while clustering identifies it as most responsible for coherent tissue layers.This removal preserves spot-level alignment but discards meso-scale information.
  • A.7. Ablation Study: Using ρ = 0.1 introduces biologically unrelated positives, whereas ρ = 0.5 reduces anchor coverage and harms heterogeneous HER2 and Kidney cohorts relative to default ρ = 0.2.The pathway-overlap threshold controls how strictly pathways must share gene composition to form positive pairs.
  • A.7. Ablation Study: Reducing λ3 from 0.25 to 0.1 drops PCC from 0.3172 to 0.2854 on HER2 and from 0.2627 to 0.2358 on Kidney, while 250 tokens balances truncation and noise.At 500 tokens, Breast Cancer MSE changes from 0.6082 to 0.6406; 100 tokens loses discriminative genes.
Loading 2608.14924v1…