Source-linked AI summary
GATE-ST: Gene-Aware Text-image Encoder for Spatial Transcriptomics
Lucas Ni, Jian Luo, Wentao Huang, Chao Chen
TL;DR
Spatial gene-expression prediction from images offers spatial context, but text-based optimization remains relatively unexplored. GATE-ST integrates gene descriptions with image features through cross-attention, and outperforms other text-integration methods with both tested image encoders. The authors report potential for text-guided prediction while noting that evaluation used a relatively small dataset.
Problem
Method
GATE-ST encodes gene summaries and pathology images separately, then uses gene-conditioned cross-attention to align text embeddings with image morphology.
Source: §I · p. 1, B. Text Encoder · p. 2, C. Cross-attention Layer · p. 3Results
GATE-ST outperforms all other text-integration methods with both CONCH and UNI image encoders; with UNI, it yields 0.6774 MSE and 0.6613 Pearson.
Source: A. Main Experiment · p. 4Takeaways & Limitations
The results support the potential of text-based inputs for optimizing spatial transcriptomics predictions.
Source: §V · p. 5The experiments used a relatively small dataset, and the authors identify larger-dataset evaluation and improved text implementation as remaining opportunities.
Source: §V · p. 5
Abstract
from arXiv · showhide
Spatial transcriptomics enables spatially resolved gene expression analysis from slide-level images while preserving morphological features, providing valuable information for studying disease mechanisms and developing treatments. However, spatial gene expression profiling typically requires expensive and time-consuming tests. While existing image-based prediction optimizations mostly revolve around including positional embeddings and further image-based changes, text-based optimizations remain relatively unexplored. We present GATE-ST, which incorporates text-based inputs into image-based spatial gene expression predictions. With this approach, generated text descriptions of genes are utilized to better spatial transcriptomics prediction results. Gene summaries are put through a text encoder, generating embeddings that integrate with image embeddings through cross-attention layers to align with morphological features. We demonstrate the effectiveness of such text inputs by benchmarking performance against random gene embeddings and multiple other image-text fusion architectures, and show that GATE-ST outperforms these alternatives. Our results demonstrate the effectiveness of GATE-ST in pathology imaging, which may greatly reduce the time and cost of accurate spatial transcriptomic predictions, proving the potential of text-guided spatial gene expression prediction.
I. INTRODUCTION
Spatial transcriptomics preserves tissue context, but text-based optimization remains comparatively underexplored. GATE-ST uses gene descriptions alongside image features, extending existing image-based prediction approaches.
I. INTRODUCTION
- Spatial transcriptomics measures expression at tissue locations while retaining context, and H&E slides are inexpensive and widely available inputs.Bulk and single-cell RNA sequencing are costly and compromise morphological information; improved prediction could support disease detection.Source: §I · p. 1
- Textual gene descriptions offer biological information complementary to morphology, an underexplored source for image-based prediction.The passage contrasts this with visual and spatial-context optimization, where further morphological information is limited by patch content and may require larger datasets.Source: §I · p. 1
- GATE-ST incorporates gene-text summaries into image-based spatial gene-expression prediction and aligns text and image representations through cross-attention and MLP layers.The paper reports improvements over non-text-based models.Source: §I · p. 1
II. RELATED WORK
- Prior methods progress from patch-level prediction to spatial or broader-context modeling and reference-based approaches, but generally do not model individual genes’ semantic information.The passage names ST-Net, HisToGene, Hist2ST, M2OST, TRIPLEX, MERGE, BLEEP, EGN, and RankByGene.Source: §II · pp. 1–2
- GATE-ST uses gene-description embeddings as semantic queries over patch-level visual tokens and adds image-residual pathways to preserve morphology-derived information during fusion.Source: §II · p. 2
III. METHODOLOGY
GATE-ST predicts spatial gene expression by combining H&E image morphology with descriptions of individual genes. Cross-attention aligns the two, with the model targeting the 250 most highly expressed genes.
III. METHODOLOGY
- The method combines morphological information from H&E images with textual descriptions of individual genes, using cross-attention to align descriptions with relevant features.It trains a distinct representation for each gene and aims to minimize the difference between predicted expression and ground-truth measurements.Source: §III · p. 2
- The prediction targets are the 250 most highly expressed genes, matching the selection used by other leading models such as TRIPLEX.Source: §III · p. 2
- An image encoder and a text encoder project patch and gene-summary embeddings into a common space for three cross-attention layers, with a global image pathway preserving patch information.A final prediction network converts the resulting representations into predicted gene-expression values.Source: §III · p. 2
A. Image Encoder
GATE-ST encodes H&E patches into morphological features and gene summaries into text embeddings. Both streams are projected for cross-attention, with a trainable residual adapter applied to gene embeddings.
A. Image Encoder
- The image encoder extracts morphological features from H&E patches using a pretrained UNI or CONCH encoder.The input is a batch of image patches; UNI produces 196 spatial tokens of dimension 1024, whereas CONCH pools to one 512-dimensional embedding.Source: A. Image Encoder · p. 2
B. Text Encoder
- A pretrained CONCH text encoder processes summaries describing each target gene’s biological function, and a trainable residual adapter refines the resulting embeddings before projection.The passage notes that CONCH was pretrained to pair pathology images with textual descriptions, and that these representations may be trained to align with image-encoder morphology.Source: B. Text Encoder · p. 2
C. Cross-attention Layer
Gene embeddings query spatial image tokens through cross-attention to form gene-specific, image-conditioned representations. A parallel global image pathway preserves patch information through the attention stages.
C. Cross-attention Layer
- Each gene serves as a query over image keys and values, selecting morphological features to form an image-conditioned representation for that gene.Attention weights independently relate each gene representation to each spatial image token.Source: C. Cross-attention Layer · pp. 2–3
- The model stacks three cross-attention layers with four heads and a shared dimension of 256, passing updated gene embeddings forward as queries.Source: C. Cross-attention Layer · p. 3
- A mean-pooled global image representation is projected and broadcast across gene representations, then incorporated at every cross-attention stage.This residual pathway is added to preserve patch information.Source: C. Cross-attention Layer · p. 3
D. Prediction Head
After cross-attention, each gene has a feature vector that is processed independently by a shared prediction network. The network produces a prediction matrix with one expression value per gene and image patch.
D. Prediction Head
- A shared prediction network is applied independently to each gene’s feature vector after the final cross-attention layer.The final cross-attention representation is dc-dimensional.Source: D. Prediction Head · p. 3
- The network maps each gene representation through a hidden dimension to a scalar, producing one predicted expression value per gene and image patch.The predictions form a final matrix across genes and images.Source: D. Prediction Head · p. 3
IV. EXPERIMENTS
The experiments assess spatial gene-expression prediction on a HER2+ breast cancer dataset using image-only and text-image architectures. Performance is evaluated with MSE and Pearson correlation averaged across the selected genes.
IV. EXPERIMENTS
- Experiments use 36 spatially profiled HER2+ breast cancer tissue samples from 8 patients, with 13,136 spots and the 250 most highly expressed genes.Expressions are smoothed using each spot’s eight neighbors in a local 3 × 3 collection.Source: §IV · p. 3
- The baseline set includes image-only MLP prediction and text-image fusion variants using concatenation, embedding-image similarity, or cross-attention.A further cross-attention baseline adds a global image representation at each layer.Source: §IV · pp. 3–4
- MSE and Pearson correlation are averaged across 250 genes, with predicted and measured expression standardized across spots for each gene.MSE averages squared errors per gene; Pearson correlation is calculated across spots for each gene.Source: §IV · p. 4
- Table I lists the final hyperparameter configurations for models using UNI and CONCH image encoders.The table’s scope is the selected configurations for the two encoders.Source: Table I · p. 4
A. Main Experiment
GATE-ST outperforms the other text-integration methods with both CONCH and UNI image encoders. The results also suggest that text inputs help prediction, while stronger image encoders may benefit less from those additions.
A. Main Experiment
- 0.7280 MSE and 0.6360 Pearson coefficient with CONCH, and 0.6774 MSE and 0.6613 Pearson with UNI; GATE-ST outperforms all other text-integration methods.Table II compares gene-expression prediction architectures using the two image encoders.Source: Table II · p. 4, A. Main Experiment · p. 4
- Textual inputs that enable greater incorporation improve results, while stronger image encoders may benefit less, as reflected in UNI’s smaller improvements.The authors describe the additional information from text encoding as likely useful for gene prediction.Source: A. Main Experiment · p. 4
- The IGKC heatmaps compare predicted expression with ground truth across a particular slide, and the predictions show strong correspondence.The figure presents comparison heatmaps for IGKC expression.Source: A. Main Experiment · p. 4, Figure 2 · p. 4
B. Ablation
The ablation compares information-less Gaussian vectors with true text embeddings and reports better results with the full model. The comparison supports a contribution from textual inputs beyond the architecture alone.
B. Ablation
- CONCH achieved 0.7280 MSE and 0.6360 Pearson with final optimizations, versus 0.8191 MSE and 0.5910 Pearson with Gaussian vectors.The table caption identifies this as an ablation of individual GATE-ST components using CONCH and UNI image encoders.Source: Table III · p. 4, B. Ablation · p. 5
- UNI achieved 0.6774 MSE and 0.6613 Pearson with final optimizations, versus 0.7338 MSE and 0.6311 Pearson with Gaussian vectors.The random-vector control follows the same architecture as the fully optimized model.Source: B. Ablation · pp. 4–5
- The authors attribute the improvement over information-less vectors to textual inputs carrying meaningful information for morphological-feature alignment.The full model also improves with cross-attention, LoRA adaptation, and a global image residual.Source: B. Ablation · pp. 4–5
C. Hyperparameter Tuning
The tuning experiments tested learning rates and cross-attention layer counts. Performance changed little across the tested settings, and the final configurations used the best-performing settings.
C. Hyperparameter Tuning
- Across the tested settings, changing learning rates and cross-attention layer counts had little to no effect on model performance.The layer-count and learning-rate tuning figures correspond to the tested hyperparameters.Source: C. Hyperparameter Tuning · p. 5, Figure 3 · p. 5, Figure 4 · p. 5
- The experiments tested model stability across learning rates and numbers of cross-attention layers.The authors describe the model as resistant to small hyperparameter adjustments.Source: C. Hyperparameter Tuning · p. 5
- The final configurations were selected from the best-performing settings, although the tested changes did not meaningfully alter results.The reported stability applies to the settings tested.Source: C. Hyperparameter Tuning · p. 5
V. CONCLUSION
GATE-ST introduces text-based gene inputs for spatial transcriptomics predictions, a direction less explored than positional or image-based optimization. The authors report improvements over non-text inputs but note that evaluation used a relatively small dataset.
V. CONCLUSION
- GATE-ST uses text-based gene inputs, a relatively unexplored approach to optimizing spatial transcriptomics predictions.Prior optimization methods focused on positional encoding or image- and morphology-based changes.Source: §V · p. 5
- Text-based inputs improve performance and outperform random embeddings, supporting their information-carrying nature.The conclusion presents text-guided prediction as a promising direction for future work.Source: §V · p. 5
- The experiments used a relatively small dataset, and the authors identify larger-dataset evaluation and improved text implementation as future opportunities.The authors state that the work demonstrates the potential of text-based inputs for optimizing spatial transcriptomics predictions.Source: §V · p. 5